How to unifiedly detect sprites in the observation overall substrates?

[fmxFranky]

Hi, I am planning to generate a text-based description for RGB-based observation. To do this, I need to know the type of all sprites. For example, given a [88, 88, 3] observation, which contains 11x11=121 sprites, I wanna know what type it is for each sprite, i.e. classify it as "agent", "wall", "water", "apple", "sand", etc. Can you give me some advice for achieving my goal? Thanks a lot!

[duenez]

You probably would be better off using the LAYERS observation for this. This was discussed before https://github.com/google-deepmind/meltingpot/issues/67. However, sometimes we put some special and invisible objects in the world, so this method would violate those assumptions that they are invisible.

Another possibility would be to just add a custom observation in plain text. This is possible creating some Lua components, and we've had some success doing this internally.

[jzleibo]

Yeah, don't use the LAYERS observation, we have not documented its dynamics (and they could be extremely confusing).\

As Edgar said, the best way to do this is by adding custom observations. You'll have to read through the tutorial docs on substrate creation to see how to do this. If you still have questions after that then let us know, we would be happy to help.

[fmxFranky]

Thanks a lot! I will read tutorial docs on substrate creation and try to add custom observations to generate plain text for the original observations. And i will ask you if i still meet problems.

[fmxFranky]

Hi @jzleibo @duenez , recently I tried to read the source code for the Melting Pot. I am curious about the spacial relation of prefabs, i.e. the definition of "upperPhysical", "overlay" and "supperOverlay" and their mechanism for aggregating RGBA pixels. I wanna calculate the combined pixel results for the given two prefabs. For example, in Clean Up substrate, I can directly calculate "AVATAR_W + SAND" since the RGB color of AVATAR_W will entirely overwrite the background prefab's color so that one can get the sprite if one knows their ASCII map and corresponding color Platte. However, I cannot calculate the correct result for "SAND + SHADOW_N + SHADOW_E" since some prefabs' layer combination rule is "upperPysical" which seems like I need to know the fusion mechanism with some RGBA pixels. In addition, I don't find the relative code about this. Can you tell me how to obtain the combined image given two or more prefabs' ASCII maps with color platte( which is defined in shape.py and xxx_substrate.py) and layer relation?

[jzleibo]

The question of how precisely to fuse the RGBA values for sprites on different layers at the same x,y position is really a question about the underlying engine: dmlab2d. Maybe try asking it there https://github.com/google-deepmind/lab2d