Accessing environment state

[NiklasZ]

Hi,

I have managed to train a population of agents using your Rllib tutorial as a basis (many thanks for that). During the evaluation, I would like to collect some more detailed information about an environment's state, so I can better compare the agent's behaviours outside of the base return. Is there some way I can access the state of individual tiles + other environment properties via MeltingPotEnv?

Thanks!

[NiklasZ]

A concrete example problem I would like to solve: in the Capture the Flag environment, how can I count the number of tiles coloured red?

[duenez]

Hello,

The best way to extract (custom) information from a substrate is to push them as events. This is assuming you don't need them for agent observation (as observations require more machinery). This means you need to edit the Lua (unless we already have a component in Lua that exposes what you want).

In the case of your example for CTF, probably the easiest thing to do would be to add a component to the Scene object, and in that component you have an update function iterates over all objects (checking if they) that contain a Ground component (https://github.com/deepmind/meltingpot/blob/22f0a448fdcc19f0f9a1eb115c95f923900a0f8c/meltingpot/lua/levels/paintball/shared_components.lua#L80), then you emit an event counting how many of those are in state blue and how many in state red.

Something like:

local Ground = class.Class(component.Component)

function GroundCounter:__init__(kwargs)
  kwargs = args.parse(kwargs, {
      {'name', args.default('GroundCounter')},
  })
  GroundCounter.Base.__init__(self, kwargs)
end

function GroundCounter:update()
  local grounds = self.gameObject.simulation:getAllGameObjectsWithComponent('Ground')
  local counts = {
    clear = 0,
    red = 0,
    blue = 0,
  }
  for _, ground in pairs(grounds) do
    counts[ground:getState()] = counts[ground:getState()] + 1
  end
  events:add('grounds', 'dict',
               'red', counts['red'], 
               'blue', counts['blue'])
end
...

local allComponents = {
    ...
    GroundCounter = GroundCounter,
    ...
}

Then, you would add this component to the Scene object config: https://github.com/deepmind/meltingpot/blob/22f0a448fdcc19f0f9a1eb115c95f923900a0f8c/meltingpot/python/configs/substrates/paintball__capture_the_flag.py#L529

And you can now get the event by checking the substrate (as a python object) events() call, or by registering an observable on the events channel (similar to how you can register renderers for the third person view WORLD.RGB)

[NiklasZ]

Hey @duenez,

Thanks for the explanation and sample code. I was kind of afraid that I would have to dig into the lua code. I would still prefer to add this to the info returned by the environment, but I'll see if I can bake the observable into the MeltingPotEnv.