matteobettini
commented
3 months ago The next PR will allow any shape observation and state.
However there is a problem. DDPG and SAC critics take as input the concatenation of the state and the actions.
Up until now we had only 1-dimensional states and observation so concatenating the actions was easy.
Now if the state and obs can be pictures, concatenating them to actions is less trivial.
They should ideally be prrcessed by different nets before being concatenated.
This poses a non-trivial problem to the way BenchMARL is designed. In particular:
- BenchMARL models, algorithms and envs are designed to be decoupled. I want to avoid the rllib way where the codebase realizes that you are using images and instantiates a CNN for you. If the user runs Meltingpot with an MLP i want an error to be thrown telling the user to explicitly add a CNN to their model.
Proposed solution
The proposed solution is to allow models to get more than one key as input.
For example if an MLP receives 2 keys, it will concatenate them before processing (just like in torchrl https://github.com/pytorch/rl/blob/2461eb20d21b79a410e01aed71c26b77712a30d8/torchrl/modules/models/models.py#L295)
If an CNN receives multiple inputs, it will also try to concatenate them, but if they have different shapes, it will just cat the images, process them and then cat the other tensors to the output.
This means that if you have a CNN+MLP model and you pass it an image and a tensor, the CNN will process the image and cat the output to the other inputs and the MLP will process this
I implemented this in #73
codecov-commenter
This PR allows any name for the group observation key and the global state key.
Before benchmarl constrained the observation to be named "observation" and the global state to be named "state"
cc @ezhang7423