More inputs other than image

[alaliqing]

Hi, how can I give more input information to the agent, like some scalar input? Because I found different input information in the original paper (below), how can I achieve this and is it possible?

[NM512]

@alaliqing Thanks for asking! There are options named "mlp_keys" and "cnn_keys" for that purpose in config. These are used for selecting features by regular expression in MultiEncoder and MultiDecoder classes. So, you can use that to fuse any image and state features you want.

[alaliqing]

Thanks for your help, I am using Dreamer to train my autonomous vehicle to avoid obstacles in Carla, is it possible to add a specific obstacle location as an input? Should I add this information as an "obs" or "info" item after changing "mlp_keys" or "cnn_keys"? Do you have any existing examples of how to do this? I'm sorry to bother you, but I'm unfamiliar with the structure of neural networks.

[NM512]

It sounds interesting to test Dreamer in the autonomous driving task. This is an example code to overwrite those keys when you want to use image and any state inputs provided from environments.

encoder: {mlp_keys: '.*', cnn_keys: 'image'}

or you can select multiple imputs by writing like the following.

# in this case, state inputs named inventory, inventory_max and equipped with image imput of  image are passed to MultiEncoder. 
encoder: {mlp_keys: 'inventory|inventory_max|equipped|', cnn_keys: 'image'}

Don't forget to prepare the corresponding dict-type observations in environment side like followings. https://github.com/NM512/dreamerv3-torch/blob/6c861ca7cb5d59754da508abb6ce6599a503124a/envs/dmc.py#L24-L33 https://github.com/NM512/dreamerv3-torch/blob/6c861ca7cb5d59754da508abb6ce6599a503124a/envs/dmc.py#L49-L53

I hope this helps you even just a little.

[alaliqing]

Thank you! That should be helpful : )