Training Mujoco-Gym Tasks with Image Observations

[nileshop22]

Hi @takuseno thanks for this amazing library!! I want to train Mujoco-Gym Continuous Control Tasks with Image Observations. I have few questions regarding this -

1. It seems like the dataset in D4RL only contains state observations for Mujoco-Gym tasks. Is there any way to get Image Observations from this state vector?
2. The results shown in the paper for Mujoco-Gym Tasks are for state observations. Did you tried training with Image observations? If yes, how did the different algorithms performed. If no, do you think they will perform good enough from just images?

EDIT: I found out that D4RL does provide image datasets here https://github.com/rail-berkeley/d4rl/tree/image_envs. How should I get this dataset in this library d3rlpy?

Thanks a lot!

[takuseno]

@nileshop22 Thank you for the issue. Technically it should be easy and already support the vision dataset. However, it seems the dataset is not available yet. For example, this URL is not ready. http://rail.eecs.berkeley.edu/datasets/offline_rl/gym_mujoco_vision/hopper-medium-v1.hdf5

[nileshop22]

Thanks for the response @takuseno. There should be some way to get Images from just state vector right for Mujoco Tasks? Are you aware of that?

[takuseno]

Yeah, you can do that. In that case, you need to write the script to convert the current d4rl dataset (e.g. "hopper-medium-v0") to a sequence of images by overwriting the internal variables in the simulator. I think it won't be so hard, however, d3rlpy does not currently provide it. Once you make it work, it should be easy to use the dataset to train offline RL with d3rlpy.

[nileshop22]

Great! Thanks. I'll see if there already exists any such script. Also, which environments in d3rlpy currently fully supports Offline RL training with Images?

[takuseno]

Regarding the second question about the performance, my guess is that it should perform as good as the vector dataset if the data distribution is the same. Possibly, it may take longer to train until the algorithm achieves good performance since we need to train CNN, which is relatively larger than the standard [256, 256] MLP.

[takuseno]

Currently, it supports Atari 2600 datasets. https://github.com/takuseno/d3rlpy/blob/master/reproductions/offline/discrete_cql.py

I don't think there are standardized image observation datasets in continuous control.

[nileshop22]

Sounds great! Maybe you should try incorporating robomimic which has datasets for Robotic Manipulation in d3rlpy in upcoming versions. Also, do let me know if you need help in this. I'm little bit familiar with it's codebase.

[takuseno]

Ah, this looks super nice! I somehow missed their datasets. I'll support it after the upcoming v1.0.0 release.

[nileshop22]

Hi @takuseno, is there any simple way to integrate RL Unplugged in this library?

[takuseno]

@nileshop22 RL Unpluggled is not officially supported. But, you can construct the dataset manually. https://d3rlpy.readthedocs.io/en/v0.91/tips.html#create-your-own-dataset