search

ARISE-Initiative / robomimic

robomimic: A Modular Framework for Robot Learning from Demonstration
MIT License
592 stars 181 forks source link

Saving image observations from run_trained_agent.py #147

Closed masha-itkina-tri closed 5 months ago

masha-itkina-tri commented 5 months ago

I am using the --dataset_obs flag to save image observations of the evaluations in run_trained_agent.py. When I plot the images after saving, the first time step looks normal, and all subsequent ones have inverted colors.

I am hoping to confirm that this is just a saving issue and not a problem with the evaluation. robomimic_obs_saved

amandlek commented 5 months ago

I just tried evaluating an agent on the Lift task using a command like python playback_dataset.py --dataset /tmp/agent.hdf5 --use-obs --render_image_names agentview_image --video_path /tmp/agent_ds.mp4 on the master branch, and things seem to be fine. This might be something local to the branch you're on?

masha-itkina-tri commented 5 months ago

Hello! The .mp4 saving is correct because it is rendered directly. Try playing back the observations saved in hdf5 using the --dataset_obs flag. These should be 84x84 images for diffusion policy for example.

amandlek commented 5 months ago

I'm sorry I posted the wrong command, I meant I used something like: python run_trained_agent.py --agent /home/amandlekar/installed_libraries/robomimic/robomimic/../bc_trained_models/core/bc/lift/ph/image/trained_models/core_bc_lift_ph_image/20240326130827/models/model_epoch_13.pth --dataset_path /tmp/agent.hdf5 --dataset_obs --n_rollouts 2

The functionality is working as expected on the master branch. What branch are you on? Like I said, this functionality is likely broken on the branch you're using.

masha-itkina-tri commented 5 months ago

I totally agree that it runs. But if you then load the .hdf5 dataset and visualize the observations from the cameras, the images are as above for me. I am working off of master I believe, but let me double check. Also, you should have cameras set in your command I believe?

amandlek commented 5 months ago

To be clear, I did try visualizing the observations and they are running fine on my branch. I strongly suspect you are on a different branch - did you double check?

masha-itkina-tri commented 5 months ago

You are right, it was the r2d2 branch!

masha-itkina-tri commented 4 months ago

@amandlek do you know if this is a bug in the r2d2 branch or expected behavior?

amandlek commented 4 months ago

I believe it's a bug - as I mentioned before, this isn't a problem on the master branch