search

yanlai00 / bridge_data_robot_infra

21 stars 2 forks source link

time sync confusion #1

Open JiaheXu opened 1 year ago

JiaheXu commented 1 year ago

Hi Bridge_data developers,

I am doing some work with bridge_data, and I have some confusion about the pictures and the corresponding actions. In robonetv2/toykitchen_fixed_cam/berkeley/realkitchen1_counter/put_spoon_on_plate/2021-08-13_19-16-28/raw/traj_group0/traj1 the action shows that seq 0-13 the grasper's actions are 1 (I assume it is opening?) and the grasping action begins from 14th pic, but in the images/ folder, it seems the grasping starts from 16th pic. I am not sure if this is a problem, please take a look, thanks.

yanlai00 commented 1 year ago

This is a known issue of the dataset. Unfortunately, due to the delay of some components in our system, the camera image can be 1-2 time steps behind (e.g. the first couple of images are the same for every trajectory, and movement starts at 3rd image). To roughly re-sync the images and actions, you can remove these static images from the front.

JiaheXu commented 1 year ago

Is there any way to align the data automatically?

JiaheXu commented 1 year ago

If I set up the same settings following the paper, will the delay be the same? I want to do some real-life tests. I am not sure if I purchase all the components, can things work properly due to the delay on the images in this dataset? @yanlai00

yanlai00 commented 1 year ago

I believe so. In the bridge data paper, we did not do any delay adjustment when training and evaluating the models, and still achieved reasonably good results.

In our more recent PTR paper (https://paperswithcode.com/paper/pre-training-for-robots-offline-rl-enables), we did include some adjustments for the camera delay.