huggingface / lerobot

šŸ¤— LeRobot: Making AI for Robotics more accessible with end-to-end learning
Apache License 2.0
7.7k stars 746 forks source link

How to merge multiple recorded datasets? #533

Open mydhui opened 5 days ago

mydhui commented 5 days ago

Hi, Thank you so much for the automatic resume during data recordingļ¼Œsometimes ubstable camera issues or other situations (e.g. do not have enough time to finish recording) might cause process stopping.

I was wondering is there anyway to merge multiple recorded datasets? for instance I have two datasets 'cube grabbing' and 'cylinder grabbing' which were both recorded 50 episodes each and in the save environment, do you have tutorial about how to merge them into a 100-episode larger datasets?

BTW, another reason for merging datasets is because storage usage is extremely high before video encoding, and record large datasets at once can be limited by storage. but merge several encoded datasets can mitigate this problem.

Thanks

Vacuame commented 1 day ago

For me,I simply put the episodes together and run dataset.consolidate()