Open StarCycle opened 4 weeks ago
Interesting :)
Could you try your approach during data recording on a real robot?
See this PR: https://github.com/huggingface/lerobot/pull/326
Record an episode:
python lerobot/scripts/control_robot.py record \
--fps 30 \
--root data \
--repo-id $USER/koch_pick_place_lego \
--num-episodes 50 \
--warmup-time-s 2 \
--episode-time-s 30 \
--reset-time-s 10
We currently use threads to write images asychronously: https://github.com/huggingface/lerobot/blob/f179084709967fd237795df118c756901ab4535a/lerobot/scripts/control_robot.py#L413 https://github.com/huggingface/lerobot/blob/f179084709967fd237795df118c756901ab4535a/lerobot/scripts/control_robot.py#L435-L436
Then we encode video frames at the end on the recording: https://github.com/huggingface/lerobot/blob/f179084709967fd237795df118c756901ab4535a/lerobot/scripts/control_robot.py#L593
This ensures high fps during teleoperation of your robot.
Your method could be better, but we need to try!
Importantly, you need to use the exact encoding that we are using (see the default parameters with vcodec="libsvtav1"
:
https://github.com/huggingface/lerobot/blob/f179084709967fd237795df118c756901ab4535a/lerobot/common/datasets/video_utils.py#L165-L211
Hello @Cadene ,
When transfering other dataset formats to LeRobot dataset, the current approach is actually first converting all images into png, and then converting all pngs into mp4 video with the following function:
https://github.com/huggingface/lerobot/blob/fab037f78d63ba578fffb25548eb37093cf1d7a7/lerobot/common/datasets/video_utils.py#L165-L190
It directly calls then ffmpeg command. It's fine for converting an exist dataset to LeRobot dataset. However, if you are recording a dataset in real time, it's a little complicated.
Another approach is to add images to videos in a stream, like link:
An example is:
Do you prefer this approach? Or there are some special reasons to first convert all images into png?
cc @Nirviaje