search

huggingface / lerobot

🤗 LeRobot: End-to-end Learning for Real-World Robotics in Pytorch
Apache License 2.0
4.29k stars 335 forks source link

Upload `replay.mp4` alongside models #227

Open qgallouedec opened 1 month ago

qgallouedec commented 1 month ago

The hub will soon be able to render the policy (as we can currently do with the hub's rl policies).

It works as follows: if the model is tagged "robotics" and if there is replay.mp4 in the root directory of the repo, then this file will be rendered on the right as shown here:

image

It might be cool to

  1. Replace the GIF of the current policies by a replay video file.
  2. Add an option in the eval script that allow to render and save a video in the pretrained folder. A concatenation of these videos?: https://github.com/huggingface/lerobot/blob/2c2e4e14edd5586bd6cced4a3a7af656548ff282/lerobot/scripts/eval.py#L351

In the future, what would be really hot would be to have a real-world video rendering.

Links: https://github.com/huggingface/moon-landing/pull/10170 https://github.com/huggingface/moon-landing/issues/10161 https://github.com/huggingface/huggingface.js/issues/722

alexander-soare commented 3 weeks ago

Thanks Quentin! That's a very useful feature.

I'm thinking maybe this should be left manual for now? Because we haven't even got a script yet for uploading models to the hub. We aren't going to be chugging out tonnes of models in the very near future, and a lot of manual work goes into making a model anyway. It doesn't feel sensible to automate this. I'd say we can just pick a video from the eval, drag it into the folder, and rename it. If you want to concat videos, use ffmpeg in your terminal.

I don't mean to disagree for the sake of it. Someone approached me about covering this issue with a PR and we discussed and agreed on my thoughts summarised above.

qgallouedec commented 3 weeks ago

I agree with you, not urgent nor sensitive at all.

Realmbird commented 3 weeks ago

Can I try working on this issue?

alexander-soare commented 3 weeks ago

@Realmbird thanks for volunteering. Based on the discussion above, I think we don't want to do anything about this issue for the time being (mainly because we don't want to automate something that doesn't need to be automated). But do let us know what you had in mind if you think it makes sense to do.

If you're on the Discord, do feel free to reach out on a public channel or even DM me if you want to scope out other issues to tackle.