huggingface / lerobot

🤗 LeRobot: Making AI for Robotics more accessible with end-to-end learning
Apache License 2.0
6.54k stars 582 forks source link

[Experimental] Decouple Experiment Tracking from Logger #262

Open WaelKarkoub opened 3 months ago

WaelKarkoub commented 3 months ago

What this does

NOTE: Please do not review the implementation, I need guidance/ideas to improve the architecture. I found that the experiment tracking is way too tightly coupled with the logger. I need to add tests, docs, etc... once the design is approved.

The goal for this PR is to decouple the experiment tracker which will allow us to add more experiment trackers besides wandb (eg. tensorboard, mlflow, etc.) easily.

cc @alexander-soare @Cadene

How it was tested

How to checkout & try? (for the reviewer)

poetry run python lerobot/scripts/train.py \
    policy=act \
    env=aloha \
    env.task=AlohaInsertion-v0 \
    wandb.enable=true \
    dataset_repo_id=lerobot/aloha_sim_insertion_human

This change is Reviewable

Cadene commented 3 months ago

@WaelKarkoub Thanks for this addition. As a high level guidance, the goal of LeRobot is to provide a lightweight and standardized end-to-end training framework. We dont want to support all experiment tracking, but just one that works very well for us (wandb for now). However, decoupling experiment tracking from logger is a good addition, as long as it provides a simple way for people to use there own experiment tracking. That's why I would suggest to also add a tutorial file to this PR, with an example of how to add a different experiment tracking. What do you think? What do you use as experiment tracking? What are the other experiment trackers that people would like to use?

Best

WaelKarkoub commented 3 months ago

@Cadene Thank you for the feedback! I might have to close this PR since the logger is heavily reliant on the config. I'll keep experimenting with the idea till I come to a good solution.