Add a separate `eval.n_envs` config for setting number of evaluation environments

huggingface / lerobot

🤗 LeRobot: End-to-end Learning for Real-World Robotics in Pytorch

Apache License 2.0

4.39k stars 357 forks source link

Add a separate `eval.n_envs` config for setting number of evaluation environments #297

Open eugeneteoh opened 1 week ago

eugeneteoh commented 1 week ago

Currently, there are two problems:

eval.batch_size is used for spinning up multiple environments (see code). I think this should be renamed to eval.n_envs for clarity.
eval.batch_size should be used for computing validation loss.

alexander-soare commented 1 week ago

Thanks for posting an issue on this @eugeneteoh. Btw what would you call it when we are rolling out in the env, vs when we are cross-validating loss calculation with a holdout set? Right now we have the config eval namespace dedicated only to the former.

eugeneteoh commented 5 days ago

I think keeping it under the eval namespace is fine. So it would be eval.n_envs and eval.batch_size.