search

Denys88 / rl_games

RL implementations
MIT License
848 stars 142 forks source link

Evaluation worker feature #192

Open alex-petrenko opened 2 years ago

alex-petrenko commented 2 years ago

This adds a new feature that allows real-time evaluation and visualization of agents during the training session. The evaluation worker is supposed to run in a separate process from the training session and thus enables evaluation on a small number of agents (i.e. 1 or 64) which still leaves enough resources to train on thousands of agents.

This will be triggered in isaacgymenvs using a new flag. Alternatively player can be started with evaluation=True dir_to_monitor=/some/experiment/dir/containing/checkpoints to monitor an existing training session.

Player with evaluation=True will continuously monitor the experiment dir for new checkpoints, load them, and visualize a new policy.

alex-petrenko commented 2 years ago

@ViktorM FYI

Denys88 commented 2 years ago

@alex-petrenko will it work with IG on one gpu? as I remember I cannot create second IG on same gpu anyway?

alex-petrenko commented 2 years ago

@Denys88 it worked fine on my 1080Ti provided there's enough memory, although I only tried on my machine. Is there a fundamental reason why two IG can't coexist?

alex-petrenko commented 2 years ago

@ViktorM this is the version we'll need to use for the demo

Denys88 commented 2 years ago

@alex-petrenko please let me know if you are going to add a few more changes or I can just merge it and refactor later. Btw do you still need https://github.com/Denys88/rl_games/pull/195 this one?

alex-petrenko commented 2 years ago

@Denys88 I think it's solid and works reliably. We were able to use it with both IGE and Omniverse IsaacGym. It should be rather safe to merge since it does not do anything unless the evaluation flag is turned on.

If you don't want the file monitor thing (watchdog) to be in the main list of dependencies, you can remove it from setup py and add a warning that it should be installed under the evaluation section in the code.

@Denys88 not sure about #195 - this is something @ArthurAllshire should know more about

ViktorM commented 2 years ago

@Denys88 is it good to go?

Denys88 commented 2 years ago

@ViktorM not yet. need to test envpool and ray vecenvs first. and update readme. you can create a block with a new version.