search

AIcrowd / real_robots

Gym environments for Robots that learn to interact with the environment autonomously
https://www.aicrowd.com/challenges/neurips-2019-robot-open-ended-autonomous-learning
MIT License
34 stars 18 forks source link

[Benchmarking] Good estimation of Timelimits & Resources for participants #7

Closed spMohanty closed 4 years ago

spMohanty commented 5 years ago

Currently here are our estimated for the timelimits and resources that will be made available to the participants, at least in the extrinsic phase (Round-1). 

    - intrinsic phase : 1e7
    - extrinsic phase : 2e3
    - extrinsic trials : 350
    # Resources :
        # Round-1 
            - time : 6 hours 
            - cpu : 
                - 8 
                - memory : 30 gb
                - gpu : 1 k80

We need to do some benchmarking to ensure that these numbers correlate well with each other. @emilio-cartoni : Did you do some benchmarking internally ?

I think we can include a cli script in the library which can do the benchmarking for any controller, so that participants have a good sense of the performance of their controllers.

emilio-cartoni commented 5 years ago

Currently here are our estimated for the timelimits and resources that will be made available to the participants, at least in the extrinsic phase (Round-1).

    - intrinsic phase : 1e7
    - extrinsic phase : 2e3
    - extrinsic trials : 350
    # Resources :
        # Round-1 
            - time : 6 hours 
            - cpu : 
                - 8 
                - memory : 16 gb
                - gpu : 1 k80

We need to do some benchmarking to ensure that these numbers correlate well with each other. @emilio-cartoni : Did you do some benchmarking internally ?

I think we can include a cli script in the library which can do the benchmarking for any controller, so that participants have a good sense of the performance of their controllers.

We tried a few things, including running some algorithms like PPO and the bottleneck was running the environment (note that we have a GTX 1080 Ti, which is quite good as a desktop GPU - it was top end a couple years ago).

I see in the progress bars that you have added in the evaluation function that the intrinsic phase runs @60fps here (and it uses 1 CPU). This means 10M -> about 46 hours of 1 CPU.

emilio-cartoni commented 5 years ago

6 hours @ 60 fps would be 3.24 hours (using RandomPolicy which probably takes negligible time to compute its step) Notice that unless the controller does some job in parallel to the environment, this means that their controller is stuck waiting for 3.24 hours out of 6...