Closed spMohanty closed 4 years ago
Currently here are our estimated for the timelimits and resources that will be made available to the participants, at least in the extrinsic phase (Round-1).
- intrinsic phase : 1e7 - extrinsic phase : 2e3 - extrinsic trials : 350 # Resources : # Round-1 - time : 6 hours - cpu : - 8 - memory : 16 gb - gpu : 1 k80
We need to do some benchmarking to ensure that these numbers correlate well with each other. @emilio-cartoni : Did you do some benchmarking internally ?
I think we can include a cli script in the library which can do the benchmarking for any controller, so that participants have a good sense of the performance of their controllers.
We tried a few things, including running some algorithms like PPO and the bottleneck was running the environment (note that we have a GTX 1080 Ti, which is quite good as a desktop GPU - it was top end a couple years ago).
I see in the progress bars that you have added in the evaluation function that the intrinsic phase runs @60fps here (and it uses 1 CPU). This means 10M -> about 46 hours of 1 CPU.
6 hours @ 60 fps would be 3.24 hours (using RandomPolicy which probably takes negligible time to compute its step) Notice that unless the controller does some job in parallel to the environment, this means that their controller is stuck waiting for 3.24 hours out of 6...
Currently here are our estimated for the timelimits and resources that will be made available to the participants, at least in the extrinsic phase (Round-1).
We need to do some benchmarking to ensure that these numbers correlate well with each other. @emilio-cartoni : Did you do some benchmarking internally ?
I think we can include a cli script in the library which can do the benchmarking for any controller, so that participants have a good sense of the performance of their controllers.