Live Grader not Reporting Time

stanfordnmbl / osim-rl

Reinforcement learning environments with musculoskeletal models

http://osim-rl.stanford.edu/

MIT License

896 stars 249 forks source link

Live Grader not Reporting Time #194

Open Ryan-Amaral opened 5 years ago

Ryan-Amaral commented 5 years ago

When submitting to the competition through the live grader (not docker), the mean simulation time comes out as 0. An example can be seen here https://www.aicrowd.com/challenges/neurips-2019-learn-to-move-walk-around/submissions/10540, and on other live grader submissions that I saw.

Edit: Though I now realize that timing will be unreliable since it is ran on the user's hardware, so I guess this is not so relevant.

smsong commented 5 years ago

@Ryan-Amaral Thanks for your note and update. Yes, it was a decision made by AIcrowd not to include simulation time for some reasons including the one you pointed out. Please let us know if there is a specific concern related to not showing simulation times; otherwise, we will probably go with the current set up.