search

RobotLocomotion / drake

Model-based design and verification for robotics.
https://drake.mit.edu
Other
3.25k stars 1.25k forks source link

Benchmark Testing #11100

Closed acxz closed 5 years ago

acxz commented 5 years ago

I would like to preface this by saying this is not an issue regarding a bug or anything based on the code. If this is not the appropriate place to post the following message, please point me to the right location.

There are many physics engines out there for the simulation/robotics community. As such, there is a need for benchmarking various engines for accurate and objective comparisons. Benchmarking also helps end users determine which is the appropriate engine for their task and for developers to help debug and better their own engines.

There is ongoing work for benchmarking at SimBenchmark. As of now RaiSim, Bullet, ODE, MuJoCo, and DART have been benchmarked (results), but it would be great to have more engines benchmarked.

As such, I would like to invite members of the Drake community to help test & benchmark Drake under the same framework. Based on my (limited) knowledge, this is the first time such benchmarking has been openly available. Utilizing this instead of performing closed benchmarking is beneficial for both the robotics community as well as the Drake community.

RussTedrake commented 5 years ago

I think simulator benchmarks are great, but admit that I am always a bit suspicious of benchmarks hosted by authors of one of the packages being benchmarked -- somehow the host's package invariably comes out ahead! Drake is open source and anyone is welcome to benchmark it and hopefully improve it also.

Note that we have made a commitment to accuracy here, with comparatively extensive investment in accuracy, and testing against any known (especially closed-form) mechanics solutions.

acxz commented 5 years ago

I very much agree with that sentiment. That is why when I found out about an open-sourced benchmark tool, I was very happy. I have not seen any other packages open-source their benchmark software, which is why I believe this is a great tool to the community. I personally am not interested in closed source engines at all, so I could care less about RaiSim or Mujuco, but the tool that RaiSim developers have made available is promising.