search

dmah42 / dart-box2d

Box2D for Dart
http://dart-lang.github.io/dart-box2d/
34 stars 5 forks source link

Benchmarks should check consistent results #5

Open dmah42 opened 11 years ago

dmah42 commented 11 years ago

Currently, benchmarks exercise behaviour but do not check whether the calculations are correct (consistent with previous runs). Ideally, the benchmarks might be switched to unit tests but a start would be having the program spot and highlight deviations.

dmah42 commented 11 years ago

So began investigating...

For Box2D (C++), the closest to the demos we have are in the Testbed app (which uses .h files as loadable 'levels'). Don't see and checksums here though (yet)?

For JBox2D (Java), there are identically named demos in jbox2d-testbed, e.g: jbox2d-testbed/src/main/java/org/jbox2d/testbed/tests/DominoTest.java jbox2d-testbed/src/main/java/org/jbox2d/testbed/tests/DominoTower.java

However, (a) "mvn install" is failing for me (still investigating) and (b) there's again no mention of checksums!? (grep -ri checksum in C++ and Java repos find nothing. benchmark finds nothing in C++ and Java finds "jbox2d-testbed/src/main/java/org/jbox2d/testbed/perf/" which includes some values in comments.)

Searching, I think the answer to this may be "there is no such thing as consistency between platforms" per this link from the FAQ: http://code.google.com/p/box2d/wiki/FAQ#Is_Box2D_deterministic?

As such, is it sufficient (for now) to strive for intra-Dart consistency (definitely between fix-versions and hopefully between Dartium, DartVM and dart2js versions in Chrome and command-line V8)?