It feels like benchmarks are missing the point

[mayakwd]

Hi!

I think your benchmarks are a bit off the road. I just found them, and it was astonishing that tick-knock is doing only 44 operations per second. So I deep dive into simple iterations benchmark.

I introduced a simple counter, like: const counter = { ape: 0, javelin: 0, ...etc }, and put it outside benchmarks. Values of it are increasing whenever the system does position.x =+ velocity.x.

So I checked how many iterations were executed for each ECS engine.

• for ape, bitecs, javelin, and tick-knock, the system update was invoked 1.050.000 times
• for sim-ecs, it's 2.500 iterations in total.

What am I missing? Please advice.

[minecrawler]

Hi @mayakwd ,

Thank you for taking interest in sim-ecs! The benchmarks were based on the Rust ECS Bench Suite. While I'm not happy with their workings and the tuning, yet, they seem to reflect changes in my code rather well and have helped me understand performance regressions in the past. Pitching them against other engines is a nice bonus, without too much benefit, though. The benchmark is synthetic and does not reflect how each library may perform in a real project.

The benchmarks are open-source and you can take a peek at how the code works. If you aren't happy with tick-knocks numbers, you can check the tick-knock bench code and look for any obvious mistakes or improvement potentials.

Just as well you can take a look at the sim-ecs simple iter bench code in order to understand how it is executed.

Unfortunately, I cannot comment on your test without actually seeing your code. Please provide it (ideally as an executable project) so I can take a look :)

[mayakwd]

Thank you for the reply! I was looking for ways to optimize tick-knock. And was wondering why it is so slow compared to your library. In the end, I found out that the numbers were just misleading, so I was chasing for nothing :) So I've prepared a sample for you. It's just the same bench but with the counters and commented-out all benchmarks other than simple iteration. Please check the code: https://github.com/mayakwd/sim-ecs/tree/benchmark-iterations-displaying Or maybe I still don't understand the whole picture.

[minecrawler]

I found the error. The reset code for the sim-ecs iter benchmark is wrong, which leads to the result described by you. I'll fix it in a future update. Thank you for testing and reporting the issue! I'll leave this open until I push the fix. Do you have any more issues or questions for this topic?

[mayakwd]

No, thank you!

[minecrawler]

@mayakwd I pushed the changes to my benchmarks and re-tested the iteration counts. In addition, I added some more tweaks to also re-enable the Scheduling test, which is not a simple iteration over entities, but also adds scheduling. It's a bit closer to real scenarios. I guess the next step would be to compare timings of the Pong example for a few frames with the different libraries, but to be honest, that sounds like a lot of work.

Anyway, the new results are in the README. While tick-knock looks good, it seems it has a deficite in scheduling, or I'm doing something wrong. Maybe you can check that and if my code is ok, I hope you can find performance-improvements in the code!

[mayakwd]

Oh, thank you for the update! I'll take a look.