Benchmarking

[leenozara]

When changes to kernel and system modules are made, as well as changes to dependencies, we should run a series of standard benchmarks. These would include both rust/cargo benchmarks and common actor system benchmarks (such as Ring Token and Skynet).

These tests would provide a benchmark between riker versions to ensure changes made don’t impact performance.

Additionally, benchmarks could be made between different actor systems both rust based and Scala, Erlang, C++, etc. This is less of a priority however at this point.

See: https://arxiv.org/pdf/1505.07368.pdf https://github.com/actor-framework/benchmarks

[hardliner66]

I might be wrong, but I think something like #58 is needed for some/most of these benchmarks in order to get the time from system start to system end. The tutorials currently solves this by using Thread::sleep(), which shouldn't be used in benchmarks. Otherwise we are only measuring the Thread::sleep.

[leenozara]

It may be that what’s being tested doesn’t include system start and end, since that would skew the results favoring systems that have high start/end times, regardless of test.

In any case though, the usage of Thread::sleep() throughout the project does need to be replaced as per #58. Testing overall really needs this as CI build times are becoming longer, since some (many?) tests involve some sleep.

[hardliner66]

You are correct. I didn't think about tests not involving startuptime. Though I do think that we need a benchmark for that in order to show more complete performance numbers.

Depending on the use-case startup time can be a big factor. If I write a server which runs for weeks/months/years, startup time doesn't really matter. If I write a desktop application or something on embedded, it might matter more.