How do we check changes don't affect performance

[szhorvat]

Well, this was a minor part of the proposal we submitted to the CZI, and something I always wanted to do, but we didn't really have the manpower.

This should be done primarily for the C core.

It would be nice to have benchmarks running continuously, and having daily reports. We do have some benchmarks in https://github.com/igraph/igraph/tree/master/tests/benchmarks

What is missing:

• We need a physical computer that is dedicated to the task. This is easy, as a start even a Raspberry Pi would do. Visrtual machine in the cloud are not suitable for benchmarking.
• We need a place to keep this computer, in such a way that several of us would have remote access, and at least one person would have physical access. This is surprisingly hard. I asked my institute, but after the cyberattack they suffered, I don't know if they'll allow external people access to a machine on their network ...
• We need some scripts to parse the benchmark output, summarize it, and push some sort of report to a webpage. This takes some time to develop.

[maelle]

I opened this because of part of the discussion in #1327, for context.

[maelle]

Virtual machine in the cloud are not suitable for benchmarking:

Why? Because it's not the same machine over time?

[szhorvat]

Yes, because it's not the same machine, and because a single physical machine hosts multiple virtual machines. Basically, performance is unpredictable.

[szhorvat]

I opened this because of part of the discussion in #1327, for context.

Sorry, so this was meant to be R specific.

For that, just keep in mind the issues with RStudio significantly slowing down interruption checks (likely by an unpredictable amount that depends on the state of its GUI ...). If you develop some benchmarks for R specifically, it's a good idea to run them from the command line, with no GUI.

[maelle]

https://github.com/lorenzwalthert/touchstone