Note: The target branch is set to feature/improve-ci-cd-workflow as it depends on it and so you can see the changes related to this branch. Once the target branch is merged, the base branch of this PR will be updated automatically
This adds support for automated benchmarking and profiling to our benchmarks project.
These benchmarks are automatically executed and the data updated on every push to master and periodically once a week on a dedicated self hosted physical windows runner (see the corresponding readme).
It is gathered on a linux runner as pyroscope currently only supports linux and macos.
As i do currently not yet have a physical self hosted linux runner setup, the data is gathered using a regular github action runner. Hence the data will be inconsistent atm.
The data is also automatically collected and updated on every push to master and periodically once a week.
Future plans
Move the spreadsheet data to an influx db so it can also be queried and displayed using grafana which is way more powerful than a spreadsheeet based solution
Setup a physical linux runner for consistent data
Note: the maintainers received a user for grafana to query data at will and create/edit dashboards. The two links above are read only and cannot be edited.
This data also contains code for a task which compares new benchmark data with existing benchmark data to detect if new code has a worse performance. This task is unused atm as the data is too inconsistent yet to make save assumptions on code performance and deviations. I left it here as the task might me useful none the less. But i can also remove it again if preferred.
This adds support for automated benchmarking and profiling to our
benchmarksproject.Data
Benchmark data
Can be found here: https://docs.google.com/spreadsheets/d/1aIclio2oYs_ua_4tENVjfn-OrwKkgVSCRfo50NTMve0/edit?usp=sharing Everything in this spread sheet is autogenerated from a google app script which takes the benchmark data json we send it.
These benchmarks are automatically executed and the data updated on every push to master and periodically once a week on a dedicated self hosted physical windows runner (see the corresponding readme).
Profiling data
Can be found here (public read only link): https://monitoring.hippmann.ch/public-dashboards/22a47dbec7e04eed8fdad1135e22ef09 The profiling data is taken from the
benchmarksproject using pyroscope and grafana.It is gathered on a linux runner as pyroscope currently only supports linux and macos. As i do currently not yet have a physical self hosted linux runner setup, the data is gathered using a regular github action runner. Hence the data will be inconsistent atm. The data is also automatically collected and updated on every push to master and periodically once a week.
Future plans
Note: the maintainers received a user for grafana to query data at will and create/edit dashboards. The two links above are read only and cannot be edited.
This data also contains code for a task which compares new benchmark data with existing benchmark data to detect if new code has a worse performance. This task is unused atm as the data is too inconsistent yet to make save assumptions on code performance and deviations. I left it here as the task might me useful none the less. But i can also remove it again if preferred.