CI for generating results

arey / java-object-mapper-benchmark

JMH benchmark of Java object-to-object mapping frameworks

287 stars 53 forks source link

CI for generating results #23

Closed filiphr closed 5 years ago

filiphr commented 5 years ago

This is an idea how the results could be automatically generated by a CI.

It uses GitHub Actions which is currently in Beta, but you can request access to it. I think they are accepting everyone really fast.

The PR consists of 3 commits:

Adding the maven wrapper - In order to be able to use it in the CI
Add Github actions worfklow - The workflow consists of the following steps
- Run the tests for the project
- Build and run the benchmark for every mapper runs separately for each mapper
- Collect the benchmark results and create a plot
- Upload the benchmark results and plot as an artifact to the build
Update readme for the command to run a benchmark for a single mapper

If you want to see how it looks you can have a look in the actions from my fork

arey commented 5 years ago

Nice work @filiphr. I've subscribed to the waitlist for the GitHub Actions Beta. I see that the benchmark_results and the results archives could be downloaded from the Download artifacts menu. We may go further by automatically pushing the results to the repo (without relaunching the benchmark). On the other hand, by launching the microbenchmark by a CI, we don't manage the CPU and the RAM allocated to the bench. Moreover, they may be shared with other builds. Results will not be reliable. Isn't it?

filiphr commented 5 years ago

Thanks @arey, glad that you liked it.

We may go further by automatically pushing the results to the repo (without relaunching the benchmark).

I was thinking about this as well, I even found an action that would allow you to do the commit you need. However, I noticed that sometimes the numbers were off. I couldn't find the CPU and RAM allocated to the jobs for the Github actions, so it could be unreliable. What we could do though is to pass -Xmx when running it, not sure if that might make it more reliable.

We could also print some system information for each microbenchmark.

arey commented 4 years ago

Hi @filiphr I've now access to the beta. On my account, the build is failling while gnuplopt installation. Do you have any idea? https://github.com/arey/java-object-mapper-benchmark/runs/265133955

filiphr commented 4 years ago

It was failing on mine as well. Figured out the problem and created a new PR https://github.com/arey/java-object-mapper-benchmark/pull/24.

arey commented 4 years ago

Thank you @filiphr. It's re-working.

filiphr commented 4 years ago

I managed to find the hardware resource resources for the runners.

Each virtual machine has the same hardware resources available.

2 core CPUs 7 GB of RAM memory 14 GB of SSD disk space

More information can be found here

arey commented 4 years ago

The DSv2-series link references that runners runs on the Intel® Xeon® 8171M 2.1GHz (Skylake) or the the Intel® Xeon® E5-2673 v4 2.3 GHz (Broadwell) or the Intel® Xeon® E5-2673 v3 2.4 GHz (Haswell) processors with Intel Turbo Boost Technology 2.0 and use premium storage.

With the Add & Commit plugin, we could mix those informations with command line like lscpu? Do yoy wanna try?