Best practices for use in CI

[danieleades]

This crate is amazing! It does exactly what I'm looking for.

I'm wondering if you have any pointers for best practices for using this is CI. Specifically in a github workflow.

my requirements would be

• run the benchmarks on pull requests
• fail the workflow if significant regression from main/master

how best to achieve that?

[danieleades]

if it helps, i'm using the setup described in #6 (thanks @HEnquist)

[mimoo]

I'm trying to make the following githb action work atm:

name: Benchmarks

on:
  push:
    branches:
    - main
  pull_request:

jobs:
  run_checks:
    runs-on: ubuntu-latest
    name: Run some basic checks
    steps:

      - name: Set up cargo/rust and valgrind (for iai)
        run: |
          rustup component add rustfmt 
          sudo apt install -y valgrind

      - name: Checkout master
        uses: actions/checkout@v2
        with:
          ref: master

      - name: Run baseline bench
        run: |
          cargo bench --bench proof_criterion -- --save-baseline master
          cargo bench --bench proof_iai > iai_baseline

      - name: Checkout PR
        uses: actions/checkout@v2

      - name: Run bench against baseline
        run: |
          cargo bench -q --bench proof_criterion -- --baseline master > bench_result
          cargo bench --bench proof_iai > iai_feature

      - name: Write result in PR
        uses: actions/github-script@v5
        with:
          script: |
            const fs = require('fs');

            // read the output file
            const criterion_result = fs.readFileSync("bench_result", {encoding:'utf8', flag:'r'});
            const iai_baseline = fs.readFileSync("iai_baseline", {encoding:'utf8', flag:'r'});
            const iai_feature = fs.readFileSync("iai_feature", {encoding:'utf8', flag:'r'});

            // form message
            const message = '👋 criterion benchmark result:\n' + criterion_result + '\n\n iai baseline result:\n' + iai_baseline + '\n\n iai feature result:\n' + iai_feature;

            // post comment
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: message
            })

[danieleades]

@mimoo which bit's not working?

You should turn that into a generic GitHub action when you get it working.

Would be good to hear from @bheisler on this!

[mimoo]

It is taking ages to run iai and debug this, but I guess this is expected :o also, I'm trying to figure out how I can have a clean text output without using a json formatted output (the -q option doesn't seem to remove some meta stuff)

[hanabi1224]

fail the workflow if significant regression from

I think this is only possible when the baseline and feature reports are generated within a single github action job. As described in the doc, github action uses Azure DSv2-series VM which has a pool of CPU models released from 2013-2019 whose performance difference is non-trivial. It's not possible to fix a job to a certain CPU model and a diff between results generated on a 2013 CPU and a 2019 CPU is invalid, thus any setup that uses historical data generated from a different job is invalid.

GitHub hosts Linux and Windows runners on Standard_DS2_v2 virtual machines in Microsoft Azure with the GitHub Actions runner application installed.

DSv2-series sizes run on Intel® Xeon® Platinum 8272CL (Cascade Lake), Intel® Xeon® 8171M 2.1GHz (Skylake) or the the Intel® Xeon® E5-2673 v4 2.3 GHz (Broadwell) or the Intel® Xeon® E5-2673 v3 2.4 GHz (Haswell) processors with Intel Turbo Boost Technology 2.0 and use premium storage.

[Mathieu-Lala]

It would be great to see iai be integrated to this github action https://github.com/benchmark-action/github-action-benchmark.

See the already opened issue

[sigaloid]

Here's my take on @mimoo 's existing code:

name: Commit Benchmark

on:
  push:
    branches:
    - main

jobs:
  run_checks:
    runs-on: ubuntu-latest
    name: Run benchmarking
    steps:
      - name: Install nightly toolchain
        uses: dtolnay/rust-toolchain@stable
        with:
          toolchain: stable

      - name: Set up cargo/rust and valgrind (for iai)
        run: |
          rustup component add rustfmt 
          sudo apt install -y valgrind

      - name: Checkout previous commit
        uses: actions/checkout@v3
        with:
          fetch-depth: 2
      - run: git checkout HEAD^

      - name: Run previous bench
        run: |
          cargo bench --bench my_benchmark | tee previous
          readlink -f previous

      - name: Checkout main
        uses: actions/checkout@v3
        with:
          ref: ${{ github.sha }}
          clean: 'false'

      - name: Run baseline bench
        run: |
          cargo bench --bench my_benchmark | tee current
          readlink -f current

      - name: Write result in PR
        uses: actions/github-script@v5
        with:
          script: |
            const fs = require('fs');

            // read the output file
            const previous = fs.readFileSync("/full-path/previous", {encoding:'utf8', flag:'r'});
            const current = fs.readFileSync("/full-path/current", {encoding:'utf8', flag:'r'});

            // form message
            const message = '## Benchmarks\n---\n Current:\n```\n' + current + '\n```\n Previous:\n\n```\n' + previous + '\n```';

            // post comment
            github.rest.repos.createCommitComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              commit_sha: "${{ github.sha }}",
              body: message
            })

This will run and comment on every git commit to main (not on PR's as OP requested but can easily be tweaked). It does require some set up - run it once and check the output logs on the action for the full path of the file (mine was /home/runner/work/REPO_NAME/REPO_NAME/FILE_NAME) and replace it in the script section.

It will run benchmarks on the previous commit then compare it to the current commit, which resolves the worries about differing machines picking up your actions. At least within one comment it will be the same action.

[epompeii]

@Mathieu-Lala if you want to easily run, track, and catch performance regressions with Iai, Bencher supports this out of the box: https://github.com/bencherdev/bencher/tree/main#supported-benchmark-harnesses

Here's an example of how to use Bencher with GitHub Actions: https://bencher.dev/docs/how-to/github-actions