Run a benchmark in CI to make sure that we catch changes to the runner.

softdevteam / yksom

Other

8 stars 6 forks source link

Run a benchmark in CI to make sure that we catch changes to the runner. #220

Closed ltratt closed 1 year ago

ltratt commented 1 year ago

https://github.com/SOM-st/SOM/pull/82 changed how the benchmark runner works, but we didn't notice, and confused ourselves! With luck, this addition might spot such problems in the future.

[CC @smarr -- and thanks!]

smarr commented 1 year ago

For at least SOM-java, I do this:

https://github.com/SOM-st/som-java/blob/master/.github/workflows/ci.yml#L39-L50

      - name: Checkout AWFY
        uses: actions/checkout@v3
        with:
          repository: smarr/are-we-fast-yet
          path: are-we-fast-yet

      - name: AWFY Test Run
        run: |
          pwd
          export JAVA_HOME=$JAVA_HOME_8_X64
          pip install ReBench
          rebench awfy-test.conf

https://github.com/SOM-st/som-java/blob/master/awfy-test.conf

Mostly, because I don't really run the AWFY benchmarks in that version else where, and still haven't managed to integrated them into the SOM repo. And because test coverage isn't 100%...

ltratt commented 1 year ago

Interesting! We should perhaps do something a bit like that at some point, but at least this current PR is better than nothing (I hope...).