Save benchmark + Haddock artifacts during CI

[xsebek]

It would be nice to mark the benchmark logs as artifacts.

We could then write a script that would use GitHub CLI tool to download the artifacts of main branch and plot the benchmarks.

[byorgey]

Since I didn't know what this meant, here's a link for the benefit of others who don't know either: https://docs.github.com/en/actions/advanced-guides/storing-workflow-data-as-artifacts

[byorgey]

Marking the benchmark logs as artifacts sounds fine to me. But as for plotting them, couldn't we just add some commands to the github action to plot the benchmarks right after generating them, and save the plot(s) as artifacts too?

[xsebek]

If we install gh (may also need secrets) and some plotter in the container, then probably yes.

If the actions do not measure performance well, then at least we would have a script to do it locally 🙂

The only other useful artefact I can think of is the generated documentation, which could be cool to show off. 🤓

[xsebek]

As a note, we do not run benchmarks in the CI. We could add it if we wanted to. For local comparison, criterion-compare is quite nice:

cabal run benchmark -- --csv "main.csv" --output "main.html"
git switch feature
cabal run benchmark -- --csv "feature.csv" --output "feature.html"
criterion-compare main.csv feature.csv

^{For example see #415.}

[xsebek]

We do run haddock in the CI so it is only a matter of getting it from there to GitHub pages.

For example, Cabal seems to have been using GitHub pages and automatically pushing the Haddock documentation there:

• Cabal Issue, Commit, Repo, Website with Haddock

But that is outdated now, so we should check if there is an easy way for Github Actions.

Worst case scenario we add a secret to allow pushing to a new swarm-website repository and manually do the clone/push after Haddock is built. This at least we could copy-paste from the linked commit.

[byorgey]

Examples of pushing built Haddocks somewhere during CI:

• https://github.com/agda/agda/pull/5115 does this for Agda using GitHub actions.
• https://github.com/gshen42/HasChor/blob/main/.github/workflows/doc.yml
• https://sjoerdvisscher.github.io/proarrow/ from https://github.com/sjoerdvisscher/proarrow

See also https://github.com/haskell/actions .

[xsebek]

There is https://github.com/benchmark-action/github-action-benchmark which can be passed results from any benchmark tool in JSON containing a list of benchmarks with names and values. It creates nice HTML graphs on GitHub pages and posts alerts on the commit page if it really degrades performance.

The history is stored using actions/cache which we already use.

I think we should make this a separate GitHub action running only on the main branch. Not sure what needs to be done to create GitHub pages, but it should be relatively easy.

[xsebek]

The GitHub runners are often under different loads, so we might want to run the benchmark and a comparison benchmark:

• https://labs.quansight.org/blog/2021/08/github-actions-benchmarks

I think we could publish a benchmark binary in the release and use it as a baseline - run A, B, A, B take averages and compare.

---

Another interesting thing to consider is which compiler to use, as the numbers change quite a lot. (See #752)

[xsebek]

But to keep this issue reasonable, let's just set up GitHub pages and an Action that will publish haddock and benchmark. 😅

By using the benchmarks as is, we might be able to use the same format as Rust criterion benchmarks. 🤔