add benchmark failure check

[xspronken]

Add a job that fails the workflow if a benchmark fails.

[hodgestar]

There is a much simpler and better way to achieve this because pytest.main returns the exit code.

def run_benchmarks(args):
    "Run pytest benchmark with sensible defaults."
    pytest.main(

In run_benchmark above, change the first line to return pytest.main( so the exit code is returned.

Then in your own def main function change

    run_benchmarks(other_args)
    benchmark_latest = get_latest_benchmark()
    add_packages_to_json(benchmark_latest)

Save the exit code in a variable, don't save the benchmarks if the exit code is not 0, and return the exit code from your function.

Finally, change the end of your script to:

if __name__ == "__main__":
    sys.exit(main())

[xspronken]

This is indeed much better. I don't think there is a need to only save the benchmarks on exit code 0. They are saved automatically when running:

pytest.main(
        [
            "qutip_benchmark/benchmarks",
            "--benchmark-only",
            "--benchmark-columns=Mean,StdDev,rounds,Iterations",
            "--benchmark-sort=name",
            "--benchmark-autosave",
            "-Wdefault",
        ]
        + args
    )

This means they will be saved regardless of the exit code. I could delete the file if exit code is not 0 but i don't think this is useful, since the entire workflow stops on failure and all the subsequent jobs are cancelled. So regardless of whether the benchmarks are saved or not, if they fail they will not be uploaded to S3, nor will an artefact be created (see here).

[hodgestar]

If they are written already, happy for them to be kept.