ASV Benchmarking Pull Request Workflow

[philipc2]

Overview

Introduces a Github Actions Workflow for running ASV Benchmarks on Pull Requests, getting compared to the main branch.

The workflow is triggered using the run-benchmark label.

By default, the workflow is skipped unless it is queued using the label above.

The workflow can be re-queued by removing and adding the label.

Once the workflow is complete, the results are posted (or updated) by a Github Bot.

New benchmarks, including those that test new functionality, can be run using this bot as well. They will fail for older versions that raise an error, but that is expected.

Example screenshot of the output:

[philipc2]

An example of this workflow in action can be seen in this dummy PR on my fork.

https://github.com/philipc2/uxarray/pull/2

[github-actions[bot]]

ASV Benchmarking

Benchmark Comparison Results

Benchmarks that have improved: | Change | Before [3b27d3f3] | After [7bd5fec0] | Ratio | Benchmark (Parameter) | |----------|----------------------|---------------------|---------|-------------------------------------------------| | - | 388M | 301M | 0.78 | mpas_ocean.Integrate.peakmem_integrate('480km') | Benchmarks that have stayed the same: | Change | Before [3b27d3f3] | After [7bd5fec0] | Ratio | Benchmark (Parameter) | |----------|----------------------|---------------------|---------|---------------------------------------------------------------------------------------------------------------------------------------| | | 240M | 240M | 1 | face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc')) | | | 300M | 300M | 1 | face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc')) | | | 318M | 305M | 0.96 | face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc')) | | | 316M | 301M | 0.95 | face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc')) | | | 13.9±0.02s | 14.0±0.09s | 1.01 | face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc')) | | | 2.06±0.01s | 2.09±0s | 1.01 | face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc')) | | | 19.2±0.02s | 19.8±0.01s | 1.03 | face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc')) | | | 72.2±0.4ms | 74.1±2ms | 1.03 | face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc')) | | | 1.66±0.02s | 1.66±0.01s | 1 | import.Imports.timeraw_import_uxarray | | | 5.70±0.05μs | 5.72±0.05μs | 1 | mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('120km') | | | 5.24±0.01μs | 5.31±0.02μs | 1.01 | mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('480km') | | | 367M | 369M | 1.01 | mpas_ocean.GeoDataFrame.peakmem_to_geodataframe('120km', False) | | | 346M | 349M | 1.01 | mpas_ocean.GeoDataFrame.peakmem_to_geodataframe('120km', True) | | | 320M | 320M | 1 | mpas_ocean.GeoDataFrame.peakmem_to_geodataframe('480km', False) | | | 319M | 319M | 1 | mpas_ocean.GeoDataFrame.peakmem_to_geodataframe('480km', True) | | | 1.20±0s | 1.22±0.01s | 1.01 | mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', False) | | | 55.1±0.5ms | 55.5±0.4ms | 1.01 | mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', True) | | | 93.9±0.4ms | 93.6±0.3ms | 1 | mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', False) | | | 5.33±0.07ms | 5.20±0.1ms | 0.98 | mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', True) | | | 253M | 253M | 1 | mpas_ocean.Gradient.peakmem_gradient('120km') | | | 233M | 233M | 1 | mpas_ocean.Gradient.peakmem_gradient('480km') | | | 2.65±0.03ms | 2.66±0.01ms | 1.01 | mpas_ocean.Gradient.time_gradient('120km') | | | 282±0.5μs | 282±0.9μs | 1 | mpas_ocean.Gradient.time_gradient('480km') | | | 318M | 320M | 1.01 | mpas_ocean.Integrate.peakmem_integrate('120km') | | | 176±1ms | 174±2ms | 0.99 | mpas_ocean.Integrate.time_integrate('120km') | | | 11.8±0.1ms | 11.9±0.1ms | 1.01 | mpas_ocean.Integrate.time_integrate('480km') | | | 229M | 231M | 1.01 | quad_hexagon.QuadHexagon.peakmem_open_dataset | | | 229M | 229M | 1 | quad_hexagon.QuadHexagon.peakmem_open_grid | | | 6.73±0.1ms | 6.63±0.06ms | 0.98 | quad_hexagon.QuadHexagon.time_open_dataset | | | 5.75±0.05ms | 5.70±0.02ms | 0.99 | quad_hexagon.QuadHexagon.time_open_grid |

[philipc2]

This is excellent and very timely for our project. We can tackle some of my comments in another future PR.

The table units of before and after and the function calls look a little verbose.

Thanks for the review! Yeah, the table can definitely be trimmed a bit to look cleaner. I didn't do any processing of the data, simply returned the result of the asv compare function call.

I'll definitely look into at least removing the Change column, since that's not being populated at all and is just taking up space.