search

airspeed-velocity / asv_runner

Core pure python benchmarking for asv
https://asv.readthedocs.io/projects/asv-runner/en/latest/index.html
BSD 3-Clause "New" or "Revised" License
5 stars 3 forks source link

ENH: More skip conditionals #16

Closed HaoZeke closed 1 year ago

HaoZeke commented 1 year ago 
          Thank you so much for promptly working on this @HaoZeke!

Now also skips conditionally (only some parameter sets):

Yes, the interface looks good to me! :)

As a follow-up or in this PR directly, do you think it makes sense to have additional decorators called:

skip_params_if(...)
skip_benchmark_if(...)

just like pytest.mark.skipif? https://docs.pytest.org/en/7.1.x/reference/reference.html?highlight=skipif#pytest-mark-skipif

_Originally posted by @akihironitta in https://github.com/airspeed-velocity/asv_runner/pull/13#pullrequestreview-1523122575_

akihironitta commented 1 year ago

@HaoZeke Thanks for reposting this. I might be able to contribute this feature over the weekend.

Let's leave further improvements in this until a real need arises.

@mattip I have a use case where we only want to enable or disable "cuda" parameters only when torch.cuda.is_available() == True. This is something similar to what we've done in the PyTorch Lightning tests using pytest where we want to run certain test cases only on a specific accelerator like GPU, TPU, IPU, etc. (e.g. https://github.com/Lightning-AI/lightning/blob/b16c35d6730bd6b1b178ec3e02a85f578441e670/tests/tests_pytorch/accelerators/test_gpu.py#L26), and I'm sure it's also valuable to many projects (e.g. https://github.com/dmlc/dgl/blob/e6e54304198334ceb0a898d91eab8982145ce5fd/benchmarks/benchmarks/utils.py#L533)