search

qutip / qutip-benchmark

Tools for continuously bench-marking qutip's development branch
BSD 3-Clause "New" or "Revised" License
5 stars 8 forks source link

Change group setting in benchmarks #8

Open xspronken opened 2 years ago

xspronken commented 2 years ago

As @hodgestar mentioned in #6 the contents of group should probably be changed. For the benchmarks in the existing PR i would suggest somehting like Qobj-operation and QobjEvo-operation or maybe static and timedependent instead of Qobj and QobjEvo.

hodgestar commented 2 years ago

The information currently recorded is:

  "group": "Add-2-dense-numpy",
  "name": "test_add[2-dense-numpy]",
  "fullname": "test_linear_algebra.py::test_add[2-dense-numpy]",
  "params": {
    "size": 2,
    "density": "dense",
    "dtype": "numpy"
  }
hodgestar commented 2 years ago

For each group me might want to add a parameter or extra_param named "implementation" that specifies the implementation used, e.g. "v5-dense", "julia", "v5-sparse", "v4-sparse", ... . I think the name of the test sort of gives this information, but not explicitly, and combines it with the group information. Have a variable that clearly names the implementation would be useful.

hodgestar commented 2 years ago

pytest-benchmark provides an existing mechanism for setting these things that looks quite nice -- https://pytest-benchmark.readthedocs.io/en/latest/usage.html#markers

hodgestar commented 2 years ago

And https://pytest-benchmark.readthedocs.io/en/latest/usage.html#extra-info

xspronken commented 2 years ago

The groups are currently being changed in #22.

hodgestar commented 2 years ago

I'd like to keep this open or create new issues for the ideas of adding a decorator (mostly stylistic) and an implementation extra_info (which would be useful for organizing results by implementation).