Closed sfmig closed 2 months ago
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 84.45%. Comparing base (
b5f62ef
) to head (f800053
).
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
Description
What is this PR
Why is this PR needed? We are exploring a systematic way to benchmark
brainglobe
workflows usingasv
.This PR fixes some issues running the
cellfinder
workflow benchmarks (1) on a small GIN dataset and (2) on data available locally.What does this PR do? This PR involves:
setup_cache
function of the benchmarks,To run the benchmarks locally on small dataset from GIN
Checkout this branch to get the latest version of the benchmarks locally.
Create a conda environment and pip install
asv
:Note that to run the benchmarks you do not need to install a development version of
brainglobe-workflows
, sinceasv
will create a separate Python virtual environment to run the benchmarks on it. However, for convenience we do includeasv
as part of thedev
dependencies, so you can use adev
environment to run benchmarks.For a quick check, run one iteration per benchmark with
-v --show-stderr
for a more verbose output.asv
virtual environment thebrainglobe-workflows
package from the tip of the local currently checked out branch, and run the (locally defined) benchmarks on it.To run the benchmarks (locally) on a locally available dataset
Define a config file for the workflow to benchmark. You can use the default one at
brainglobe_workflows/configs/cellfinder.json
for reference.input_data_dir
field pointing to the data of interest.signal
andbackground
subdirectories underinput_data_dir
. However, these defaults can be overwritten with thesignal_subdir
andbackground_subdir
fields.Create and activate an environment with
asv
(follow steps 1 and 2 from above).Run the benchmarks in "quick mode", passing the path to your config file as an environment variable
CONFIG_PATH
. In Unix systems:Troubleshooting
You may find that the conda environment creation is failing because of this issue. This seems to be because
asv
is assuming a conda syntax that changed with the latest release (in conda 24.3.0--force
became--yes
).A PR is on the way, as a temporary workaround you can try from base
conda install -y "conda<24.3"
.References
See issue #9.
Also related is issue #98 which I am currently investigating.
Further context
We currently have
asv
benchmarks for the three main steps involved in thecellfinder
workflow:We also have a benchmark for the full workflow.
We envisioned benchmarks being useful to developers in 3 main ways:
brainglobe_workflows/configs/cellfinder.json
).A reminder of how
asv
works:asv
creates a virtual environment where it installs the package to be benchmarked (in our case,brainglobe-workflows
). This virtual environment is defined in the asv config file (asv.conf.json
).asv
so that the version ofbrainglobe-workflows
that is installed in the asv-managed virtual environment is the one at the tip of the currently checked out branch (i.e., the version atHEAD
). This way developers can check if their local branch introduces regressions. Alternatively, we can choose to install a version ofbrainglobe-workflows
fetched from Github (for example, the tip of the remotemain
branch).asv
will look for benchmarks under thebenchmarks
folder (which is at the same level as theasv.conf.json
file), and run them.How has this PR been tested?
The benchmarks are checked with a CI job, rather than with explicit tests. This follows the general approach in the field - see #96 for more details.
Since we don't plan to test the benchmarks with pytest, I omitted the benchmarks from coverage.
Is this a breaking change?
No.
Does this PR require an update to the documentation?
The README has been updated to better reflect the current status.
Checklist: