search

BlueBrain / atlas-densities

Tools to compute densities in the context of brain atlases.
Apache License 2.0
3 stars 8 forks source link

Speedup mtype density creation from probability maps using voxcell's ValueToIndexVoxels #79

Closed eleftherioszisis closed 5 months ago

eleftherioszisis commented 6 months ago

Comparison on 70 processes:

Before : 6276s After : 60s Speedup : ~100x

Script used for profiling (Before time was taken from here):

from pathlib import Path
from atlas_densities.app import mtype_densities
from click.testing import CliRunner

OUTPUT_DIR = str(Path(__file__).parent / "out")
DATA_DIR = "/gpfs/bbp.cscs.ch/data/project/proj84/atlas_pipeline_runs/2024-03-11T09:54:30/"
CONFIG_DIR = DATA_DIR + "rules_config_dir/mtypes/"
DENSITY_DIR = DATA_DIR + "inhibitory_neuron_densities_linprog_correctednissl_validated/"

arglist = [
    "--log-output-path",
    OUTPUT_DIR,
    "-v",
    "create-from-probability-map",
    "--hierarchy-path", DATA_DIR + "hierarchy_ccfv2_l23split_barrelsplit.json",
    "--annotation-path", DATA_DIR + "annotation_ccfv2_l23split_barrelsplit_validated.nrrd",
    "--probability-map", CONFIG_DIR + "probability_map_L1.csv",
    "--probability-map", CONFIG_DIR + "probability_map_L23.csv",
    "--probability-map", CONFIG_DIR + "probability_map_L4.csv",
    "--probability-map", CONFIG_DIR + "probability_map_L5.csv",
    "--probability-map", CONFIG_DIR + "probability_map_L6.csv",
    "--probability-map", CONFIG_DIR + "probability_map_TH_INH.csv",
    "--marker", "gad67", DENSITY_DIR + "gad67+_density.nrrd",
    "--marker", "pv", DENSITY_DIR + "pv+_density.nrrd",
    "--marker", "sst", DENSITY_DIR + "sst+_density.nrrd",
    "--marker", "vip", DENSITY_DIR + "vip+_density.nrrd",
    "--synapse-class", "INH",
    "--n-jobs", "70",
    "--output-dir", OUTPUT_DIR,
]
print(" ".join(arglist))

res = CliRunner().invoke(
    mtype_densities.app,
    arglist,
    catch_exceptions=False,
    color=True,
)
codecov-commenter commented 6 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

:exclamation: No coverage uploaded for pull request base (main@fba1e3f). Click here to learn what that means.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #79 +/- ## ======================================= Coverage ? 97.68% ======================================= Files ? 22 Lines ? 1509 Branches ? 0 ======================================= Hits ? 1474 Misses ? 35 Partials ? 0 ``` | [Flag](https://app.codecov.io/gh/BlueBrain/atlas-densities/pull/79/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BlueBrain) | Coverage Δ | | |---|---|---| | [pytest](https://app.codecov.io/gh/BlueBrain/atlas-densities/pull/79/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BlueBrain) | `97.68% <100.00%> (?)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BlueBrain#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

mgeplf commented 5 months ago

I ran the old code path, and the new ones, and all the outputs are exactly the same. The runtime is very much improved: Old: python3 test.py 303289.59s user 825.91s system 5872% cpu 1:26:18.56 total New python3 test.py 2073.11s user 142.77s system 2978% cpu 1:14.40 total