[FEAT EXTRACTION] Best way to parallelise multiple feature extractors

AIM-Harvard / pyradiomics

Open-source python package for the extraction of Radiomics features from 2D and 3D images and binary masks. Support: https://discourse.slicer.org/c/community/radiomics

http://pyradiomics.readthedocs.io/

BSD 3-Clause "New" or "Revised" License

1.11k stars 485 forks source link

[FEAT EXTRACTION] Best way to parallelise multiple feature extractors #816

Closed dorkylever closed 1 year ago

dorkylever commented 1 year ago

Hi Devs,

Is there a good way to parallelise multiple feature extractors? I have a set of uCT scans with multiple organs labelled.

Something like the Joblib library? I would rather not use job lock files as I already parallelise this way (so I can run multiple specimens in parallel).

Let me know your opinions.

Kind regards, Kyle

JoostJM commented 1 year ago

Parallelization is already implemented in de commandline entrypoints. This allows to run multiple extractions in parallel (parallelization on the case level). See documentation (TLDR: add parameter --jobs <n_threads>).

dorkylever commented 1 year ago

Hi Joost,

Sorry, what I meant was playing around with the python:

extractor = featureextractor.RadiomicsFeatureExtractor()

and playing around with multiprocessing inside the extractor. Is there no option for this? I can't find any in the docs.

I'm using high performance computing and the library I'm using is heavily Python based, so I would rather load the pyradiomics python package than install the command line (I have to create a TCL module for it) and subprocess it.