cta-observatory / ctapipe

Low-level data processing pipeline software for CTAO or similar arrays of Imaging Atmospheric Cherenkov Telescopes
https://ctapipe.readthedocs.org
BSD 3-Clause "New" or "Revised" License
64 stars 268 forks source link

Allow overriding number of cores used by reconstructors on the CLI #2307

Closed maxnoe closed 11 months ago

maxnoe commented 1 year ago

Please describe the use case that requires this feature.

Useful for GRID processing / other job systems with resource management.

Describe the solution you'd like

Add a CLI option that allows overriding the number of cores used by reconstructors e.g. SKLearn based reconstructors to the train and apply tools.

This is needed to be able to set different n_cores for training and application.

Describe alternatives you've considered

Additional context

maxnoe commented 1 year ago

I'd propose to add an n_jobs property to the Reconstructor base API. Reconstructors that don't do multiprocessing can just ignore this but reconstructors that are able to use multiple threads must obey this setting.