search

icecube / flarestack

Unbinned likelihood analysis code for astroparticle physics datasets
https://flarestack.readthedocs.io/en/latest/?badge=latest
MIT License
8 stars 7 forks source link

Proper single CPU processing #381

Closed JannisNe closed 1 month ago

JannisNe commented 1 month ago

Up until now, the MultiProcessor is always called and create multiprocessing processes, even if it is only a single one. Not only does this create extra overhead but also leads to problems sometimes depending on the OS concerning spawn vs fork start up method.

This is especially relevant because when running on the cluster, we typically only use a single CPU. In these cases, I suspect that the forking of a single process does introduce a lot of extra memory usage because when forking the processes the entire memory content is copied!

The quickest way to fix this is this PR: introduce a run_single_process function in multiprocess_wrapper.py. The only caveat is that the naming of the file suggest that the content is only related to multi cpu things which is only a minor issue.

mlincett commented 1 month ago

Can we take care of a bunch of pending dependabot stuff with this? With poetry up it shouldn't be to hard to sneak in some dependency update.

JannisNe commented 1 month ago

I just did poetry update now, and I think that's already getting rid of all the Dependabot PRs. It is not complaining about stuff in pyproject.toml but the versions pinned in poetry.lock I think. We could the PR like this and if any PRs are left we can deal with them separately, no?