search

schneebergerlab / msyd

MIT License
8 stars 0 forks source link

Improve parallelism using Cython OpenMP #8

Open lrauschning opened 1 year ago

lrauschning commented 1 year ago

The parallelization in the reduce_parallel function could be improved by substituting python multiprocessing with the Cytoh OpenMP backend. S. also https://cython.readthedocs.io/en/latest/src/userguide/parallelism.html

mnshgl0110 commented 1 year ago

I will be very careful using this. I am not sure how stable and platform/environment independent this is. It is possible, that in the long run, it might be more time-consuming and annoying to maintain it and to ensure that it works for everyone then to just make multiprocessing (a python standard library function) to work.

lrauschning commented 1 year ago

Hm, maybe its worth making this a compile-time or runtime flag. Another way to improve performance would be to make the input file reading async, which should be possible without using openmp or anything, and reading the files in is definitely a bottleneck. I think I'll play around with the code in pansyn.pyx a bit next week and see if these speedups can be implemented quickly-ish and cleanly; having pansyri be a bit faster would be neat also for testing, but it's not terrible as it is right now.