search

mikarubi / voluseg

pipeline for volumetric cell segmentation
MIT License
11 stars 9 forks source link

Repeated imports #20

Open luiztauffer opened 2 months ago

luiztauffer commented 2 months ago

We can avoid repeated imports throughout the scripts by moving them to the top of the script. There's one specific import that I'm concerned about though, them numpy after setting local ENV vars:

https://github.com/mikarubi/voluseg/blob/c2cf159ce371b0cf62299515ebdb24ca788189f2/voluseg/_steps/step4a.py#L7-L12

This happens multiple times across the project. Is this necessary because of pyspark opening multiple threads, or something similar? Any way we could avoid that?

mikarubi commented 2 months ago

This is a problem with numpy multithreading, rather than spark. After numpy is imported, one cannot change the number of requested threads, which leads to problems during spark parallelism.

It should work if we move these exact blocks to the top of the script. The key is to always set the environment variables before numpy is imported. Also, we may need to watch out for other modules that import numpy indirectly. So, in general, this change has to be made carefully, and we need to test the performance to see whether multithreading appears during parallelism.