rs-station / laue-dials

A package for analyzing Laue x-ray crystallography data using the DIALS framework.
https://rs-station.github.io/laue-dials/
MIT License
4 stars 3 forks source link

laue.integrate takes longer with increasing nproc #62

Open toastisme opened 1 month ago

toastisme commented 1 month ago

dials=3.17.0 laue-dials=0.4

When running laue.integrate (using e.g the dataset in /n/hekstra_lab/projects/laue-dials-tests/laue_tutorials/simdhfr/data/bandwidth/shot1/shot_1_#####.cbf), running with nproc=1 produces the output:

Image 174 took 23.01715064048767 seconds.
Integrating image 175.
Image 175 took 24.365636825561523 seconds.
Integrating image 176.
Image 176 took 26.35458254814148 seconds.
Integrating image 177.
Image 177 took 25.817972421646118 seconds.
Integrating image 178.
Image 178 took 26.819148778915405 seconds.
Integrating image 179.
Image 179 took 27.399763584136963 seconds.
Integration finished.
Constructing reflection table
Converting to MTZ format.
Saving integrated reflections to laue/integrated_reference.mtz

Time Taken for Total Processing = 3841.383531 seconds

Repeating with nproc=2 produces:

Image 172 took 49.02370619773865 seconds.
Integrating image 174.
Image 173 took 46.737916707992554 seconds.
Integrating image 175.
Image 174 took 46.61181282997131 seconds.
Integrating image 176.
Image 175 took 42.28249502182007 seconds.
Integrating image 177.
Image 176 took 42.314101696014404 seconds.
Integrating image 178.
Image 177 took 41.87906455993652 seconds.
Integrating image 179.
Image 178 took 42.62828612327576 seconds.
Image 179 took 39.019068241119385 seconds.
Integration finished.
Constructing reflection table
Converting to MTZ format.
Saving integrated reflections to laue/integrated_reference.mtz

Time Taken for Total Processing = 3975.674198 seconds

Repeating with nproc=4 causes it to go even slower

Integrating image 177.
Image 174 took 71.01042747497559 seconds.
Integrating image 178.
Integrating image 179.
Image 175 took 75.81021571159363 seconds.
Image 177 took 61.80263161659241 seconds.
Image 178 took 54.61494040489197 seconds.
Image 179 took 45.53638052940369 seconds.

I've not managed to get this to finish with nproc=4, despite running for over 8 hours.

Looking at Top while this is running, it looks like more processes are correctly run with increasing nproc, but the %CPU is divided between them. E.g. nproc=4 has 4 processes each using 25% CPU.

toastisme commented 1 month ago

With increasing nproc, I've now managed to get it to finish by increasing the available memory, so this issue might be resolvable by giving an error if the amount of memory is insufficient.