search

amanchokshi / EMBERS

Experimental Measurement of BEam Responses with Satellites
https://embers.readthedocs.io
MIT License
3 stars 1 forks source link

how parallel is the code #8

Closed teuben closed 3 years ago

teuben commented 3 years ago

The comment

WARNING: This is probably the most resource hungry section. It typically took me 2 days to process 5 months of data, on a machine with 40 cpu cores. Beware, and be patient.

prompted me to wonder if I had 4 cores, would it take 10 times longer? How well does the parallel portion of the code scale? Is there a significant portion serial only? I think knowing this would be useful for users. Alternatively, could the code be made faster by using XLA/dask/numba ?(as food for future thought)

Can the underlying reasons for this powerhungryness be understood in O(n) notation?

amanchokshi commented 3 years ago

I'm not an expert, but the way I've implemented the parallel code should scale linearly. I'd expect a 4 core machine to take approximately 10 times longer than a 40 core one. This step in crucial as it conditions relevant pairs of data files and creates time alined data. This is done with a savgol filter followed by interpolation. A loop over relevant files identifies the correct pairs of data files, the sends each pair to its own core.

I don't think there is any significant serial code which acts as a bottleneck to this process

teuben commented 3 years ago

I'm not an expert yet, though this is on my horizon. However, if a task like this takes so long is probably worthy of inspecting for bottlenecks. Nice project for another time. If you could figure out what the expected processing time is, like an O(N) analysis, that could help/give insight. I've been using the github projects to schedule new ideas. You can also keep them in the issue to reminder you, or volunteers, that this might be an issue.

amanchokshi commented 3 years ago

Will do, thanks Peter! It would be good to do this at some point, and learn more on code optimisation.