VirtualPhotonics / VTS

Virtual Tissue Simulator
https://virtualphotonics.org
Other
34 stars 9 forks source link

Monte Carlo paralellization at the photon level #19

Closed hayakawa16 closed 1 year ago

hayakawa16 commented 5 years ago

I would like to explore how we might modify the code so that individual batches of photons can be executed on different processors then added back together for a final tally. I've been reading this: https://www.codeproject.com/Articles/767997/Parallelised-Monte-Carlo-Algorithms-sharp under the section "C# Threadpool" as a possible model to follow.

hayakawa16 commented 4 years ago

There is renewed interest. In the link above, 2 methods are described: 1) Threadpool, 2) Parallel.For. I tried the Parallel.For because it is easy to add around the MonteCarloSimulation.cs for loop that loops over each photon.

I first tried on a linux 18 core, 2 threads per core machine. Without any code modifications N=1e6 photons took 1/2 hour to run. This machine utilizes the threads on its own and typically I see 8-12 threads being used during a simulation. With the code fix the code took 1.7 hours! Not sure why. I thought possibly the threading management was conflicting with the parallelized code somehow.

So I tried the same on a linux computer with 4 cores, no threads. Without the code fix N=1e6 took about an hour. With the code fix, the code compiles but the many unit tests fail.

The Threadpool requires a larger code modification. I will look into this method and continue searching for other options.

dcuccia commented 1 year ago

Based on your last comment, it sounds like we can close this Item. As you discovered, I think the Task Parallel Library (TPL) probably isn't a great fit for "per-photon" parallelization, which is best suited for parallelizing "large" jobs. Seeking this level of performance from parallelization likely requires some significant work. Have we done any profiling of our code to identify the bottlenecks? Wonder if the biggest item is the RNG? Or the double-precision cosine calculations? Or perhaps the heap- and allocation-happy simulation structure? Lots to think about and explore!

hayakawa16 commented 1 year ago

Yes, Lisa helping me with this task and we now have a command line option to run in parallel. Here is our current documentation: https://github.com/VirtualPhotonics/Vts.MonteCarlo/wiki/MCCL-Parallel-Processing

dcuccia commented 1 year ago

Neat! Thanks for sharing the link.