search

cb-geo / mpm

CB-Geo High-Performance Material Point Method
https://www.cb-geo.com/research/mpm
Other
247 stars 82 forks source link

Automatic Dynamic Load Balancing #625

Closed bodhinandach closed 4 years ago

bodhinandach commented 4 years ago

Describe the feature At the moment, I see that we are specifying the nload_balance_steps_. Wouldn't it be nice to make it automatic?

Describe alternatives Do MPI_Gather the nparticles() from each rank in rank 0 at the end of each step and judge there, e.g. we can make a limit on how many percents it is different to nparticles()/mpi_size to increase/reduce the automatic balancing frequency.

Additional context This is particularly useful when simulating problems like landslides, where initially the particle moves rapidly, but in the end they are almost static and stop propagating. If we use a constant nload_balance_steps_, we will perform quite useless domain_decomposition at the end.

kks32 commented 4 years ago

The cell->nglobal_particles() is basically a count across MPI ranks, so that would be sufficient to compute the load. We might also need to consider the number of shared nodes. The difficult part is not whether or not to do a load balancing, but for KaHIP to keep most of the partition the same, so the load balancing doesn't have to move a lot of particles. I noticed, between one partitioning to another it does change a lot, so I'm trying to see how to keep most of the partition the same.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

bodhinandach commented 4 years ago

@kks32 I am reopening this. Are we going to take a look any time soon?

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.