search

gutmann / coarray_icar

Testing implementation of CoArrays for the basic ICAR algorithms
MIT License
5 stars 6 forks source link

Address load imbalance #22

Open bcfriesen opened 6 years ago

bcfriesen commented 6 years ago

Preliminary performance analysis of the coarray version of ICAR indicates that the algorithm exhibits significant load imbalance. We believe this is due to lopsided expense of evaluating a few physics kernels in mountain ranges on the grid versus other parts of the grid which do not feature mountains. (Ethan, can you comment on this to verify that I stated the problem correctly?)

Achieving good code performance and scalability requires that we address this load imbalance. One approach would be to partition the grid in an asymmetrical way such that the regions which require expensive kernels to run are distributed evenly among images, rather than being concentrated on a few images. Other approaches may also be possible

gutmann commented 6 years ago

Yes, though I am not sure how imbalanced the load will be in a real situation (in which cloud processes could be active in almost any grid cell in the domain). I would be interested in implementing an asymmetric and adaptive grid, but this seems like a very large challenge with limited payoff (for real world simulations).

Perhaps we need to make real world simulations more of a priority to be able to test this first?

bcfriesen commented 6 years ago

Yes load imbalance is a huge problem for structured-grid codes. It's definitely worth the effort figuring out how adapt the grids to distribute the load - that alone can (and usually does) lead to much larger code speedups than other types of optimization. It may take a lot of work to implement something like that, but I think the payoff would be worth it.