Cray-mpich compatibility

hel-astro-lab / runko

Modern C++/python CPU/GPU plasma toolbox

https://runko.readthedocs.io

MIT License

48 stars 19 forks source link

Cray-mpich compatibility #25

Closed natj closed 5 years ago

natj commented 5 years ago

Cray-mpich does not conform to MPI standard and has a lower limit for maximum tag than other MPI implementations.

This means that tile tag system needs to be redesigned. Probably by splitting main communicator...

Current implementation is a hack to make it work for now.

natj commented 5 years ago

Note: actually mpich-cray does conform to standard but the maximum tag value is ridiculously small (2^15-1 = 32,767). Cray has 2^21-1 ~3e6 after which tags start to overflow.

natj commented 5 years ago

Tried:

[x] communicator splitting does not help because max tile size is still 512^2
[x] dropping tag cycling values and adding mpi barriers to code
- loop evaluation hangs for some weird reason
[x] new tag system where instead of tile id (that can be very large; e.g., 1024^2 = ~1e6) create tag based on cid ordering per each mpi origin/destination rank.

natj commented 5 years ago

Fixed also corgi tile initialization. It no longer sends info message to everybody about number of incoming tiles but instead uses the deterministic nature of the algorithm and each rank computes incoming tile numbers themselves.

natj commented 5 years ago

Closing this - performance improvements are moved to another branch....