MPI script for large data-sets

[manodeep]

For data-sets that are actually too large to hold on a single node, a MPI python script that distributes and collects the pair counts over MPI using mpi4py would be a valuable addition to the repo. Does not need to be part of the formal distribution since that would require a dependency onmpi4py; however, if mpi4py is installed, then may be we can enable this script. (Comes from a discussion about DES and how long their DR computations take, plus writing the abstract/intro for the paper).

For v2.1/v3 perhaps.

Likely steps:

• calculate the min-max x/y/z extents of the particles (over MPI), including both the data and randoms (convert from spherical as appropriate)
• create 3-D cells of minimum size as rmax (the maximum spatial scale to probe) and tile the spatial domain with these cells
• for each cpu, assign spatial cell co-ordinates for data and randoms (load-balancing if/as necessary)
• compute the pair-counts on each processor
• sum the results over MPI

(if the spatial domain and the number of tasks and the number of tasks are fixed, then the randoms can be preemptively sub-divided on a per-task basis)

@lgarrison - what do you think?

[lgarrison]

mpi4py sounds like a good way to go. What are you imagining as the input file organization? If the points are all in one file, does the master process decide on a domain decomposition ahead of time so it can divvy out points to CPUs via MPI?

[manodeep]

Not sure yet. Seems to me that the galaxies might be fine to read/store on all tasks, but during the computation itself, the randoms need to be on a larger domain.

Perhaps figure out the total number of galaxy tiles per task, and then assign the galaxy tile-ids per task (hopefully, in some contiguous 3d chunks but that might not always be possible). With the assigned tile-ids, we then have a spatial domain per task (i.e., an array of tile-ids per task) over which the randoms are required. Assuming MPI reads, we can then compute tile ids for each particle and send to appropriate task. Each task then has all the required pieces to compute the pair-counts.