Open manodeep opened 7 years ago
mpi4py
sounds like a good way to go. What are you imagining as the input file organization? If the points are all in one file, does the master process decide on a domain decomposition ahead of time so it can divvy out points to CPUs via MPI?
Not sure yet. Seems to me that the galaxies might be fine to read/store on all tasks, but during the computation itself, the randoms need to be on a larger domain.
Perhaps figure out the total number of galaxy tiles per task, and then assign the galaxy tile-ids per task (hopefully, in some contiguous 3d chunks but that might not always be possible). With the assigned tile-ids, we then have a spatial domain per task (i.e., an array of tile-ids per task) over which the randoms are required. Assuming MPI
reads, we can then compute tile ids for each particle and send to appropriate task. Each task then has all the required pieces to compute the pair-counts.
For data-sets that are actually too large to hold on a single node, a
MPI
python script that distributes and collects the pair counts overMPI
usingmpi4py
would be a valuable addition to the repo. Does not need to be part of the formal distribution since that would require a dependency onmpi4py
; however, ifmpi4py
is installed, then may be we can enable this script. (Comes from a discussion about DES and how long theirDR
computations take, plus writing the abstract/intro for the paper).For v2.1/v3 perhaps.
Likely steps:
MPI
), including both the data and randoms (convert from spherical as appropriate)rmax
(the maximum spatial scale to probe) and tile the spatial domain with these cellsMPI
(if the spatial domain and the number of tasks and the number of tasks are fixed, then the randoms can be preemptively sub-divided on a per-task basis)
@lgarrison - what do you think?