search

GeoscienceAustralia / dea-cogger

Convert NetCDFs to Cloud Optimized GeoTIFFs
15 stars 4 forks source link

parallel with mpi #7

Closed emmaai closed 6 years ago

emmaai commented 6 years ago

add src_template so it will deal with different format of file names; add a new command so that it can be parallelized with mpi.

omad commented 6 years ago

This sounds cool Emma! Couple of questions:

  1. What do I need to do to run this? mpiexec python3 streamer.py mpi_convert_cog ... ??

  2. To use mpi4py I made a private virtualenv based off the NCI's mpi4py and python3 modules. Is that what you had to do too?

  3. Chunking things up then indexing into the chunk based on rank surprised me, it'll still have some inefficiencies if there are a combination of fast and slow tasks. I'd planned on using MPIExecutor instead. But I guess it worked okay for the run you did?

emmaai commented 6 years ago
  1. a script of cog converting of wofls answers the question
  2. I compiled the wheel of mpi4py with openmpi-3.1.0 and python3.6 provided by NCI
  3. Using spawn is to tune the dynamic of splitting the job in the future, tbh, it's working the same as mpirun -n $NCPUS currently.