Refactor streamer file to better manage worker-master interaction.
Proposed solution
Rename streamer/mpi_wofls_convert.sh file to streamer/mpi_cog_convert.sh.
Update streamer/mpi_cog_convert.sh to take parameters as an user argument rather than hard-coded values.
Add tagged MPI send()/receive() mode to synchronise a large number of independent tasks when there are more tasks than processors, especially when the run times vary for each task.
Update logging to timestamp each task activity.
Update README.md as per the latest changes.
[x] Tested following scenarios:
1) Attempt COG conversion with no netCDF files in the source path
2) Start MPI run with 0 processes (Num_nodes = 1, MEM=31GB) No netCDF file in the source path
3) Start MPI run with 0 processes (Num_nodes = 1, MEM=31GB) 1 netCDF file in the source path
4) Attempt COG conversion with single netCDF file in the source path
5) Attempt double COG conversion on the same netCDF file that is already processed
6) Attempt COG conversion with 16 netCDF files in the source path (distribute 2 files per qsub job)
Request for this pull request
Refactor streamer file to better manage worker-master interaction.
Proposed solution
streamer/mpi_wofls_convert.shfile tostreamer/mpi_cog_convert.sh.streamer/mpi_cog_convert.shto take parameters as an user argument rather than hard-coded values.send()/receive()mode to synchronise a large number of independent tasks when there are more tasks than processors, especially when the run times vary for each task.README.mdas per the latest changes.COGconversion with nonetCDFfiles in the source path 2) StartMPIrun with 0 processes (Num_nodes = 1, MEM=31GB) NonetCDFfile in the source path 3) StartMPIrun with 0 processes (Num_nodes = 1, MEM=31GB) 1netCDFfile in the source path 4) AttemptCOGconversion with singlenetCDFfile in the source path 5) Attempt doubleCOGconversion on the samenetCDFfile that is already processed 6) AttemptCOGconversion with 16netCDFfiles in the source path (distribute 2 files perqsubjob)