Add parallel parsing with MPI

[rmclaren]

Now able to read BUFR files in parallel via MPI. This is supported through bufr2netcdf.x (just run it as mpi process), and also via the python interface (please see the updated docs).

Adds Dependency

• MPI

Examples

• bufr2netcdf.x: mpirun -n 4 bufr2netcdf.x ./test/testinput/data/gdas.t12z.mtiasi.tm00.bufr_d ./test/testinput/bufrtest_mtiasi_mapping.yaml temporary_{splits/satId}.nc

• python

  import sys
  import bufr
  from bufr.encoders import netcdf
  
  def mpi_example():
    DATA_PATH = 'testinput/data/gdas.t18z.1bmhs.tm00.bufr_d'
    YAML_PATH = 'testinput/bufrtest_mhs_basic_mapping.yaml'
    OUTPUT_PATH = 'testrun/mhs_basic_parallel.nc'
  
    bufr.mpi.App(sys.argv)  # Initialize the MPI application (not needed if ioda script)
    comm = bufr.mpi.Comm("world")  # Get the MPI communicator
    container = bufr.Parser(DATA_PATH, YAML_PATH).parse(comm)  # Parallel parse the BUFR file
    container.gather(comm)  # (OPTIONAL) Gather the DataContainer data from all the ranks
  
    if comm.rank() == 0:
        netcdf.Encoder(YAML_PATH).encode(container, OUTPUT_PATH) # Encode the DataContainer object

[emilyhcliu]

@rmclaren I built the bufr-query with this feature branch. Will test it with ATMS and IASI, check the timing, and report back to here.

[rmclaren]

@emilyhcliu Thanks! Now I am nervious :)

[emilyhcliu]

@rmclaren Here are my MPI test results for IASI and CrIS using bufr2netcdf.x

The processing time for reading BUFR:
IASI - 1 processor is ~280 seconds
IASI - 4 processors is ~ 68 second

CrIS - 1 processor is ~343 seconds
CrIS - 4 processors is ~ 84 seconds

However, the CrIS MPI processing has problems in the "gathering" part at rank 0.  
The receiving data size is less than the buffer size (see messages below).  

IASI with single processor

mpirun -n 1 ${src_dir}/build/bin/bufr2netcdf.x ${input_file} $yaml ${output_file}
+ mpirun -n 1 /work/noaa/da/eliu/EMC-bufr-query-mpi/bufr-query/build/bin/bufr2netcdf.x /work/noaa/da/eliu/EMC-bufr-query-mpi/run_iasi/testinput/2021080100/gdas.t00z.mtiasi.tm00.bufr_d /work/noaa/da/eliu/EMC-bufr-query-mpi/run_iasi/bufr2ioda_mtiasi_mapping.yaml '/work/noaa/da/eliu/EMC-bufr-query-mpi/run_iasi/testoutput/2021080100/bufr_backend/gdas.t00z.mtiasi_{splits/satId}.tm00.nc'
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
BufrParser: Parsing file /work/noaa/da/eliu/EMC-bufr-query-mpi/run_iasi/testinput/2021080100/gdas.t00z.mtiasi.tm00.bufr_d
Executing Queries
Building Bufr Data
Exporting Data
Finished [277.964s]

IASI with 4 processors

mpirun -n 4 ${src_dir}/build/bin/bufr2netcdf.x ${input_file} $yaml ${output_file}
+ mpirun -n 4 /work/noaa/da/eliu/EMC-bufr-query-mpi/bufr-query/build/bin/bufr2netcdf.x /work/noaa/da/eliu/EMC-bufr-query-mpi/run_iasi/testinput/2021080100/gdas.t00z.mtiasi.tm00.bufr_d /work/noaa/da/eliu/EMC-bufr-query-mpi/run_iasi/bufr2ioda_mtiasi_mapping.yaml '/work/noaa/da/eliu/EMC-bufr-query-mpi/run_iasi/testoutput/2021080100/bufr_backend/gdas.t00z.mtiasi_{splits/satId}.tm00.nc'
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
BufrParser: Parsing file /work/noaa/da/eliu/EMC-bufr-query-mpi/run_iasi/testinput/2021080100/gdas.t00z.mtiasi.tm00.bufr_d
BufrParser: Parsing file /work/noaa/da/eliu/EMC-bufr-query-mpi/run_iasi/testinput/2021080100/gdas.t00z.mtiasi.tm00.bufr_d
BufrParser: Parsing file /work/noaa/da/eliu/EMC-bufr-query-mpi/run_iasi/testinput/2021080100/gdas.t00z.mtiasi.tm00.bufr_d
BufrParser: Parsing file /work/noaa/da/eliu/EMC-bufr-query-mpi/run_iasi/testinput/2021080100/gdas.t00z.mtiasi.tm00.bufr_d
MPI task: 0 Executing Queries for message 0 to 8104
MPI task: 3 Executing Queries for message 24315 to 32419
MPI task: 1 Executing Queries for message 8105 to 16209
MPI task: 2 Executing Queries for message 16210 to 24314
MPI task: 0 Building Bufr Data
MPI task: 2 Building Bufr Data
MPI task: 3 Building Bufr Data
MPI task: 1 Building Bufr Data
MPI task: 0 Exporting Data
MPI task: 2 Exporting Data
MPI task: 3 Exporting Data
MPI task: 1 Exporting Data
MPI task: 2 Finished [68.162s]
MPI task: 0 Finished [68.643s]
MPI task: 3 Finished [68.918s]
MPI task: 1 Finished [69.26s]

CrIS with single processor

mpirun -n 1 ${src_dir}/build/bin/bufr2netcdf.x ${input_file} $yaml ${output_file}
+ mpirun -n 1 /work/noaa/da/eliu/EMC-bufr-query-mpi/bufr-query/build/bin/bufr2netcdf.x /work/noaa/da/eliu/EMC-bufr-query-mpi/run_cris/testinput/2021080100/gdas.t00z.crisf4.tm00.bufr_d /work/noaa/da/eliu/EMC-bufr-query-mpi/run_cris/bufr2ioda_crisf4_mapping.yaml '/work/noaa/da/eliu/EMC-bufr-query-mpi/run_cris/testoutput/2021080100/bufr_backend/gdas.t00z.crisf4_{splits/satId}.tm00.nc'
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
BufrParser: Parsing file /work/noaa/da/eliu/EMC-bufr-query-mpi/run_cris/testinput/2021080100/gdas.t00z.crisf4.tm00.bufr_d
Executing Queries
Building Bufr Data
Exporting Data
Finished [343.05s]

CrIS with 4 processors

mpirun -n 4 ${src_dir}/build/bin/bufr2netcdf.x ${input_file} $yaml ${output_file}
+ mpirun -n 4 /work/noaa/da/eliu/EMC-bufr-query-mpi/bufr-query/build/bin/bufr2netcdf.x /work/noaa/da/eliu/EMC-bufr-query-mpi/run_cris/testinput/2021080100/gdas.t00z.crisf4.tm00.bufr_d /work/noaa/da/eliu/EMC-bufr-query-mpi/run_cris/bufr2ioda_crisf4_mapping.yaml '/work/noaa/da/eliu/EMC-bufr-query-mpi/run_cris/testoutput/2021080100/bufr_backend/gdas.t00z.crisf4_{splits/satId}.tm00.nc'
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
BufrParser: Parsing file /work/noaa/da/eliu/EMC-bufr-query-mpi/run_cris/testinput/2021080100/gdas.t00z.crisf4.tm00.bufr_d
BufrParser: Parsing file /work/noaa/da/eliu/EMC-bufr-query-mpi/run_cris/testinput/2021080100/gdas.t00z.crisf4.tm00.bufr_d
BufrParser: Parsing file /work/noaa/da/eliu/EMC-bufr-query-mpi/run_cris/testinput/2021080100/gdas.t00z.crisf4.tm00.bufr_d
BufrParser: Parsing file /work/noaa/da/eliu/EMC-bufr-query-mpi/run_cris/testinput/2021080100/gdas.t00z.crisf4.tm00.bufr_d
MPI task: 2 Executing Queries for message 3297 to 4944
MPI task: 0 Executing Queries for message 0 to 1648
MPI task: 1 Executing Queries for message 1649 to 3296
MPI task: 3 Executing Queries for message 4945 to 6592
MPI task: 2 Building Bufr Data
MPI task: 0 Building Bufr Data
MPI task: 3 Building Bufr Data
MPI task: 1 Building Bufr Data
MPI task: 2 Exporting Data
MPI task: 0 Exporting Data
MPI task: 3 Exporting Data
MPI task: 2 Finished [83.773s]
MPI task: 0 Finished [84.216s]
MPI task: 3 Finished [84.329s]
MPI task: 1 Exporting Data
MPI task: 1 Finished [87.826s]
Abort(1009870350) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Gatherv: Message truncated, error stack:
PMPI_Gatherv(401)..........................: MPI_Gatherv failed(sbuf=0x1271526b0, scount=667239, MPI_CHAR, rbuf=0x1295adc20, rcnts=0x4991ebf0, displs=0x4991b3b0, datatype=MPI_CHAR, root=0, comm=MPI_COMM_WORLD) failed
MPIR_Gatherv(554)..........................: 
MPIDI_Gatherv_intra_composition_alpha(2658): 
MPIDI_NM_mpi_gatherv(397)..................: 
MPIR_Gatherv_intra_auto(39)................: 
MPIR_Gatherv_allcomm_linear(86)............: 
MPIR_Localcopy(44).........................: Message truncated; 667239 bytes received but buffer size is 222413

[rmclaren]

@emilyhcliu Would it be possible to copy the CRIS data you ran (data and configs) to the HERA HPC?

[rmclaren]

I improved the printing of time stamp information. It now prints time to parse, time to encode, and the total time elapsed.

[emilyhcliu]

@emilyhcliu Would it be possible to copy the CRIS data you ran (data and configs) to the HERA HPC?

@rmclaren Yes, I will do that now. I will let you know when I have the files on HERA in a few minutes.

[emilyhcliu]

@rmclaren Here is my EMC-bufr-query (with your feature/in_parallel branch) on HERA:

/scratch1/NCEPDEV/da/Emily.Liu/EMC-bufr-query-mpi/bufr-query

The input files, mapping files, and the associate scripts are in the following directory:

/scratch1/NCEPDEV/da/Emily.Liu/EMC-bufr-query-mpi/run_cris

bufr2netcdf_bufr.sh is the script submitted to SLURM to do mpi run

sbatch bufr2netcdf_bufr.sh

I just submitted the script and here is the output: SLURM_bufr2netcdf.o63173267 The run reproduced the error at the MPI gathering step I described above.
Please let me know if you need more information.

The MPI run for IASI was successful.
You can find the input, configuration and script files under ./run_iasi

[rmclaren]

@emilyhcliu Thanks!

[rmclaren]

@emilyhcliu So the problem happened while gathering string fields, which are a little more complicated than the numerical fields. I believe I fixed the problem.

[emilyhcliu]

@rmclaren Thanks for making the MPI work correctly.
The satwind tests looked good (see Issue #14) I will repeat tests for IASI and CrIS again and document the results here.

@nicholasesposito You have the high-level API converter worked for aircraft. Could you please test it with MPI?
@PraveenKumar-NOAA Could you also test the ADPUPA converter with MPI?

I can show you examples to test the MPI during JEDI Junction.

[emilyhcliu]

@rmclaren Can I test bufr-query MPI in JEDI using feature/bufr_in_parallel from the IODA repository?

[PraveenKumar-NOAA]

@emilyhcliu @rmclaren built bufr-query and ioda-bundle/ioda using their corresponding feature/parallel branches on Hercules and successfully created the output file for the ADPUPA prepBUFR data.

It seems that MPI is taking a bit more time to run the job. Should this not be in other way? Without MPI: Runtime: 1.89 sec MPI: Runtime: 1.91 sec

[rmclaren]

@PraveenKumar-NOAA How are you running it exactly?

[PraveenKumar-NOAA]

@rmclaren thanks, now I ran it correctly using 'srun -n ..' I created a plot for the number of nodes vs time for my ADPUPA test.

Chart1.pdf

[nicholasesposito]

I ran aircraft profiles (prepbufr) withs run and mpi and got these results:

No mpi:

• Runtime: 18.17 sec, Memory: total: 813.37 Mb, per task: min = 813.37 Mb, max = 813.37 Mb

With mpi: 1 . Runtime: 17.87 sec, Memory: total: 926.29 Mb, per task: min = 926.29 Mb, max = 926.29 Mb 2 . Runtime: 5.41 sec, Memory: total: 1.17 Gb, per task: min = 312.46 Mb, max = 857.89 Mb 4 . Runtime: 3.51 sec, Memory: total: 1.64 Gb, per task: min = 264.66 Mb, max = 566.26 Mb 6 . Runtime: 3.02 sec, Memory: total: 2.07 Gb, per task: min = 230.46 Mb, max = 460.74 Mb 12 . Runtime: 2.64 sec, Memory: total: 3.40 Gb, per task: min = 225.84 Mb, max = 354.71 Mb

[rmclaren]

Wonder if someone could approve this PR so I can go ahead and merge it?

[nicholasesposito]

@rmclaren I seem to remember the output nc files did not contain all the data due to the merge not working correctly or something like that, but I may be wrong. Was that fixed, and if so, is that in this PR?

[rmclaren]

@nicholasesposito Were you comparing data generated with IODA to data (output files) generated with bufr-query? The output files were not expected to be identical in these cases due to small differences in the way header information is encoded, and the fact that IODA filters out fields with missing data (this is what I recall you ran into, but don't remember excactly). There is also the issue @emilyhcliu reported (#14), but this has been fixed.

[nicholasesposito]

That works. Thanks !

[rmclaren]

Thanks everyone!