bufr-query does not work when MPI number is set to 1 or 2 for satwind BUFR data

[emilyhcliu]

Test Test BUFR to NetCDF (IODA format) conversion with various MPI numbers (1, 2, 4, 8, 12, 24) for satwind GOES.
Mapping file was used, not python script involved

Results

• The conversion process hangs when setting MPI numbers to 1 or 2; this only occurred when running on HERCULES (not tested on ORION yet)
• The conversion process runs successfully when setting MPI numbers for 4, 8, 12, and 24
• Oddly, the test runs successful for all selected MPI numbers (1, 2, 4, 8, 12, 24) on HERA

Here is the run-time output for MPI = 2

srun -n $nproc --time 00:30:00 ${src_dir}/build/bin/bufr2netcdf.x ${input_file} $yaml ${output_file}
+ srun -n 2 --time 00:30:00 /work/noaa/da/eliu/HERCULES/EMC-bufr-query/bufr-query/build/bin/bufr2netcdf.x /work/noaa/da/eliu/HERCULES/EMC-bufr-query/run_satwind/testinput/2021080100/gdas.t00z.satwnd.tm00.bufr_d /work/noaa/da/eliu/HERCULES/EMC-bufr-query/run_satwind/bufr2ioda_satwind_amv_goes_mapping.yaml '/work/noaa/da/eliu/HERCULES/EMC-bufr-query/run_satwind/testoutput/2021080100/bufr_backend_mpi2/gdas.t00z.satwind_abi_{splits/satId}.tm00.nc'
srun: job 3109233 queued and waiting for resources
srun: job 3109233 has been allocated resources
BufrParser: Parsing file /work/noaa/da/eliu/HERCULES/EMC-bufr-query/run_satwind/testinput/2021080100/gdas.t00z.satwnd.tm00.bufr_d
BufrParser: Parsing file /work/noaa/da/eliu/HERCULES/EMC-bufr-query/run_satwind/testinput/2021080100/gdas.t00z.satwnd.tm00.bufr_d
MPI task: 1 Executing Queries for message 5062 to 10123
MPI task: 0 Executing Queries for message 0 to 5061
MPI task: 0 Building Bufr Data
MPI task: 1 Building Bufr Data

The process was hung until running out of allocated time.

Here is the Mapping file:

bufr:
  subsets:
    - NC005030
    - NC005031
    - NC005032
    - NC005034
    - NC005039

  variables:
    # MetaData
    timestamp:
      datetime:
        year: "*/YEAR"
        month: "*/MNTH"
        day: "*/DAYS"
        hour: "*/HOUR"
        minute: "*/MINU"
        second: "*/SECO"

    latitude:
      query: "*/CLATH"

    longitude:
      query: "*/CLONH"

    satelliteId:
      query: "*/SAID"

    satelliteZenithAngle:
      query: "*/SAZA"

    sensorCentralFrequency:
      query: "*/SCCF"

    pressure:
      query: "*/PRLC[1]"

    # Processing Center
    dataProviderOrigin:
      query: '*/OGCE[1]'

    # Quality Information - Quality Indicator  
    qualityInformationWithoutForecast:
      query: '*/AMVQIC{2}/PCCF'

    # Quality Information - Expected Error  
    expectedError:
      query: '*/AMVQIC{4}/PCCF'

    # Derived Motion Wind (DMW) Intermediate Vectors - Coefficient of Variation
    coefficientOfVariation:
      query: '*/AMVIVR{1}/CVWD'

    # Wind Retrieval Method Information - Computation 
    windComputationMethod:
      query: '*/SWCM'

    # Wind Retrieval Method Information - Hight Assignment 
    windHeightAssignMethod:
      query: '*/EHAM'

    # ObsValue - Wind Direction
    windDirection:
      query: '*/WDIR'

    # ObsValue - Wind Speed
    windSpeed:
      query: '*/WSPD'

  splits:
    satId:
      category:
        variable: satelliteId
        map:
          _270: goes-16
          _271: goes-17
          _272: goes-18

encoder:
  type: netcdf

# dimensions:

  globals:
    - name: "platformCommonName"
      type: string
      value: "Meteorological Operational Satellite"

    - name: "platformLongDescription"
      type: string
      value: "EUMETSAT Polar System in sunsynchronous orbit"

    - name: "source"
      type: string
      value: "MTYP 021-241 IASI 1C RADIANCES (VARIABLE CHNS) (METOP)"

    - name: "sourceFiles"
      type: string
      value: "gdas.t00z.mtiasi.tm00.bufr_d"

    - name: "datetimeReference"
      type: string
      value: "2021-08-01T00:00:00Z"

    - name: "sensor"
      type: string
      value: "IASI"

    - name: "processingLevel"
      type: string
      value: "Level-1C"

    - name: "converter"
      type: string
      value: "BUFR"

  variables:
    # MetaData 
    - name: "MetaData/dateTime"
      source: variables/timestamp
      longName: "Datetime"
      units: "seconds since 1970-01-01T00:00:00Z"

    - name: "MetaData/latitude"
      source: variables/latitude
      longName: "Latitude"
      units: "degrees_north"
      range: [ -90, 90 ]

    - name: "MetaData/longitude"
      source: variables/longitude
      longName: "Longitude"
      units: "degrees_east"
      range: [ -180, 180 ]

    - name: "MetaData/satelliteIdentifier"
      source: variables/satelliteId
      longName: "Satellite Identifier"

    - name: "MetaData/satelliteZenithAngle"
      source: variables/satelliteZenithAngle
      longName: "Satellite Zenith Angle"
      units: "degree"
      range: [ 0, 90 ]

    - name: "MetaData/sensorCentralFrequency"
      source: variables/sensorCentralFrequency
      longName: "Satellite Channel Center Frequency"
      units: "Hz"

    - name: "MetaData/dataProviderOrigin"
      source: variables/dataProviderOrigin
      longName: "Identification of Originating/Generating Center"

    - name: "MetaData/qualityInformationWithoutForecast"
      source: variables/qualityInformationWithoutForecast
      longName: "Quality Information Without Forecast"

    - name: "MetaData/expectedError"
      source: variables/expectedError
      longName: "Expected Error"
      units: "m/s"

    - name: "MetaData/coefficientOfVariation"
      source: variables/coefficientOfVariation
      longName: "Coefficient of Variation"

    - name: "MetaData/windComputationMethod"
      source: variables/windComputationMethod
      longName: "Satellite-derived Wind Computation Method"

    - name: "MetaData/windHeightAssignMethod"
      source: variables/windHeightAssignMethod
      longName: "Wind Height Assignment Method"

    - name: "MetaData/pressure"
      source: variables/pressure
      longName: "Pressure"
      units: "pa"

    - name: "ObsValue/windDirection"
      source: variables/windDirection
      longName: "Wind Direction"
      units: "degree"

    - name: "ObsValue/windSpeed"
      source: variables/windSpeed
      longName: "Wind Direction"
      units: "m/s"

[emilyhcliu]

@rmclaren I found the problem why the bufr-query did not run when setting MPI numbers < 4 on HERCULES. It is a memory problem!! The problem went away after more memory size was allocated.

Setting MPI number = 1

srun -n $nproc --mem 96G --time 00:30:00 ${src_dir}/build/bin/bufr2netcdf.x ${input_file} $yaml ${output_file}
+ srun -n 1 --mem 96G --time 00:30:00 /work/noaa/da/eliu/HERCULES/EMC-bufr-query/bufr-query/build/bin/bufr2netcdf.x /work/noaa/da/eliu/HERCULES/EMC-bufr-query/run_satwind/testinput/2021080100/gdas.t00z.satwnd.tm00.bufr_d /work/noaa/da/eliu/HERCULES/EMC-bufr-query/run_satwind/bufr2ioda_satwind_amv_goes_mapping.yaml '/work/noaa/da/eliu/HERCULES/EMC-bufr-query/run_satwind/testoutput/2021080100/bufr_backend_mpi1/gdas.t00z.satwind_abi_{splits/satId}.tm00.nc'
srun: job 3134183 queued and waiting for resources
srun: job 3134183 has been allocated resources
BufrParser: Parsing file /work/noaa/da/eliu/HERCULES/EMC-bufr-query/run_satwind/testinput/2021080100/gdas.t00z.satwnd.tm00.bufr_d
Executing Queries
Building Bufr Data
Exporting Data
Parser Finished [158.523s]
Category (goes-18) was not found in file.
Encoder Finished [2.628s]
Total Time [166.154s]

Setting MPI number = 2

srun -n $nproc --mem 96G --time 00:30:00 ${src_dir}/build/bin/bufr2netcdf.x ${input_file} $yaml ${output_file}
+ srun -n 2 --mem 96G --time 00:30:00 /work/noaa/da/eliu/HERCULES/EMC-bufr-query/bufr-query/build/bin/bufr2netcdf.x /work/noaa/da/eliu/HERCULES/EMC-bufr-query/run_satwind/testinput/2021080100/gdas.t00z.satwnd.tm00.bufr_d /work/noaa/da/eliu/HERCULES/EMC-bufr-query/run_satwind/bufr2ioda_satwind_amv_goes_mapping.yaml '/work/noaa/da/eliu/HERCULES/EMC-bufr-query/run_satwind/testoutput/2021080100/bufr_backend_mpi2/gdas.t00z.satwind_abi_{splits/satId}.tm00.nc'
srun: job 3134182 queued and waiting for resources
srun: job 3134182 has been allocated resources
BufrParser: Parsing file /work/noaa/da/eliu/HERCULES/EMC-bufr-query/run_satwind/testinput/2021080100/gdas.t00z.satwnd.tm00.bufr_d
BufrParser: Parsing file /work/noaa/da/eliu/HERCULES/EMC-bufr-query/run_satwind/testinput/2021080100/gdas.t00z.satwnd.tm00.bufr_d
MPI task: 0 Executing Queries for message 0 to 5061
MPI task: 1 Executing Queries for message 5062 to 10123
MPI task: 0 Building Bufr Data
MPI task: 1 Building Bufr Data
MPI task: 0 Exporting Data
MPI task: 0 Parser Finished [78.092s]
MPI task: 1 Exporting Data
MPI task: 1 Parser Finished [82.648s]
Category (goes-18) was not found in file.
Encoder Finished [2.572s]
Total Time [88.75s]

[emilyhcliu]

@rmclaren Summary:

• The hanging problem when setting MPI number less than 4 is due to insufficient memory. The intel compiler prints out an error message about insufficient memory on HERA, but not on HERCULES.
• The problem goes away after allocating more memory for srun.

The above, is testing bufr conversion using mapping file only; no Python script is involved. I am testing bufr conversion with mapping and python script together.