Open RussTreadon-NOAA opened 1 month ago
debufr
was run in a batch script on each machine. The batch script loads bufr/11.7.0
. The following debufr
executables were used on each machine
/apps/ops/prod/libs/intel/19.1.3.304/bufr/11.7.0/bin/debufr
/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/gsi-addon-dev-rocky8/install/intel/2021.5.0/bufr-11.7.0-w62mmgj/bin/debufr
/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/gsi-addon-env/install/intel/2021.9.0/bufr-11.7.0-7qdgt6m/bin/debufr
/work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/gsi-addon-env-rocky9/install/intel/2021.9.0/bufr-11.7.0-clawunp/bin/debufr
Repeat the debufr
test on Orion using the Orion bufr/11.7.0
debufr
, Hercules bufr/11.7.0
debufr
, and Orion bufr/12.0.1
debufr
executables. No appreciable change in Orion timings using different debufr
executables.
$bufrfile | orion bufr11.7.0 debufr |
hercules bufr/11.7.0 debufr |
orion bufr/12.0.1 debufr |
---|---|---|---|
gdas.t00z.ahicsr.tm00.bufr_d | 7m16.483s | 7m17.608s | 7m17.237s |
gdas.t00z.1bamua.tm00.bufr_d | 3m48.376s | 3m49.540s | 3m54.525s |
gdas.t00z.gpsro.tm00.bufr_d | 3m46.107s | 3m43.427s | 3m44.194s |
gdas.t00z.ompsn8.tm00.bufr_d | 0m14.483s | 0m14.469s | 0m14.903s |
gdas.t00z.prepbufr | 22m10.262s | 22m5.952s | 22m5.952s |
GSI issue #771 and spack-stack issue #1166 document an increase in
gsi.x
wall time on Orion following the Rocky 9 upgrade. The addition ofdclock
timers to the GSI source code identified the routines which read observations from bufr files as being responsible for the increased wall clock time The total time forgsi.x
to process bufr dump files on Hercules was 168.035 seconds. The Orion run used 608.087 seconds to read the same set of bufr dump files. These timings are from the global GSI ctest.As a second test the
debufr
executable was used to process select gdas bufr dump files for 20240223 00Z. This is the case which global GSI ctest runs. Five bufr files were processed usingtime debufr -c $bufrfile
. Tabulated below arereal time
for each$bufrfile
as a function of machine$bufrfile
Dogwood and Hercules timings are comparable except for
prepbufr
. The Herculesdebufr
time forprepbufr
is 2 minutes greater than Dogwood. Thedebufr
times are a bit higher on Hera than Dogwood. The Oriondebufr
times are about 3x to 4x greater than other machines.It is not clear why bufr processing takes longer on Orion. GSI ctests ran noticeably faster on Orion prior to the Rocky 9 upgrade. I do not have
debufr
timings from Orion prior to the Rocky 9 upgrade.This issue is opened to document this behavior. Hopefully the cause(s) for the increased run time can be identified and a solution developed to bring Orion bufr processing times in line with other machines.