use zstd and data quantization in GFS high resolution runs

junwang-noaa commented 2 months ago

Description

The zstandard compression and data quantization are in netcdf 4.9.2. Work needs to be done to test the proper zstandard level and quantization number in GFS high resolution runs used in GFS prototypes.

Solution

Alternatives

Related to

yangfanglin commented 2 months ago

Jun, does zstandard compression provide better compression than what we have been using ?

junwang-noaa commented 2 months ago

@yangfanglin It has about same compression rate as zlib, but it runs 20% faster

junwang-noaa commented 1 month ago

@RatkoVasic-NOAA @ulmononian @AlexanderRichert-NOAA May I ask if the netcdf library in spack-stack 1.6.0 is built with zstd or not on R&D platforms?

AlexanderRichert-NOAA commented 1 month ago

Yes, the 1.6.0 release has netcdf with zstd support.

junwang-noaa commented 1 month ago

Thanks, @AlexanderRichert-NOAA

@ulmononian @RatkoVasic-NOAA would you please confirm that the netcdf in spack-stack 1.6.0 used in UFS weather model was built with zstd?

Hera: /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env-rocky8/install/modulefiles/Core Hercules: /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core jet: /mnt/lfs4/HFIP/hfv3gfs/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env-rocky8/install/modulefiles/Core gaea: /ncrc/proj/epic/spack-stack/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core

Thanks

DusanJovic-NOAA commented 1 month ago

We run several regression tests with zstandard compression:

tests/control_debug_p8-if [[ " hera orion hercules jet " =~ " ${MACHINE_ID} " ]] ; then
tests/control_debug_p8:  ZSTANDARD_LEVEL=5
--
tests/control_p8-if [[ " hera orion hercules jet " =~ " ${MACHINE_ID} " ]] ; then
tests/control_p8:  ZSTANDARD_LEVEL=5
--
tests/control_p8.v2.sfc-if [[ " hera orion hercules gaea jet " =~ " ${MACHINE_ID} " ]] ; then
tests/control_p8.v2.sfc:  ZSTANDARD_LEVEL=5
--
tests/control_wam-if [[ " hera orion hercules gaea jet " =~ " ${MACHINE_ID} " ]] ; then
tests/control_wam:  ZSTANDARD_LEVEL=5
--
tests/control_wam_debug-if [[ " hera orion hercules gaea jet " =~ " ${MACHINE_ID} " ]] ; then
tests/control_wam_debug:  ZSTANDARD_LEVEL=5
--
tests/conus13km_control-if [[ " hera orion hercules jet " =~ " ${MACHINE_ID} " ]] ; then
tests/conus13km_control:  ZSTANDARD_LEVEL=5
--
tests/regional_debug-if [[ " hera orion hercules jet " =~ " ${MACHINE_ID} " ]] ; then
tests/regional_debug:  ZSTANDARD_LEVEL=5
--
tests/rrfs_v1beta-if [[ " hera orion hercules jet " =~ " ${MACHINE_ID} " ]] ; then
tests/rrfs_v1beta:  ZSTANDARD_LEVEL=5

All those tests run on hera, orion, hercules and jet, and few even run on gaea.

junwang-noaa commented 1 month ago

Thanks, Dusan. Can you also test the lossy compression ( the quantize_nsd that corresponding to nbit=14)?

ufs-community / ufs-weather-model