search

milankl / BitInformation.jl

Information between bits and bytes.
MIT License
33 stars 3 forks source link

Applying BitInformation to compress WRF model results #26

Open rsignell-usgs opened 2 years ago

rsignell-usgs commented 2 years ago

As mentioned by @rkouznetsov in @https://github.com/milankl/BitInformation.jl/issues/25#issuecomment-998113627, he has applied the BitInformation approach to compressing the WRF model, using this NCKS script:

#!/bin/bash
infile=$1
outfile=$2

list3d='U,V,T,QVAPOR,QCLOUD,QRAIN,QICE,QSNOW,QGRAUP,CLDFRA'

prc3d="--ppc U,V,T=.3 --ppc QVAPOR,QCLOUD,QRAIN,QICE,QSNOW,QGRAUP=.7 --ppc CLDFRA=.3"
  list2d='Q2,T2,PSFC,U10,V10,TSLB,SMOIS,SEAICE,SNOW,SNOWH,SSTSK,LAI,TSK,RAINC,RAINNC,SWDOWN,SWDNB,ALBEDO,UST,PBLH,HFX,QFX,ACHFX,SST'

prc2d='--ppc U10,V10,T2,SMOIS,SEAICE,SSTSK,LAI,TSK,RAINC,RAINNC,ALBEDO,SST,TSLB=.2 --ppc Q2,PBLH,UST=2'
  gridvars="SINALPHA,COSALPHA,MAPFAC_MX,MAPFAC_MY,XLONG,XLAT,XLONG_U,XLAT_U,XLONG_V,XLAT_V,MAPFAC_UX,MAPFAC_UY,MAPFAC_VX,MAPFAC_VY"
  othervars='XLON.*,XLAT.*,Times,ZNU,ZNW,ZS,DZS,HFX_FORCE,LH_FORCE,TSK_FORCE,HFX_FORCE_TEND,LH_FORCE_TEND,TSK_FORCE_TEND,FNM,FNP,RDNW,RDN,DNW,DN,CFN,CFN1,THIS_IS_AN_IDEAL_RUN,RDX,RDY,RESM,ZETATOP,CF1,CF2,CF3,ITIMESTEP,XTIME,P_TOP,T00,P00,TLP,TISO,TLP_STRAT,P_STRAT,SAVE_TOPO_FROM_REAL,ISEEDARR_SPPT,ISEEDARR_SKEBS,ISEEDARR_RAND_PERTURB,ISEEDARRAY_SPP_CONV,ISEEDARRAY_SPP_PBL,ISEEDARRAY_SPP_LSM,C1H,C2H,C1F,C2F,C3H,C4H,C3F,C4F'

ncks -4 -L 5 --baa=5 -v $gridvars,$othervars,$list3d,$list2d $prc3d $prc2d --cnk_dmn bottom_top,1 $infile $outfile
rsignell-usgs commented 2 years ago

@rkouznetsov, would you also be willing to share the code you used to analyze the WRF output to determine the keepbits?

rkouznetsov commented 2 years ago

It is not a result of any derivation. Rather some ad-hoc parameters that, allow for a decent compression, while not affecting SILAM simulations. Note that most of the variables are rounded to keep absolute precision, not relative. E.g. ".7" means that the absolute error will be 1e-7 (kg/kg for specific humidity). As I wrote, the trimming seems to be quite conservative.

rsignell-usgs commented 2 years ago

@rkouznetsov , I'm curious: did you use the BitInformation method to help guide your selection of any of the parameters you used with your NCO/WRF script above (e.g. for variables you were unsure of)? Or did you just choose them on the basis of intuition/experience and negligible changes in model-data comparison skill?

rkouznetsov commented 2 years ago

did you use the BitInformation method

No. It is a script of couple of years ago. The basis is just (subjective) common sense, some idea of sensitivity of SILAM to input uncertainties and some back-envelope estimates. I believe, one could do a better job, but for my purposes the above parameters were "good enough".

Your mileage might differ though...