search

akrherz / oats

"Oats & Horses" modelling work
Apache License 2.0
0 stars 0 forks source link

Audit d02 netcdf file sizes in hopes of making them smaller #1

Open akrherz opened 7 years ago

akrherz commented 7 years ago

For example wrfout_d02_2010-01-29_22:20:00 is 52.6 GiB in size. This presents problems as our cloud backup solution only can handle 15 GiB files and we'd like to have all these on one filesystem at the same time. So lets investigate this and see if we can trim things down. Step 1, inventory variables and their sizes.

PS. gzip'ing the file resulted in a file of 28.8GiB

akrherz commented 7 years ago

This is a grouped inventory of the wrfout_d02_2010-01-29_22:20:00 file.

(u'Time', u'bottom_top')

VNAME DESCRIPTION DTYPE SHAPE
DN d(eta) values between half (mass) levels float32 (1000, 34)
DNW d(eta) values between full (w) levels float32 (1000, 34)
FNM upper weight for vertical stretching float32 (1000, 34)
FNP lower weight for vertical stretching float32 (1000, 34)
RDN inverse d(eta) values between half (mass) levels float32 (1000, 34)
RDNW inverse d(eta) values between full (w) levels float32 (1000, 34)
ZNU eta values on half (mass) levels float32 (1000, 34)

(u'Time', u'bottom_top', u'south_north', u'west_east')

VNAME DESCRIPTION DTYPE SHAPE
CLDFRA CLOUD FRACTION float32 (1000, 34, 117, 138)
P perturbation pressure float32 (1000, 34, 117, 138)
PB BASE STATE PRESSURE float32 (1000, 34, 117, 138)
P_HYD hydrostatic pressure float32 (1000, 34, 117, 138)
QCLOUD Cloud water mixing ratio float32 (1000, 34, 117, 138)
QGRAUP Graupel mixing ratio float32 (1000, 34, 117, 138)
QICE Ice mixing ratio float32 (1000, 34, 117, 138)
QRAIN Rain water mixing ratio float32 (1000, 34, 117, 138)
QSNOW Snow mixing ratio float32 (1000, 34, 117, 138)
QVAPOR Water vapor mixing ratio float32 (1000, 34, 117, 138)
REFL_10CM Radar reflectivity (lamda = 10 cm) float32 (1000, 34, 117, 138)
T perturbation potential temperature (theta-t0) float32 (1000, 34, 117, 138)

(u'Time', u'bottom_top', u'south_north', u'west_east_stag')

VNAME DESCRIPTION DTYPE SHAPE
U x-wind component float32 (1000, 34, 117, 139)

(u'Time', u'bottom_top', u'south_north_stag', u'west_east')

VNAME DESCRIPTION DTYPE SHAPE
V y-wind component float32 (1000, 34, 118, 138)

(u'Time', u'bottom_top_stag')

VNAME DESCRIPTION DTYPE SHAPE
ZNW eta values on full (w) levels float32 (1000, 35)

(u'Time', u'bottom_top_stag', u'south_north', u'west_east')

VNAME DESCRIPTION DTYPE SHAPE
PH perturbation geopotential float32 (1000, 35, 117, 138)
PHB base-state geopotential float32 (1000, 35, 117, 138)
W z-wind component float32 (1000, 35, 117, 138)

(u'Time', u'soil_layers_stag')

VNAME DESCRIPTION DTYPE SHAPE
DZS THICKNESSES OF SOIL LAYERS float32 (1000, 10)
ZS DEPTHS OF CENTERS OF SOIL LAYERS float32 (1000, 10)

(u'Time', u'soil_layers_stag', u'south_north', u'west_east')

VNAME DESCRIPTION DTYPE SHAPE
SH2O SOIL LIQUID WATER float32 (1000, 10, 117, 138)
SMCREL RELATIVE SOIL MOISTURE float32 (1000, 10, 117, 138)
SMOIS SOIL MOISTURE float32 (1000, 10, 117, 138)
TSLB SOIL TEMPERATURE float32 (1000, 10, 117, 138)

(u'Time', u'south_north', u'west_east')

VNAME DESCRIPTION DTYPE SHAPE
ACGRDFLX ACCUMULATED GROUND HEAT FLUX float32 (1000, 117, 138)
ACHFX ACCUMULATED UPWARD HEAT FLUX AT THE SURFACE float32 (1000, 117, 138)
ACLHF ACCUMULATED UPWARD LATENT HEAT FLUX AT THE SURFACE float32 (1000, 117, 138)
ACSNOM ACCUMULATED MELTED SNOW float32 (1000, 117, 138)
ACSWDNB ACCUMULATED DOWNWELLING SHORTWAVE FLUX AT BOTTOM float32 (1000, 117, 138)
ACSWDNBC ACCUMULATED DOWNWELLING CLEAR SKY SHORTWAVE FLUX AT BOTTOM float32 (1000, 117, 138)
ACSWDNT ACCUMULATED DOWNWELLING SHORTWAVE FLUX AT TOP float32 (1000, 117, 138)
ACSWDNTC ACCUMULATED DOWNWELLING CLEAR SKY SHORTWAVE FLUX AT TOP float32 (1000, 117, 138)
ACSWUPB ACCUMULATED UPWELLING SHORTWAVE FLUX AT BOTTOM float32 (1000, 117, 138)
ACSWUPBC ACCUMULATED UPWELLING CLEAR SKY SHORTWAVE FLUX AT BOTTOM float32 (1000, 117, 138)
ACSWUPT ACCUMULATED UPWELLING SHORTWAVE FLUX AT TOP float32 (1000, 117, 138)
ACSWUPTC ACCUMULATED UPWELLING CLEAR SKY SHORTWAVE FLUX AT TOP float32 (1000, 117, 138)
AFWA_CAPE AFWA Diagnostic: Convective Avail Pot Energy float32 (1000, 117, 138)
AFWA_CLOUD_CEIL AFWA Diagnostic: Cloud ceiling float32 (1000, 117, 138)
AFWA_HAIL AFWA Diagnostic: Hail Diameter (Weibull) float32 (1000, 117, 138)
AFWA_LLWS AFWA Diagnostic: 0-2000 ft wind shear float32 (1000, 117, 138)
AFWA_PLFC AFWA Diagnostic: Pressure of LFC float32 (1000, 117, 138)
AFWA_TORNADO AFWA Diagnostic: Tornado wind speed (Weibull) float32 (1000, 117, 138)
AFWA_ZLFC AFWA Diagnostic: Level of Free Convection float32 (1000, 117, 138)
ALBBCK BACKGROUND ALBEDO float32 (1000, 117, 138)
ALBEDO ALBEDO float32 (1000, 117, 138)
CANWAT CANOPY WATER float32 (1000, 117, 138)
CLAT COMPUTATIONAL GRID LATITUDE, SOUTH IS NEGATIVE float32 (1000, 117, 138)
COSALPHA Local cosine of map rotation float32 (1000, 117, 138)
COSZEN COS of SOLAR ZENITH ANGLE float32 (1000, 117, 138)
E Coriolis cosine latitude term float32 (1000, 117, 138)
EMISS SURFACE EMISSIVITY float32 (1000, 117, 138)
F Coriolis sine latitude term float32 (1000, 117, 138)
GLW DOWNWARD LONG WAVE FLUX AT GROUND SURFACE float32 (1000, 117, 138)
GRAUPELNC ACCUMULATED TOTAL GRID SCALE GRAUPEL float32 (1000, 117, 138)
GRDFLX GROUND HEAT FLUX float32 (1000, 117, 138)
HAILNC ACCUMULATED TOTAL GRID SCALE HAIL float32 (1000, 117, 138)
HFX UPWARD HEAT FLUX AT THE SURFACE float32 (1000, 117, 138)
HGT Terrain Height float32 (1000, 117, 138)
ISLTYP DOMINANT SOIL CATEGORY int32 (1000, 117, 138)
IVGTYP DOMINANT VEGETATION CATEGORY int32 (1000, 117, 138)
I_ACSWDNB BUCKET FOR DOWNWELLING SHORTWAVE FLUX AT BOTTOM int32 (1000, 117, 138)
I_ACSWDNBC BUCKET FOR DOWNWELLING CLEAR SKY SHORTWAVE FLUX AT BOTTOM int32 (1000, 117, 138)
I_ACSWDNT BUCKET FOR DOWNWELLING SHORTWAVE FLUX AT TOP int32 (1000, 117, 138)
I_ACSWDNTC BUCKET FOR DOWNWELLING CLEAR SKY SHORTWAVE FLUX AT TOP int32 (1000, 117, 138)
I_ACSWUPB BUCKET FOR UPWELLING SHORTWAVE FLUX AT BOTTOM int32 (1000, 117, 138)
I_ACSWUPBC BUCKET FOR UPWELLING CLEAR SKY SHORTWAVE FLUX AT BOTTOM int32 (1000, 117, 138)
I_ACSWUPT BUCKET FOR UPWELLING SHORTWAVE FLUX AT TOP int32 (1000, 117, 138)
I_ACSWUPTC BUCKET FOR UPWELLING CLEAR SKY SHORTWAVE FLUX AT TOP int32 (1000, 117, 138)
LAI LEAF AREA INDEX float32 (1000, 117, 138)
LAKEMASK LAKE MASK (1 FOR LAKE, 0 FOR NON-LAKE) float32 (1000, 117, 138)
LANDMASK LAND MASK (1 FOR LAND, 0 FOR WATER) float32 (1000, 117, 138)
LAP_HGT Laplacian of orography float32 (1000, 117, 138)
LH LATENT HEAT FLUX AT THE SURFACE float32 (1000, 117, 138)
LU_INDEX LAND USE CATEGORY float32 (1000, 117, 138)
LWUP OUTGOING LONGWAVE RADIATION float32 (1000, 117, 138)
MAPFAC_M Map scale factor on mass grid float32 (1000, 117, 138)
MAPFAC_MX Map scale factor on mass grid, x direction float32 (1000, 117, 138)
MAPFAC_MY Map scale factor on mass grid, y direction float32 (1000, 117, 138)
MIDRH_MIN Min Mid-level relative humidity float32 (1000, 117, 138)
MU perturbation dry air mass in column float32 (1000, 117, 138)
MUB base state dry air mass in column float32 (1000, 117, 138)
NEST_POS - float32 (1000, 117, 138)
NOAHRES RESIDUAL OF THE NOAH SURFACE ENERGY BUDGET float32 (1000, 117, 138)
OLR TOA OUTGOING LONG WAVE float32 (1000, 117, 138)
PBLH PBL HEIGHT float32 (1000, 117, 138)
PSFC SFC PRESSURE float32 (1000, 117, 138)
Q2 QV at 2 M float32 (1000, 117, 138)
QFX UPWARD MOISTURE FLUX AT THE SURFACE float32 (1000, 117, 138)
QNWFA2D Surface aerosol number conc emission float32 (1000, 117, 138)
RAINC ACCUMULATED TOTAL CUMULUS PRECIPITATION float32 (1000, 117, 138)
RAINNC ACCUMULATED TOTAL GRID SCALE PRECIPITATION float32 (1000, 117, 138)
RAINSH ACCUMULATED SHALLOW CUMULUS PRECIPITATION float32 (1000, 117, 138)
SABG NET SOIL SOLAR RADIATION float32 (1000, 117, 138)
SABV NET VEGETATION SOLAR RADIATION float32 (1000, 117, 138)
SEAICE SEA ICE FLAG float32 (1000, 117, 138)
SFROFF SURFACE RUNOFF float32 (1000, 117, 138)
SHDMAX ANNUAL MAX VEG FRACTION float32 (1000, 117, 138)
SHDMIN ANNUAL MIN VEG FRACTION float32 (1000, 117, 138)
SINALPHA Local sine of map rotation float32 (1000, 117, 138)
SNOALB ANNUAL MAX SNOW ALBEDO IN FRACTION float32 (1000, 117, 138)
SNOW SNOW WATER EQUIVALENT float32 (1000, 117, 138)
SNOWC FLAG INDICATING SNOW COVERAGE (1 FOR SNOW COVER) float32 (1000, 117, 138)
SNOWH PHYSICAL SNOW DEPTH float32 (1000, 117, 138)
SNOWNC ACCUMULATED TOTAL GRID SCALE SNOW AND ICE float32 (1000, 117, 138)
SR fraction of frozen precipitation float32 (1000, 117, 138)
SST SEA SURFACE TEMPERATURE float32 (1000, 117, 138)
SSTSK SKIN SEA SURFACE TEMPERATURE float32 (1000, 117, 138)
SST_INPUT SEA SURFACE TEMPERATURE FROM WRFLOWINPUT FILE float32 (1000, 117, 138)
SWDNB INSTANTANEOUS DOWNWELLING SHORTWAVE FLUX AT BOTTOM float32 (1000, 117, 138)
SWDNBC INSTANTANEOUS DOWNWELLING CLEAR SKY SHORTWAVE FLUX AT BOTTOM float32 (1000, 117, 138)
SWDNT INSTANTANEOUS DOWNWELLING SHORTWAVE FLUX AT TOP float32 (1000, 117, 138)
SWDNTC INSTANTANEOUS DOWNWELLING CLEAR SKY SHORTWAVE FLUX AT TOP float32 (1000, 117, 138)
SWDOWN DOWNWARD SHORT WAVE FLUX AT GROUND SURFACE float32 (1000, 117, 138)
SWNORM NORMAL SHORT WAVE FLUX AT GROUND SURFACE (SLOPE-DEPENDENT) float32 (1000, 117, 138)
SWUPB INSTANTANEOUS UPWELLING SHORTWAVE FLUX AT BOTTOM float32 (1000, 117, 138)
SWUPBC INSTANTANEOUS UPWELLING CLEAR SKY SHORTWAVE FLUX AT BOTTOM float32 (1000, 117, 138)
SWUPT INSTANTANEOUS UPWELLING SHORTWAVE FLUX AT TOP float32 (1000, 117, 138)
SWUPTC INSTANTANEOUS UPWELLING CLEAR SKY SHORTWAVE FLUX AT TOP float32 (1000, 117, 138)
T2 TEMP at 2 M float32 (1000, 117, 138)
T2CLM 2M TEMPERATURE IN CLM float32 (1000, 117, 138)
T2M_MAX MAX TEMPERATURE AT 2 M float32 (1000, 117, 138)
T2M_MIN MIN TEMPERATURE AT 2 M float32 (1000, 117, 138)
TH2 POT TEMP at 2 M float32 (1000, 117, 138)
TMN SOIL TEMPERATURE AT LOWER BOUNDARY float32 (1000, 117, 138)
TSK SURFACE SKIN TEMPERATURE float32 (1000, 117, 138)
U10 U at 10 M float32 (1000, 117, 138)
UDROFF UNDERGROUND RUNOFF float32 (1000, 117, 138)
UST U* IN SIMILARITY THEORY float32 (1000, 117, 138)
V10 V at 10 M float32 (1000, 117, 138)
VAR OROGRAPHIC VARIANCE float32 (1000, 117, 138)
VAR_SSO variance of subgrid-scale orography float32 (1000, 117, 138)
VEGFRA VEGETATION FRACTION float32 (1000, 117, 138)
XICEM SEA ICE FLAG (PREVIOUS STEP) float32 (1000, 117, 138)
XLAND LAND MASK (1 FOR LAND, 2 FOR WATER) float32 (1000, 117, 138)
XLAT LATITUDE, SOUTH IS NEGATIVE float32 (1000, 117, 138)
XLONG LONGITUDE, WEST IS NEGATIVE float32 (1000, 117, 138)

(u'Time', u'south_north', u'west_east_stag')

VNAME DESCRIPTION DTYPE SHAPE
MAPFAC_U Map scale factor on u-grid float32 (1000, 117, 139)
MAPFAC_UX Map scale factor on u-grid, x direction float32 (1000, 117, 139)
MAPFAC_UY Map scale factor on u-grid, y direction float32 (1000, 117, 139)
XLAT_U LATITUDE, SOUTH IS NEGATIVE float32 (1000, 117, 139)
XLONG_U LONGITUDE, WEST IS NEGATIVE float32 (1000, 117, 139)

(u'Time', u'south_north_stag', u'west_east')

VNAME DESCRIPTION DTYPE SHAPE
MAPFAC_V Map scale factor on v-grid float32 (1000, 118, 138)
MAPFAC_VX Map scale factor on v-grid, x direction float32 (1000, 118, 138)
MAPFAC_VY Map scale factor on v-grid, y direction float32 (1000, 118, 138)
MF_VX_INV Inverse map scale factor on v-grid, x direction float32 (1000, 118, 138)
XLAT_V LATITUDE, SOUTH IS NEGATIVE float32 (1000, 118, 138)
XLONG_V LONGITUDE, WEST IS NEGATIVE float32 (1000, 118, 138)

(u'Time', u'subgrid_stag', u'south_north', u'west_east')

VNAME DESCRIPTION DTYPE SHAPE
H2OCAN SUBGRID VEGETATION INTERCEP WATER float32 (1000, 10, 117, 138)
LHSOI LH from soil float32 (1000, 10, 117, 138)
LHTRAN LH from transpiration float32 (1000, 10, 117, 138)
LHVEG LH from vegetation float32 (1000, 10, 117, 138)
T_GRND SUBGRID GROUND TEMPERATURE float32 (1000, 10, 117, 138)
T_REF2M TEMPERATURE AT 2 M float32 (1000, 10, 117, 138)
T_VEG SUBGRID VEGETATION TEMPERATURE float32 (1000, 10, 117, 138)

(u'Time',)

VNAME DESCRIPTION DTYPE SHAPE
CF1 2nd order extrapolation constant float32 (1000,)
CF2 2nd order extrapolation constant float32 (1000,)
CF3 2nd order extrapolation constant float32 (1000,)
CFN extrapolation constant float32 (1000,)
CFN1 extrapolation constant float32 (1000,)
HFX_FORCE SCM ideal surface sensible heat flux float32 (1000,)
HFX_FORCE_TEND SCM ideal surface sensible heat flux tendency float32 (1000,)
ITIMESTEP int32 (1000,)
LH_FORCE SCM ideal surface latent heat flux float32 (1000,)
LH_FORCE_TEND SCM ideal surface latent heat flux tendency float32 (1000,)
MAX_MSTFX Max map factor in domain float32 (1000,)
MAX_MSTFY Max map factor in domain float32 (1000,)
P00 BASE STATE PRESURE float32 (1000,)
P_TOP PRESSURE TOP OF THE MODEL float32 (1000,)
RDX INVERSE X GRID LENGTH float32 (1000,)
RDY INVERSE Y GRID LENGTH float32 (1000,)
RESM TIME WEIGHT CONSTANT FOR SMALL STEPS float32 (1000,)
SAVE_TOPO_FROM_REAL 1=original topo from real/0=topo modified by WRF int32 (1000,)
SEED1 RANDOM SEED NUMBER 1 int32 (1000,)
SEED2 RANDOM SEED NUMBER 2 int32 (1000,)
T00 BASE STATE TEMPERATURE float32 (1000,)
TISO TEMP AT WHICH THE BASE T TURNS CONST float32 (1000,)
TLP BASE STATE LAPSE RATE float32 (1000,)
TSK_FORCE SCM ideal surface skin temperature float32 (1000,)
TSK_FORCE_TEND SCM ideal surface skin temperature tendency float32 (1000,)
XTIME minutes since simulation start float32 (1000,)
ZETATOP ZETA AT MODEL TOP float32 (1000,)