yadongxuEPA
commented
2 months ago Tested with VERDI_2.1.5_linux64_20240628.tar.gz (after NetCDF 5.5.3 update) on Atmos, got the same error as above:
Open jherwehe opened 9 months ago
yadongxuEPA
commented
2 months ago Tested with VERDI_2.1.5_linux64_20240628.tar.gz (after NetCDF 5.5.3 update) on Atmos, got the same error as above:
lizadams
commented
2 months ago the example cdf5 mpas files are 17.4 GB in size. Added the file to the google drive for download: https://drive.google.com/file/d/18KaBmU0JBHrmqFkEsR18hcUkYZwPL-Oh/view?usp=drive_link
CDF-5 file format specification http://cucis.ece.northwestern.edu/projects/PnetCDF/CDF-5.html
Header information:
netcdf history.2016-07-31.cdf5 {
dimensions:
nVertLevels = 50 ;
nCells = 163842 ;
Time = UNLIMITED ; // (24 currently)
nEdges = 491520 ;
nVertices = 327680 ;
TWO = 2 ;
maxEdges = 10 ;
maxEdges2 = 20 ;
vertexDegree = 3 ;
nVertLevelsP1 = 51 ;
StrLen = 64 ;
nSoilLevels = 4 ;
variables:
float qv(Time, nCells, nVertLevels) ;
qv:long_name = "Water vapor mixing ratio" ;
qv:units = "kg kg^{-1}" ;
float qc(Time, nCells, nVertLevels) ;
qc:long_name = "Cloud water mixing ratio" ;
qc:units = "kg kg^{-1}" ;
float qr(Time, nCells, nVertLevels) ;
qr:long_name = "Rain water mixing ratio" ;
qr:units = "kg kg^{-1}" ;
float qi(Time, nCells, nVertLevels) ;
qi:long_name = "Ice mixing ratio" ;
qi:units = "kg kg^{-1}" ;
Note: skipped some of the contents;
int landmask(nCells) ;
landmask:units = "unitless" ;
landmask:long_name = "land-ocean mask (1=land ; 0=ocean)" ;
float xland(Time, nCells) ;
xland:units = "unitless" ;
xland:long_name = "land-ocean mask (1=land including sea-ice ; 2=ocean)" ;
float ter(nCells) ;
ter:units = "m" ;
ter:long_name = "terrain height" ;
float snow(Time, nCells) ;
snow:units = "kg m^{-2}" ;
snow:long_name = "snow water equivalent" ;
float snowh(Time, nCells) ;
snowh:units = "m" ;
snowh:long_name = "physical snow depth" ;
float sst(Time, nCells) ;
sst:units = "K" ;
sst:long_name = "sea-surface temperature" ;
float vegfra(Time, nCells) ;
vegfra:units = "unitless" ;
vegfra:long_name = "vegetation fraction" ;
float xice(Time, nCells) ;
xice:units = "unitless" ;
xice:long_name = "fractional area coverage of sea-ice" ;
float smois(Time, nCells, nSoilLevels) ;
smois:units = "m3 m^{-3}" ;
smois:long_name = "soil moisture" ;
float tslb(Time, nCells, nSoilLevels) ;
tslb:units = "K" ;
tslb:long_name = "soil layer temperature" ;
// global attributes:
:model_name = "mpas" ;
:core_name = "atmosphere" ;
:source = "MPAS" ;
:Conventions = "MPAS" ;
:git_version = "v8.0.1" ;
:on_a_sphere = "YES" ;
:sphere_radius = 6371229.f ;
:is_periodic = "NO" ;
:x_period = 0.f ;
:y_period = 0.f ;
:history = "mpirun -n 192 ./atmosphere_model" ;
:parent_id = "2itjejtb90\n",
"dvbpxhg4fl\n",
"" ;
:mesh_spec = "0.0" ;
:config_time_integration = "SRK3" ;
:config_time_integration_order = 2 ;
:config_dt = 120.f ;
:config_calendar_type = "gregorian" ;
:config_start_time = "2016-06-21_00:00:00" ;
:config_stop_time = "none" ;
:config_run_duration = "41_00:00:00" ;
:config_split_dynamics_transport = "YES" ;
:config_number_of_sub_steps = 2 ;
:config_dynamics_split_steps = 3 ;
:config_h_mom_eddy_visc2 = 0.f ;
:config_h_mom_eddy_visc4 = 0.f ;
:config_v_mom_eddy_visc2 = 0.f ;
:config_h_theta_eddy_visc4 = 0.f ;
:config_v_theta_eddy_visc2 = 0.f ;
:config_horiz_mixing = "2d_smagorinsky" ;
:config_len_disp = 25000.f ;
:config_visc4_2dsmag = 0.05f ;
:config_del4u_div_factor = 10.f ;
:config_w_adv_order = 3 ;
:config_theta_adv_order = 3 ;
:config_scalar_adv_order = 3 ;
:config_u_vadv_order = 3 ;
:config_w_vadv_order = 3 ;
:config_theta_vadv_order = 3 ;
:config_scalar_vadv_order = 3 ;
:config_scalar_advection = "YES" ;
:config_positive_definite = "NO" ;
:config_monotonic = "YES" ;
:config_coef_3rd_order = 0.25f ;
:config_smagorinsky_coef = 0.125f ;
:config_mix_full = "YES" ;
:config_epssm = 0.1f ;
:config_smdiv = 0.1f ;
:config_apvm_upwinding = 0.5f ;
:config_h_ScaleWithMesh = "YES" ;
:config_num_halos = 2 ;
:config_relax_zone_divdamp_coef = 6.f ;
:config_zd = 22000.f ;
:config_xnutr = 0.2f ;
:config_mpas_cam_coef = 0.f ;
:config_number_cam_damping_levels = 4 ;
:config_rayleigh_damp_u = "NO" ;
:config_rayleigh_damp_u_timescale_days = 5.f ;
:config_number_rayleigh_damp_u_levels = 6 ;
:config_apply_lbcs = "NO" ;
:config_restart_timestamp_name = "restart_timestamp" ;
:config_pio_num_iotasks = 0 ;
:config_pio_stride = 1 ;
:config_block_decomp_file_prefix = "x4.163842.graph.info.part." ;
:config_number_of_blocks = 0 ;
:config_explicit_proc_decomp = "NO" ;
:config_proc_decomp_file_prefix = "graph.info.part." ;
:config_do_restart = "NO" ;
:config_do_DAcycling = "NO" ;
:config_print_global_minmax_vel = "YES" ;
:config_print_detailed_minmax_vel = "YES" ;
:config_print_global_minmax_sca = "YES" ;
:config_IAU_option = "off" ;
:config_IAU_window_length_s = 21600.f ;
:config_jedi_da = "NO" ;
:config_halo_exch_method = "mpas_halo" ;
:input_soil_data = "STAS" ;
:input_soil_temperature_lag = 140 ;
:num_soil_layers = 4 ;
:months = 12 ;
:noznlev = 59 ;
:naerlev = 29 ;
:camdim1 = 4 ;
:config_frac_seaice = "YES" ;
:config_sfc_albedo = "YES" ;
:config_sfc_snowalbedo = "YES" ;
:config_sst_update = "YES" ;
:config_sstdiurn_update = "NO" ;
:config_deepsoiltemp_update = "NO" ;
:config_o3climatology = "YES" ;
:config_microp_re = "NO" ;
:config_ysu_pblmix = "NO" ;
:config_n_microp = 1 ;
:config_radtlw_interval = "00:12:00" ;
:config_radtsw_interval = "00:12:00" ;
:config_conv_interval = "none" ;
:config_pbl_interval = "none" ;
:config_camrad_abs_update = "06:00:00" ;
:config_greeness_update = "24:00:00" ;
:config_bucket_update = "none" ;
:config_physics_suite = "none" ;
:config_microp_scheme = "mp_wsm6" ;
:config_convection_scheme = "cu_kain_fritsch" ;
:config_lsm_scheme = "sf_noah" ;
:config_pbl_scheme = "bl_ysu" ;
:config_gwdo_scheme = "bl_ysu_gwdo" ;
:config_radt_cld_scheme = "cld_fraction" ;
:config_radt_lw_scheme = "rrtmg_lw" ;
:config_radt_sw_scheme = "rrtmg_sw" ;
:config_sfclayer_scheme = "sf_monin_obukhov" ;
:config_gfconv_closure_deep = 0 ;
:config_gfconv_closure_shallow = 8 ;
:config_bucket_radt = 1.e+09f ;
:config_bucket_rainc = 100.f ;
:config_bucket_rainnc = 100.f ;
:config_oml1d = "NO" ;
:config_oml_hml0 = 30.f ;
:config_oml_gamma = 0.14f ;
:config_oml_relaxation_time = 864000.f ;
:config_sounding_interval = "none" ;
:file_id = "ub68myvrtj" ;Found the following issue that states that CDF5 is not yet supported by netcdf-java. https://github.com/Unidata/netcdf-java/pull/1350
Is your feature request related to a problem? Please describe. Starting with NCAR's release of MPAS v8.0.0, a new I/O layer called SMIOL has been added as an option. SMIOL allows the user to compile MPAS without having to compile the PIO library first (which is often a problematic build), thus simplifying the building of MPAS.
However, using the SMIOL I/O layer only produces CDF-5 (reported as "cdf5" when issuing an "ncdump -k file.nc" command) netCDF format for all output files from MPAS. And VERDI currently cannot read cdf5 files (as tested with the 20231013 build of VERDI 2.1.4).
CDF-5 format became available starting with NetCDF 4.4.0 (about 2016).
Describe the solution you'd like Add the ability of VERDI to read the MPAS cdf5 format along with all other NetCDF format files that can be produced by MPAS (i.e., so VERDI can read cdf5, as well as the current 64-bit offset and netCDF-4 formats from MPAS).
For testing, MPAS output files in the cdf5 format are available on Atmos at: /work/MOD3DEV/jherwehe/mpas/MPAS-v8.0.1-Release/run.92-25km.Jul2016.Noah_YSU_KF_WSM6_MODIS.SMIOL.RHEL8
Describe alternatives you've considered The only work-around for VERDI not being able to read the cdf5 format is to use nccopy or an NCO command to convert the cdf5 MPAS output files to 64-bit offset format, which would be cumbersome. Better to have VERDI read cdf5 files directly.
Additional context Here is the error that VERDI produces when attempting to read a cdf5 MPAS file: