search

LBL-EESA / TECA

TECA, theToolkit for Extreme Climate Analysis, contains a collection of climate anlysis algorithms targetted at extreme event detection and analysis.
Other
55 stars 21 forks source link

test the cf reader with WRF and other data used by the team #177

Closed burlen closed 3 years ago

burlen commented 5 years ago

@taobrienlbl can you point me to a sample? it's a pain to upload to here, so maybe on cori if it's easier?

taobrienlbl commented 5 years ago

I just made a directory on scratch that contains test files for a number of datasets that we should verify that the TECA AR detector works on: see /global/cscratch1/sd/taobrien/for_teca_data_svn

The wrf files are in the 'WRF' subdirectory. For the WRF, MERRA2, and ERA5 datasets, I grabbed the same time slice (a day with a big western US AR in 2017). For the ERA20C dataset, I grabbed a time in late 2010 with a sizeable AR.

For both ERA datasets, IVT would need to be calculated from its components, which are in two separate files: e.g., sqrt(viwve2 + viwvn2) for ERA5.

I should note that the WRF files provided here are actually postprocessed from the raw model output. At some point in the future, we should consider building a detector that can work on raw WRF output; but for now, being able to operate on these postprocessed files will allow us to run TECA on Chris's Western U.S. WRF run, which I've already postprocessed.

burlen commented 5 years ago

WRF -- not working, coordinates are not understood. I think the currently we expect coordinate axes to be 1d, in this wrf file they are 2d.

$./bin/test/test_cf_reader "/work2/data/teca/for_teca_data_svn/WRF/wrf_western_us_2017-02-18\.nc$" "test_cf_reader_%t%.%e%" "0" "-1" "XLONG" "XLAT" "." "XTIME" "IVT"
ERROR: [0:140539010963008] [/home/bloring/work/teca/teca/io/teca_cf_reader.cxx:524 2.2.0]
ERROR: Failed to query x axis variable "XLONG" in file "/work2/data/teca/for_teca_data_svn/WRF/wrf_western_us_2017-02-18.nc"
NetCDF: Invalid dimension ID or name
ERROR: [0:140539010963008] [/home/bloring/work/teca/teca/core/teca_index_executive.cxx:77 2.2.0]
ERROR: No index initializer key has been specified
ERROR: [0:140539010963008] [/home/bloring/work/teca/teca/core/teca_algorithm.cxx:686 2.2.0]
ERROR: failed to initialize the executive
ERROR: [0:140539010963008] [/home/bloring/work/teca/teca/core/teca_algorithm.cxx:710 2.2.0]
ERROR: failed to update port 0

contents of file:


$ncdump -h /work2/data/teca/for_teca_data_svn/WRF/wrf_western_us_2017-02-18.nc 
netcdf wrf_western_us_2017-02-18 {
dimensions:
    south_north = 355 ;
    west_east = 655 ;
    time = 4 ;
variables:
    float XLAT(south_north, west_east) ;
        XLAT:_FillValue = NaNf ;
        XLAT:FieldType = 104 ;
        XLAT:MemoryOrder = "XY " ;
        XLAT:description = "LATITUDE, SOUTH IS NEGATIVE" ;
        XLAT:units = "degree_north" ;
        XLAT:stagger = "" ;
    float XLONG(south_north, west_east) ;
        XLONG:_FillValue = NaNf ;
        XLONG:FieldType = 104 ;
        XLONG:MemoryOrder = "XY " ;
        XLONG:description = "LONGITUDE, WEST IS NEGATIVE" ;
        XLONG:units = "degree_east" ;
        XLONG:stagger = "" ;
    float XTIME ;
        XTIME:_FillValue = NaNf ;
        XTIME:FieldType = 104 ;
        XTIME:MemoryOrder = "0  " ;
        XTIME:description = "minutes since 2016-09-01 00:00:00" ;
        XTIME:stagger = "" ;
        XTIME:units = "minutes since 2016-09-01" ;
        XTIME:calendar = "proleptic_gregorian" ;
    float IWV(time, south_north, west_east) ;
        IWV:_FillValue = NaNf ;
        IWV:long_name = "Integrated Water Vapor" ;
        IWV:short_name = "IWV" ;
        IWV:units = "kg m-2" ;
        IWV:coordinates = "XLONG XTIME XLAT" ;
    float IVT(time, south_north, west_east) ;
        IVT:_FillValue = NaNf ;
        IVT:long_name = "Integrated Vaport Transport" ;
        IVT:short_name = "IVT" ;
        IVT:units = "kg m-1 s-1" ;
        IVT:coordinates = "XLONG XTIME XLAT" ;
    float RAIN(time, south_north, west_east) ;
        RAIN:_FillValue = NaNf ;
        RAIN:long_name = "Precipitation Rate" ;
        RAIN:units = "mm d-1" ;
        RAIN:short_name = "RAIN" ;
        RAIN:coordinates = "XLONG XTIME XLAT" ;
    float SNOW(time, south_north, west_east) ;
        SNOW:_FillValue = NaNf ;
        SNOW:FieldType = 104 ;
        SNOW:MemoryOrder = "XY " ;
        SNOW:description = "SNOW WATER EQUIVALENT" ;
        SNOW:units = "kg m-2" ;
        SNOW:stagger = "" ;
        SNOW:coordinates = "XLONG XTIME XLAT" ;
    float SNOWC(time, south_north, west_east) ;
        SNOWC:_FillValue = NaNf ;
        SNOWC:FieldType = 104 ;
        SNOWC:MemoryOrder = "XY " ;
        SNOWC:description = "FLAG INDICATING SNOW COVERAGE (1 FOR SNOW COVER)" ;
        SNOWC:units = "" ;
        SNOWC:stagger = "" ;
        SNOWC:coordinates = "XLONG XTIME XLAT" ;
    float T2(time, south_north, west_east) ;
        T2:_FillValue = NaNf ;
        T2:FieldType = 104 ;
        T2:MemoryOrder = "XY " ;
        T2:description = "TEMP at 2 M" ;
        T2:units = "K" ;
        T2:stagger = "" ;
        T2:coordinates = "XLONG XTIME XLAT" ;
    float time(time) ;
        time:_FillValue = NaNf ;
        time:FieldType = 104 ;
        time:MemoryOrder = "0  " ;
        time:description = "minutes since 2016-09-01 00:00:00" ;
        time:stagger = "" ;
        time:units = "minutes since 2016-09-01" ;
        time:calendar = "proleptic_gregorian" ;

// global attributes:
        :TITLE = " OUTPUT FROM WRF V3.8.1 MODEL" ;
        :START_DATE = "2016-09-01_00:00:00" ;
        :SIMULATION_START_DATE = "2016-09-01_00:00:00" ;
        :WEST-EAST_GRID_DIMENSION = 656 ;
        :SOUTH-NORTH_GRID_DIMENSION = 356 ;
        :BOTTOM-TOP_GRID_DIMENSION = 35 ;
        :DX = 27000.f ;
        :DY = 27000.f ;
        :SKEBS_ON = 0 ;
        :SPEC_BDY_FINAL_MU = 1 ;
        :USE_Q_DIABATIC = 0 ;
        :GRIDTYPE = "C" ;
        :DIFF_OPT = 1 ;
        :KM_OPT = 4 ;
        :DAMP_OPT = 0 ;
        :DAMPCOEF = 0.2f ;
        :KHDIF = 0.f ;
        :KVDIF = 0.f ;
        :MP_PHYSICS = 2 ;
        :RA_LW_PHYSICS = 4 ;
        :RA_SW_PHYSICS = 4 ;
        :SF_SFCLAY_PHYSICS = 1 ;
        :SF_SURFACE_PHYSICS = 2 ;
        :BL_PBL_PHYSICS = 1 ;
        :CU_PHYSICS = 1 ;
        :SF_LAKE_PHYSICS = 0 ;
        :SURFACE_INPUT_SOURCE = 1 ;
        :SST_UPDATE = 1 ;
        :GRID_FDDA = 0 ;
        :GFDDA_INTERVAL_M = 0 ;
        :GFDDA_END_H = 0 ;
        :GRID_SFDDA = 0 ;
        :SGFDDA_INTERVAL_M = 0 ;
        :SGFDDA_END_H = 0 ;
        :HYPSOMETRIC_OPT = 2 ;
        :USE_THETA_M = 0 ;
        :SF_URBAN_PHYSICS = 0 ;
        :SHCU_PHYSICS = 0 ;
        :MFSHCONV = 0 ;
        :FEEDBACK = 1 ;
        :SMOOTH_OPTION = 0 ;
        :SWRAD_SCAT = 1.f ;
        :W_DAMPING = 0 ;
        :DT = 90.f ;
        :RADT = 30.f ;
        :BLDT = 0.f ;
        :CUDT = 4.5f ;
        :AER_OPT = 0 ;
        :SWINT_OPT = 0 ;
        :AER_TYPE = 1 ;
        :AER_AOD550_OPT = 1 ;
        :AER_ANGEXP_OPT = 1 ;
        :AER_SSA_OPT = 1 ;
        :AER_ASY_OPT = 1 ;
        :AER_AOD550_VAL = 0.12f ;
        :AER_ANGEXP_VAL = 1.3f ;
        :AER_SSA_VAL = 1.401298e-45f ;
        :AER_ASY_VAL = 1.401298e-45f ;
        :MOIST_ADV_OPT = 1 ;
        :SCALAR_ADV_OPT = 1 ;
        :TKE_ADV_OPT = 1 ;
        :DIFF_6TH_OPT = 0 ;
        :DIFF_6TH_FACTOR = 0.12f ;
        :OBS_NUDGE_OPT = 0 ;
        :BUCKET_MM = -1.f ;
        :BUCKET_J = -1.f ;
        :PREC_ACC_DT = 0.f ;
        :SF_OCEAN_PHYSICS = 0 ;
        :ISFTCFLX = 0 ;
        :ISHALLOW = 0 ;
        :ISFFLX = 1 ;
        :ICLOUD = 1 ;
        :ICLOUD_CU = 0 ;
        :TRACER_PBLMIX = 1 ;
        :SCALAR_PBLMIX = 0 ;
        :YSU_TOPDOWN_PBLMIX = 0 ;
        :GRAV_SETTLING = 0 ;
        :DFI_OPT = 0 ;
        :SIMULATION_INITIALIZATION_TYPE = "REAL-DATA CASE" ;
        :WEST-EAST_PATCH_START_UNSTAG = 1 ;
        :WEST-EAST_PATCH_END_UNSTAG = 655 ;
        :WEST-EAST_PATCH_START_STAG = 1 ;
        :WEST-EAST_PATCH_END_STAG = 656 ;
        :SOUTH-NORTH_PATCH_START_UNSTAG = 1 ;
        :SOUTH-NORTH_PATCH_END_UNSTAG = 355 ;
        :SOUTH-NORTH_PATCH_START_STAG = 1 ;
        :SOUTH-NORTH_PATCH_END_STAG = 356 ;
        :BOTTOM-TOP_PATCH_START_UNSTAG = 1 ;
        :BOTTOM-TOP_PATCH_END_UNSTAG = 34 ;
        :BOTTOM-TOP_PATCH_START_STAG = 1 ;
        :BOTTOM-TOP_PATCH_END_STAG = 35 ;
        :GRID_ID = 1 ;
        :PARENT_ID = 0 ;
        :I_PARENT_START = 1 ;
        :J_PARENT_START = 1 ;
        :PARENT_GRID_RATIO = 1 ;
        :CEN_LAT = 26.99999f ;
        :CEN_LON = -145.f ;
        :TRUELAT1 = 30.f ;
        :TRUELAT2 = 60.f ;
        :MOAD_CEN_LAT = 26.99999f ;
        :STAND_LON = -145.f ;
        :POLE_LAT = 90.f ;
        :POLE_LON = 0.f ;
        :GMT = 0.f ;
        :JULYR = 2016 ;
        :JULDAY = 245 ;
        :MAP_PROJ = 3 ;
        :MAP_PROJ_CHAR = "Mercator" ;
        :MMINLU = "MODIFIED_IGBP_MODIS_NOAH" ;
        :NUM_LAND_CAT = 21 ;
        :ISWATER = 17 ;
        :ISLAKE = 21 ;
        :ISICE = 15 ;
        :ISURBAN = 13 ;
        :ISOILWATER = 14 ;
}```
burlen commented 5 years ago

MERRA - works

bin/test/test_cf_reader "/work2/data/teca/for_teca_data_svn/MERRA2/ARTMIP_MERRA_2D_20170218_.*\.nc$" "test_cf_reader_%t%.%e%" "0" "-1" "lon" "lat" "." "." "IVT"

contents of file:

$ncdump -h /work2/data/teca/for_teca_data_svn/MERRA2/ARTMIP_MERRA_2D_20170218_00.nc 
netcdf ARTMIP_MERRA_2D_20170218_00 {
dimensions:
    lat = 361 ;
    lon = 576 ;
variables:
    float PS(lat, lon) ;
        PS:time = 0. ;
        PS:vmin = -1.e+15f ;
        PS:vmax = 1.e+15f ;
        PS:fmissing_value = 1.e+15f ;
        PS:missing_value = 1.e+15f ;
        PS:_FillValue = 1.e+15f ;
        PS:units = "hPa" ;
        PS:long_name = "surface_pressure" ;
        PS:standard_name = "surface_pressure" ;
    double lat(lat) ;
        lat:standard_name = "latitude" ;
        lat:long_name = "latitude" ;
        lat:units = "degrees_north" ;
        lat:axis = "Y" ;
    double lon(lon) ;
        lon:standard_name = "longitude" ;
        lon:long_name = "longitude" ;
        lon:units = "degrees_east" ;
        lon:axis = "X" ;
    float IVT(lat, lon) ;
        IVT:time = 0. ;
        IVT:standard_name = "eastward_wind" ;
        IVT:long_name = "Integrated Water Vapor Transport" ;
        IVT:units = "kg/m/s" ;
        IVT:_FillValue = 1.e+15f ;
        IVT:missing_value = 1.e+15f ;
        IVT:fmissing_value = 1.e+15f ;
        IVT:vmax = 1.e+15f ;
        IVT:vmin = -1.e+15f ;
        IVT:lev = 1000. ;
        IVT:short_name = "IVT" ;
    float uIVT(lat, lon) ;
        uIVT:time = 0. ;
        uIVT:_FillValue = 1.e+15f ;
        uIVT:lev = 1000. ;
        uIVT:vmin = -1.e+15f ;
        uIVT:vmax = 1.e+15f ;
        uIVT:fmissing_value = 1.e+15f ;
        uIVT:missing_value = 1.e+15f ;
        uIVT:units = "kg/m/s" ;
        uIVT:long_name = "U-component of IVT" ;
        uIVT:standard_name = "eastward_wind" ;
        uIVT:short_name = "uIVT" ;
    float vIVT(lat, lon) ;
        vIVT:time = 0. ;
        vIVT:_FillValue = 1.e+15f ;
        vIVT:lev = 1000. ;
        vIVT:vmin = -1.e+15f ;
        vIVT:vmax = 1.e+15f ;
        vIVT:fmissing_value = 1.e+15f ;
        vIVT:missing_value = 1.e+15f ;
        vIVT:units = "kg/m/s" ;
        vIVT:long_name = "V-component of IVT" ;
        vIVT:standard_name = "eastward_wind" ;
        vIVT:short_name = "vIVT" ;
    float IWV(lat, lon) ;
        IWV:time = 0. ;
        IWV:_FillValue = 1.e+15f ;
        IWV:lev = 1000. ;
        IWV:vmin = -1.e+15f ;
        IWV:vmax = 1.e+15f ;
        IWV:fmissing_value = 1.e+15f ;
        IWV:missing_value = 1.e+15f ;
        IWV:units = "mm" ;
        IWV:long_name = "Integrated Water Vapor" ;
        IWV:standard_name = "eastward_wind" ;
        IWV:short_name = "IWV" ;
}
burlen commented 5 years ago

ERA5 -- not currently working, time axis is integer, teca expected a floating point.

$./bin/test/test_cf_reader "/work2/data/teca/for_teca_data_svn/ERA5/e5.*\.nc$" "test_cf_reader_%t%.%e%" "0" "-1" "longitude" "latitude" "." "time" "VIWVE"
ERROR: [0:140523024606976] [/home/bloring/work/teca/teca/io/teca_cf_reader.cxx:289 2.2.0]
ERROR: netcdf type code_ 4 is not a floating point type
ERROR: [0:140523024606976] [/home/bloring/work/teca/teca/io/teca_cf_reader.cxx:293 2.2.0]
ERROR: Failed to read variable "time" from "e5.oper.an.vinteg.162_071_viwve.regn320sc.20170217.nc". Unsupported data type
ERROR: [0:140522999428864] [/home/bloring/work/teca/teca/io/teca_cf_reader.cxx:289 2.2.0]
ERROR: netcdf type code_ 4 is not a floating point type
ERROR: [0:140522999428864] [/home/bloring/work/teca/teca/io/teca_cf_reader.cxx:293 2.2.0]
ERROR: Failed to read variable "time" from "e5.oper.an.vinteg.162_072_viwvn.regn320sc.20170217.nc". Unsupported data type
ERROR: [0:140523414570560] [/home/bloring/work/teca/teca/io/teca_cf_reader.cxx:768 2.2.0]
ERROR: Failed to read time axis
ERROR: [0:140523414570560] [/home/bloring/work/teca/teca/core/teca_index_executive.cxx:77 2.2.0]
ERROR: No index initializer key has been specified
ERROR: [0:140523414570560] [/home/bloring/work/teca/teca/core/teca_algorithm.cxx:686 2.2.0]
ERROR: failed to initialize the executive
ERROR: [0:140523414570560] [/home/bloring/work/teca/teca/core/teca_algorithm.cxx:710 2.2.0]
ERROR: failed to update port 0

contents of file

$ncdump -h /work2/data/teca/for_teca_data_svn/ERA5/e5.oper.an.vinteg.162_071_viwve.regn320sc.20170217.nc 
netcdf e5.oper.an.vinteg.162_071_viwve.regn320sc.20170217 {
dimensions:
    time = 4 ;
    latitude = 640 ;
    longitude = 1280 ;
variables:
    float VIWVE(time, latitude, longitude) ;
        VIWVE:_FillValue = 9.999e+20f ;
        VIWVE:long_name = "Vertical integral of eastward water vapour flux" ;
        VIWVE:short_name = "viwve" ;
        VIWVE:units = "kg m**-1 s**-1" ;
        VIWVE:original_format = "WMO GRIB 1 with ECMWF local table" ;
        VIWVE:ecmwf_local_table = 162 ;
        VIWVE:ecmwf_parameter = 71 ;
        VIWVE:grid_specification = "1280 longitude x 640 Gaussian latitude (0.281deg x ~0.281deg from 0E to 359.719E and 89.785N to 89.785S)" ;
        VIWVE:original_data_representation = "Reduced Gaussian grid (N320)" ;
        VIWVE:rda_dataset = "ds630.0" ;
        VIWVE:rda_dataset_url = "https:/rda.ucar.edu/datasets/ds630.0/" ;
        VIWVE:rda_dataset_doi = "DOI: 10.5065/D6X34W69" ;
        VIWVE:rda_dataset_group = "ERA5 atmospheric vertical integrals (ECMWF Table 162)" ;
        VIWVE:number_of_significant_digits = 7 ;
    double latitude(latitude) ;
        latitude:_FillValue = NaN ;
        latitude:long_name = "latitude" ;
        latitude:short_name = "lat" ;
        latitude:units = "degrees_north" ;
        latitude:qualifier = "Gaussian" ;
    double longitude(longitude) ;
        longitude:_FillValue = NaN ;
        longitude:long_name = "longitude" ;
        longitude:short_name = "lon" ;
        longitude:units = "degrees_east" ;
    int time(time) ;
        time:long_name = "time" ;
        time:units = "hours since 1900-01-01" ;
        time:calendar = "gregorian" ;
    int utc_date(time) ;
        utc_date:long_name = "UTC date yyyy-mm-dd hh:00:00 as yyyymmddhh" ;
        utc_date:units = "Gregorian_year month day hour" ;

// global attributes:
        :DATA_SOURCE = "ECMWF: ERA5 GRIB 1 spherical harmonics (T639) and reduced Gaussian grid data (N320)." ;
        :DATA_INTERPOLATION = "CISL RDA: ERA5 GRIB 1 N320/T639 data interpolated (or synthesized) to 1280 longitude x 640 Gaussian latitude GRIB 1 data using emoslib-4.3.9 (compiled with cmake with grib_api-1.14.7 and fftw-3.3.4 support for gnu/gfortran-4.4.7)." ;
        :NETCDF_CONVERSION = "CISL RDA: Conversion from interpolated/synthesized GRIB 1 data to netCDF4." ;
        :NETCDF_VERSION = "4.3.0" ;
        :CONVERSION_PLATFORM = "Linux geyser01 2.6.32-358.el6.x86_64 #1 SMP Wed Nov 2 11:00:18 MDT 2016 x86_64 x86_64 x86_64 GNU/Linux" ;
        :CONVERSION_DATE = "Mon Feb 12 17:38:07 MST 2018" ;
        :Conventions = "CF-1.6" ;
        :NETCDF_COMPRESSION = "NCO: Precision-preserving compression to netCDF4/HDF5 (see \"history\" and \"NCO\" global attributes below for specifics)." ;
        :history = "Mon Feb 12 17:39:18 2018: ncks -4 --ppc default=7 e5.oper.an.vinteg.162_071_viwve.regn320sc.2017020100_2017022823.unc.nc e5.oper.an.vinteg.162_071_viwve.regn320sc.2017020100_2017022823.nc" ;
        :NCO = "4.6.9" ;
}
burlen commented 5 years ago 
ncdump: /work2/data/teca/for_teca_data_svn/ERA5/e5.oper.an.vinteg.162_071_viwve.regn320sc.20170217.nc: NetCDF: Unknown file format

these issues may be that fedora's default netcdf is not hdf5 based? Or (see travis comment) a backward incompatibilty.

burlen commented 5 years ago

Running with netcdf from the TECA_Superbuild, netcdf can read the files

taobrienlbl commented 5 years ago

Okay, glad you found that. I've also recently encountered issues at NERSC where it seems like there might be some backwards-incompatibility issues across versions of netCDF+HDF5: but good that the superbuild version works.

burlen commented 5 years ago

@taobrienlbl do you have any info on the coordinate system used by WRF?Looks like a logically Cartesian hexahedral mesh (aka curvilinear mesh)?

variables:
    float XLAT(south_north, west_east) ;
        XLAT:_FillValue = NaNf ;
        XLAT:FieldType = 104 ;
        XLAT:MemoryOrder = "XY " ;
        XLAT:description = "LATITUDE, SOUTH IS NEGATIVE" ;
        XLAT:units = "degree_north" ;
        XLAT:stagger = "" ;
    float XLONG(south_north, west_east) ;
        XLONG:_FillValue = NaNf ;
        XLONG:FieldType = 104 ;
        XLONG:MemoryOrder = "XY " ;
        XLONG:description = "LONGITUDE, WEST IS NEGATIVE" ;
        XLONG:units = "degree_east" ;
        XLONG:stagger = "" ;
taobrienlbl commented 5 years ago

In this case, WRF is using a regional Mercator projection (see the :MAP_PROJ_CHAR = "Mercator" ; property in the global metadata, which is standard WRF metadata output), which I believe is actually rectilinear. However, WRF supports a variety of curvilinear coordinate systems, and so it uses a generic coordinate system in the netCDF files that can accommodate both rectilinear and curvilinear.

Unless I am mistaken, I think that we can safely assume that WRF grids will always be logically cartesian, and we should assume that there is a nonlinear mapping between the south_north/west_east indices and the actual lat/lon values (i.e., that lat/lon should be treated as 2D grids). In this case, lat/lon are actually also arranged in a logically cartesian fashion, but WRF simulations often don't use the Mercator projection.

burlen commented 5 years ago

I think supporting WRF is going to require a new "curvilinear" mesh type.

taobrienlbl commented 5 years ago

Maybe one approach could be to support the rectilinear mesh type in this simulation with something like a '--wrf_rectilinear' option? We could then add the curvilinear mesh support to our list of issues to tackle down the line.

burlen commented 5 years ago

Yes, that sounds like a reasonable approach as re-implementing all the algorithms that rely on mesh coordinates might be a heavy lift. We might want to make a list of these and figure out what changes would be needed on a case by case too.

to implement your --wrf_recilinear` option, we might add the curvilinear mesh type, it's essentially structurally equivalent to the recilinear mesh, and at the same time add an algorithm that would be used to convert from curvilinear to rectilinear type, when the flag is passed to an application.

maybe we can brain storm some more at our next non-cascade meeting

taobrienlbl commented 5 years ago

That sounds good to me.

On Fri, Mar 22, 2019 at 2:08 PM Burlen Loring notifications@github.com wrote:

Yes, that sounds like a reasonable approach as re-implementing all the algorithms that rely on mesh coordinates might be a heavy lift. We might want to make a list of these and figure out what changes would be needed on a case by case too.

to implement your --wrf_recilinear` option, we might add the curvilinear mesh type, it's essentially structurally equivalent to the recilinear mesh, and at the same time add an algorithm that would be used to convert from curvilinear to rectilinear type, when the flag is passed to an application.

maybe we can brain storm some more at our next non-cascade meeting

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LBL-EESA/TECA/issues/177#issuecomment-475783525, or mute the thread https://github.com/notifications/unsubscribe-auth/AIY6FvC0V2-uakpn0XfXSirgYTEDlIFGks5vZUYygaJpZM4cCjGt .

-- Travis A. O'Brien

http://eesa.lbl.gov/profiles/travis-a-obrien/ +1 (510) 495-8047

areminusfarestar commented 5 years ago

Wrf output can also have the spatial dimensions be a function of time. Wrf can use both nested and time varying spatial coordinates (for moving high res nests)). Best to build in some checks. MH

On Fri, Mar 22, 2019 at 4:31 PM taobrienlbl notifications@github.com wrote:

In this case, WRF is using a regional Mercator projection (see the :MAP_PROJ_CHAR = "Mercator" ; property in the global metadata, which is standard WRF metadata output), which I believe is actually rectilinear. However, WRF supports a variety of curvilinear coordinate systems, and so it uses a generic coordinate system in the netCDF files that can accommodate both rectilinear and curvilinear.

Unless I am mistaken, I think that we can safely assume that WRF grids will always be logically cartesian, and we should assume that there is a nonlinear mapping between the south_north/west_east indices and the actual lat/lon values (i.e., that lat/lon should be treated as 2D grids). In this case, lat/lon are actually also arranged in a logically cartesian fashion, but WRF simulations often don't use the Mercator projection.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/LBL-EESA/TECA/issues/177#issuecomment-475771702, or mute the thread https://github.com/notifications/unsubscribe-auth/AQTJLZX-P9c2tPrNFnQMzD9GC0ITfekBks5vZT2jgaJpZM4cCjGt .

taobrienlbl commented 5 years ago

Good point--thanks!

@burlen, we'll have to decide whether we want to support time-varying coordinates as well when we meet. I'm inclined to say no, due to the cost-benefit of implementing that. I expect little benefit, since from what I understand, such grids are often used so that the WRF grid itself is tracking a weather feature like a TC: which would sort of defeat the purpose of TECA.

burlen commented 5 years ago

@areminusfarestar @taobrienlbl do you know of a reference document describing wrf output capabilities?

without a use case for the time varying coordinates, would probably be best to leave it to the future for now. On the other hand, if there's a user in need of this feature for something specific, that would likely make it a higher priority.

to make it happen we'd need to re-arrange some of the metadata caching and reading code so that the time axis remains cached, but the spatial coordinates are not.

taobrienlbl commented 5 years ago

Off the top of my head, no, though I'm sure there must be something. The technical documentation doesn't describe it. I did find an old (16 years) description of the I/O infrastructure that might have some relevant information, but it looks like it is more focused on implementation details: http://www2.mmm.ucar.edu/wrf/WG2/software_2.0/io.pdf

The user's guide also has a bit of information, but I'm not sure that's quite what we want either: http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V4/WRFUsersGuide.pdf

On Mon, Mar 25, 2019 at 11:19 AM Burlen Loring notifications@github.com wrote:

@areminusfarestar https://github.com/areminusfarestar @taobrienlbl https://github.com/taobrienlbl do you know of a reference document describing wrf output capabilities?

without a use case for the time varying coordinates, would probably be best to leave it to the future for now. On the other hand, if there's a user in need of this feature for something specific, that would likely make it a higher priority.

to make it happen we'd need to re-arrange some of the metadata caching and reading code so that the time axis remains cached, but the spatial coordinates are not.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LBL-EESA/TECA/issues/177#issuecomment-476319664, or mute the thread https://github.com/notifications/unsubscribe-auth/AIY6FsNd2H_uUWpGcy6Y_TMaREcuFsSvks5vaRNDgaJpZM4cCjGt .

-- Travis A. O'Brien

http://eesa.lbl.gov/profiles/travis-a-obrien/ +1 (510) 495-8047

burlen commented 4 years ago

Emily shared another WRF dataset, /work2/data/teca/wrf. similar to the one above

burlen commented 3 years ago

all the features we talked about in this issue have been implemented and are working, including time varying coordinates in WRF data.