LBL-EESA / TECA

TECA, theToolkit for Extreme Climate Analysis, contains a collection of climate anlysis algorithms targetted at extreme event detection and analysis.
Other
57 stars 21 forks source link

ability to add extra non-spatial vars to cf_writer #261

Closed taobrienlbl closed 5 years ago

taobrienlbl commented 5 years ago

@burlen - in looking at teca_cf_writer.cxx, it appears that there is not currently an easy way to add extra, non-spatial variables to the output. Do you agree?

To be specific, here is the ncdump -h output of a netCDF file from #258, and below are the additional fields that I would want to add:

netcdf bayesian_ar_detect_0 {
dimensions:
        time = UNLIMITED ; // (8 currently)
        lat = 160 ;
        lon = 320 ;
variables:
        double time(time) ;
                time:calendar = "proleptic_gregorian" ;
                time:long_name = "initial time" ;
                time:units = "hours since 1959-01-01" ;
        float lat(lat) ;
                lat:Di = 1.125f ;
                lat:GridType = "Gaussian Latitude/Longitude Grid" ;
                lat:La1 = 89.142f ;
                lat:La2 = -89.142f ;
                lat:Lo1 = 0.f ;
                lat:Lo2 = 358.875f ;
                lat:N = 80.f ;
                lat:long_name = "latitude" ;
                lat:units = "degrees_north" ;
        float lon(lon) ;
                lon:Di = 1.125f ;
                lon:GridType = "Gaussian Latitude/Longitude Grid" ;
                lon:La1 = 89.142f ;
                lon:La2 = -89.142f ;
                lon:Lo1 = 0.f ;
                lon:Lo2 = 358.875f ;
                lon:N = 80.f ;
                lon:long_name = "longitude" ;
                lon:units = "degrees_east" ;
        float ar_probability(time, lat, lon) ;
                ar_probability:long_name = "posterior AR flag" ;
                ar_probability:units = "probability" ;

Additional fields to add:

        int mcmc_sample_index(mcmc_sample_index) ;
        float area_threshold(mcmc_sample_index) ;
                area_threshold:long_name = "Area Threshold" ;
                area_threshold:units = "m^2" ;
        float filter_width(mcmc_sample_index) ;
                filter_width:long_name = "Gaussian Filter Width" ;
                filter_width:units = "degrees" ;
                filter_width:note = "width is half-height-half-mass" ;
        float segmentation_threshold(mcmc_sample_index) ;
                segmentation_threshold:long_name = "Segmentation Threshold" ;
                segmentation_threshold:units = "percentile" ;
        int ar_count(time, mcmc_sample_index) ;
                ar_count:long_name = "AR Detections" ;
                ar_count:units = "count" ;

I think what prevents this right now is that the additional variables have a non-spatial dimension: mcmc_sample_index. (This dimension corresponds to the index of parameter values in teca_bayesian_ar_detect_parameters.cxx.) The current cf_writer code creates only spatial dimensions, and it doesn't appear to have a generalizeable way to create new ones. Relatedly, it doesn't look like there is currently a way to add variables to the arrays argument in teca_cf_writer_internals::write() that have non-spatial dimensions.

Do you have a recommendation for how we might approach this?

One thought that comes to mind is that we could add two new arguments to teca_cf_writer_internals::write(): extra_arrays and extra_arrays_metadata. The extra_arrays_metadata argument would contain the information necessary to create the non-spatial dimensions/variables. extra_arrays_metadata would look something like this:

{
 dims = {mcmc_sample_index = 1022},
 variables = 
 {
  mcmc_sample_index = 
  {
   dims = "mcmc_sample_index",
   attributes = {}
  }
  area_threshold = 
  {
    dims = "mcmc_sample_index",
    attributes = 
    {
     long_name = "Area Threshold",
     units = "m^2"
    }
  },
  filter_width = 
  {
    dims = "mcmc_sample_index",
    attributes = 
    {
     long_name = "Gaussian Filter Width",
     units = "degrees",
     note = "width is half-height-half-mass"
    }
  },
...
 },
}

What do you think @burlen?

burlen commented 5 years ago

sure we can add capability to write non-geometric data. I have questions for you, probably more suitable for a face to face. Can we meet early next week?

taobrienlbl commented 5 years ago

@burlen - cool, and yes, I figured this might benefit from some verbal discussion. I'll follow up in e-mail for a time to meet next week.

burlen commented 5 years ago

solved in #285