search

Reading-eScience-Centre / edal-java

Environmental Data Abstraction Layer libraries
Other
39 stars 30 forks source link

can't handle an ncml aggregated using dateFormatMark #147

Closed WeatherGod closed 1 year ago

WeatherGod commented 1 year ago

I have a large, monthly climatology to serve out using thredds. Because it is so large, I am having thredds determine its time coordinate using dateFormatMark in order to avoid having it read each file for its time coordinate value. The OpenDAP, WCS, and ncml services all handle this properly, but the WMS service is failing because edal's CdmDatasetFactory is somehow choking on this.

This is what comes up in my threddsServlet.log:

2022-10-20T20:50:22.515 +0000 [    141662][      16] WARN  - uk.ac.rdg.resc.edal.wms.WmsServlet - Wms Exception caught: "Problem creating dataset gwsc/gdm at gwsc/gdm" from:uk.ac.rdg.resc.edal.dataset.cdm.CdmDatasetFactory:133 Cause: No grids found in underlying NetCDF dataset

I'd welcome any suggestions on how to debug this further. I really don't understand how edal handles netcdf data differently than everyone else does. If I don't use dateFormatMark and let thredds use the time information already contained in the data files (via a joinExisting), then everything works, but indexing is significantly slower.

The relevant part of my ncml:

    ...
      <aggregation dimName="time" type="joinNew" recheckEvery="15 min">
        <variableAgg name="FDI"/>
        <variableAgg name="spei01"/>
        <variableAgg name="spei03"/>
        <variableAgg name="spei12"/>
        <scan location="/data/gdm/drought" subdirs="true" regEx="GDMv4_[0-9]{6}\.nc$"
            dateFormatmark="GDMv4_#yyyyMM" />
      </aggregation>
    ...
WeatherGod commented 1 year ago

... and I just noticed my typo of dateFormatmark instead of dateFormatMark. Testing things out with that corrected, but that would be really strange if the other services didn't care, but WMS did care about that?

WeatherGod commented 1 year ago

Now I'm really confused. Correcting the typo yields working behavior, but the ncml and opendap service is reporting that the grids have dimensions of "time time lat lon"??? I'm wondering if the other services weren't actually working as I expected when I had the typo and created a time dimension with filename strings, and the WMS capabilities doc was balking on that since filenames aren't datetimes.

Anyway, this seems to be entirely in thredds' ballpark, and edal is working as expected. I'll close this issue. Looks like I rubber-ducked-solved this myself!