search

Unidata / netcdf-java

The Unidata netcdf-java library
https://docs.unidata.ucar.edu/netcdf-java/current/userguide/index.html
BSD 3-Clause "New" or "Revised" License
146 stars 71 forks source link

CF-Radial not recognized as radial feature type. #207

Open JohnLCaron opened 4 years ago

JohnLCaron commented 4 years ago

Add a Convention parser that handles CF-Radial Convention. Currently there is none and those files are not recognized as radial data types.

The test files are in ~cdmUnitTest/conventions/cfradial. Are these current or are there more recent examples? Has the spec evolved? Is it being used?

To do this we need a medium/long range strategy for radial feature types. What should the API look like? Is there feedback from MetPy/Siphon and the radar experts?

ethanrd commented 4 years ago

Hey @JohnLCaron - There is now a 1.4 CfRadial spec (which I don't think has changed since 2016 or 2017) and a 2.0 CfRadial spec. The 2.0 spec uses groups and a few other netCDF Enhanced Data Model features. The docs are in the NCAR/CfRadial GitHub repo. @mike-dixon should be able to give you an idea of current usage of the 2.0 spec and maybe some example files. -- Cheers

dopplershift commented 4 years ago

Right now I think the only users of the radial feature types is the IDV, at least in-house. I think we need a longer look at what we want these feature types to actually accomplish before going down the path further. My recollection is last time I looked, in order to get a radial feature type, one had to first see if that makes sense for the data file and then coerce it.

JohnLCaron commented 4 years ago

With the exception of Grid and GRIB, these feature types seem mostly unused, so theres a question if we should keep developing them. Long ago we had a query service where users could request particular radar scans by time range, station, and ? from radar collections like nexrad2 and 3. Is something along those lines useful to anyone?

dopplershift commented 4 years ago

That query service still exists and is very useful, but it currently doesn’t (and didn’t at the time that I rewrote it) rely on any of the radial feature classes. What I would love, though, is to be able to treat a set of radial data on disk as a single collection that I didn’t need to query, but could treat no different than our GRIB collections.

JohnLCaron commented 4 years ago

The problem is its hard to do index based aggregation on large collection of data, unless they are rigidly homogenous in structure. That leads to using "coordinate range querying" (better name?) which is what feature types were trying to get to. Can you say exaclty what properties you want from of a "radial data collection"? What would an API or query look like? How does that differ from what you already can do?

dopplershift commented 4 years ago

It's not really about what I want in terms of a service or query API. What I want is a consistent way I can train our users to access data. It's not great in a workshop to show users that we have one way to access model data (i.e. go through the feature collection and maybe use NCSS), one way to access radar data (use the radar query service, which gives you a list of files, then access one-by-one using opendap or something), and then a third way to access satellite data (list the catalog and pray that there's a sensible naming structure, which we actually don't currently have).

While I understand why the differences between the data sources exists, that doesn't help users with confusion. I think it's a gigantic usability problem that the process of selecting datasets of interest is tightly coupled to the type of data source. My dream is to be able to teach a workshop integrating multiple datasets without spending half the day talking about the different ways we access each of those datasets.

So I guess to answer "What do I want?", I would say: give me a way to do at least a time-based query off a single end-point on the catalog, that users find just like the GRIB collections and works just like it. I realize that's not at all specific to radial, but I feel like that's such a more important problem than designing a class that provides access to data in a polar coordinate system.

ArnaudDumont commented 4 years ago

Has there been any more thought to adding support for CF-Radial to the CDM? Even better, is there any hope that CF-Radial would ever be supported in TDS for rendering though ncWMS? Longshot, maybe, but it seems that if the netCDF library could do the coordinate transformation to produce a cartesian coverage, ncWMS would be able to render it. We'd really like to generate radar imagery on-the-fly without having to chain a series of preprocessors and custom display engines like we do now.