search

ioos / system-test

IOOS DMAC System Integration Test project
github.com/ioos/system-test/wiki
The Unlicense
7 stars 14 forks source link

Problematic time units from PMEL #146

Open wckoeppen opened 10 years ago

wckoeppen commented 10 years ago

Perhaps tangential to the discovery phase (more in the analysis phase), but I've been finding that when accessing climate forecast models from PMEL I have to manually set the time units in order to use the netCDF module.

Apparently PMEL is setting their time units to: sec since YYYY-MM-DD HH:MM:SS

which causes netCDF4.num2date to choke because it only accepts "days, hours, minutes, seconds, milliseconds or microseconds", not "sec".

Common type of problem?

wckoeppen commented 10 years ago

I guess it is common. Apparently we modify nearly every time unit string we get from data providers (using ncml). Great!

rsignell-usgs commented 10 years ago

"we" meaning Axiom, right? Do you also report these errors back to the data providers (e.g. PMEL)?

wckoeppen commented 10 years ago

I wish I could say that was the regular policy. The reality is that it takes us 5 minutes to change the ncml files and correct problems like this, whereas reporting errors, waiting for a response (if one ever comes), and then accepting new files takes significantly longer. So we generally only send reports back to data providers if something is unintelligible or unusable. For one-off (probably not the right word) models like the ones I'm using from PMEL, AOOS may be the only web-accessible host anyway.

dpsnowden commented 10 years ago

I'm not sure I understand the "problem". The use of time units as described above is the CF convention. See http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#time-coordinate. The udunits.dat library accepts [sec(s), s, second(s)] since YYYY-MM-DD HH:MM:SS. If the netCDF4.num2date function only accepts a subset of those options I don't think it's an error on the data providers part, it just means that netCDF4 didn't implement the full specification.

How should we document this for the final report? Is the issue closed?

wckoeppen commented 10 years ago

Fair enough, as I said, it's probably tangential to data discovery. But it has been problematic for me for actual data analysis. Maybe we can put in a request for the netCDF library to be improved? Meanwhile, we'll just keep fixing the units so they'll work with the available tools.

dpsnowden commented 10 years ago

OK, I'd like remember this for the final report and describe it as a "nice to have" extension to client software. Not sure where the netCDF4 library is developed but I would hope they have a way to request enhancements. @rsignell-usgs ?

jkupiec commented 10 years ago

I have added the Enhancement label so that we will know to list this item in the final report.

rsignell-usgs commented 10 years ago

This is not a problem with NetCDF4. The problem is a provider issue -- "sec" is not an allowable UDUNIT, and therefore this data is not CF compliant. The only acceptable way to specify seconds as a base is as "second" or "s"

From there you can append prefixes such as "milli", or "micro", but "sec" is NOT allowed.