Add observational data to the catalog

rsignell-usgs commented 10 years ago

We need to get the observational data added to the new COMT catalog.

I'm not sure where we were with this process on the old testbed-dev server, and I'm not sure who has the corporate memory.

I believe that at least some of the original text files were converted into CF-1.6 timeSeries files or timeSeries collections.

@daf or @birdage, do you know where this data is?

@corbittkerr and I want to make it available on the testbed archive thredds server (http://comt.sura.org/thredds)

Thanks, Rich

brianmckenna commented 10 years ago

Rich,

There does appear to be some data on testbedapps.sura.org, see

/data/ftp/upload/Inundation/observations/tropical/netcdf/Ike/NOAA

for a few examples.

Is this what you're after?

rsignell-usgs commented 10 years ago

Brian, Thanks! Do we know whether these are the latest netcdf files? These do appear to be CF-1.6 compliant (at least they say they are) -- we can test them with ncSOS to make sure.

These are individual time series datasets, which is fine, but I thought we also had some time series collections (essentially a bunch of time series together in one NetCDF file). Or am I dreaming?

Thanks, Rich

brianmckenna commented 10 years ago

Rich, can't say if you're dreaming or not. I'm unaware of the collections, but as I came on in Jan13 that doesn't necessarily mean they don't exist. Let me poke around a bit with Andy who was more familiar with the IMEDS territory, there might be some indication in there how to find those.

Would the collections just be NcML?

rsignell-usgs commented 10 years ago

I did find that this thredds directory: http://testbedapps-dev.sura.org/thredds/inundation.html there are two directories called "IMEDS simulations station output" and "IMEDS observation station measurements" that contain CF-1.6 compliant NetCDF files with SOS endpoints.

Alex chimed in on an e-mail: "As far as I can remember there are two different sets of obs data converted to netcdf. Kyle did the NOAA ones I think, and I did the others (which I believed were the ones that the modelers were actually using to do their skill assessment and those are that which are SOS enabled). My code was hacky at best but is available on my github: https://github.com/acrosby/imeds2netcdf_crawler Andy Bird at ASA may be able to put something much nicer together as he has been working on Imeds itself and did the original work on NCSOS."

birdage commented 10 years ago

Hi Rich,

I remember alex doing the generation of the Netcdf files he mentioned, but I was not directly involved in their creation. I was however very involved with ncsos and imeds. For imeds both observation and model data is required at the same location, hence the simulation and observation station measurements, the files would of been created from plain text files. Both the obs and model data sets contain multiple station time series.

Andy

A member of the RPS Group plc

From: Rich Signell notifications@github.com<mailto:notifications@github.com> Reply-To: ioos/comt_catalog reply@reply.github.com<mailto:reply@reply.github.com> Date: Tue, 4 Feb 2014 09:00:42 -0500 To: ioos/comt_catalog comt_catalog@noreply.github.com<mailto:comt_catalog@noreply.github.com> Cc: RPS Developer abird@asascience.com<mailto:abird@asascience.com> Subject: Re: [comt_catalog] Add observational data to the catalog (#11)

I did find that this thredds directory: http://testbedapps-dev.sura.org/thredds/inundation.html there are two directories called "IMEDS simulations station output" and "IMEDS observation station measurements" that contain CF-1.6 compliant NetCDF files with SOS endpoints.

Alex chimed in on an e-mail: "As far as I can remember there are two different sets of obs data converted to netcdf. Kyle did the NOAA ones I think, and I did the others (which I believed were the ones that the modelers were actually using to do their skill assessment and those are that which are SOS enabled). My code was hacky at best but is available on my github: https://github.com/acrosby/imeds2netcdf_crawler Andy Bird at ASA may be able to put something much nicer together as he has been working on Imeds itself and did the original work on NCSOS."

— Reply to this email directly or view it on GitHubhttps://github.com/ioos/comt_catalog/issues/11#issuecomment-34061256.

rsignell-usgs commented 9 years ago

Bringing @jsignell (my daughter Julia) in here. She is going to try to figure this obs stuff out and make it accessible on the comt.sura.org/thredds server.

rsignell-usgs commented 9 years ago

I transferred the inundation obs data from the old testbed server testbedapps.sura.org to the current testbed server comt.sura.org. This is exactly what I did:

ssh root@testbedapps.sura.org
rsync -aHAX /data/ftp/upload/Inundation/observations /data1/comt_1_archive/inundation_tropical

which makes them available on comt.sura.org at /data/comt_1_archive/inundation_tropical/observations. I changed the owner:group to be testbed:testbed for all the obs files.

@jsignell, can you look over all these files (and the previous information in this issue) and see if you can figure out what is already in CF-1.6 netcdf files, and what needs to be converted?

rsignell-usgs commented 9 years ago

BTW, we have a THREDDS datasetScan on all .nc, .ncml, .docx, and .txt files residing in the /data directory, so if you want to see the THREDDS view of this folder, you can go to: http://comt.sura.org/thredds/catalog/comt_1_archive_full/inundation_tropical/observations/catalog.html

ioos / comt_catalog

Add observational data to the catalog #11