search

aodn / content

Tracks AODN Portal content and configuration issues
0 stars 0 forks source link

For some SRS files the indexed time does not match the actual time recorded in the file #242

Closed jonescc closed 7 years ago

jonescc commented 8 years ago

For example:

http://thredds.aodn.org.au/thredds/catalog/IMOS/SRS/SST/ghrsst/L3U-S/n19/2009/catalog.html?dataset=IMOS/SRS/SST/ghrsst/L3U-S/n19/2009/20090224192314-ABOM-L3U_GHRSST-SSTskin-AVHRR19_D-Des_Southern.nc

date/time recorded in the database (and in the file name) is "2009-02-24 19:23:14+00" whereas actual time recorded in the file is "2009-02-24T19:23:15.000000"

Causes issues for GoGoDuck and ncwms abomination

julian1 commented 8 years ago

This shows that completing the reconciliation stuff is valuable in terms of uncovering potential issues.

jonescc commented 8 years ago

Another example from https://github.com/aodn/content/issues/240:

http://thredds.aodn.org.au/thredds/catalog/IMOS/SRS/SST/ghrsst/L3U/n19/2009/catalog.html?dataset=IMOS/SRS/SST/ghrsst/L3U/n19/2009/20090225191220-ABOM-L3U_GHRSST-SSTskin-AVHRR19_D-Des.nc

lbesnard commented 8 years ago

@jonescc if we want to fix this issue (before putting everything on the generic harvester) we would have to push around 600Gb of data to INCOMING. is is worth it ?

jonescc commented 8 years ago

I'm happy with disabling the collection if you think its too much to push through - I've no idea how long it would take to do this. I assume there would be no cost implications if it was all done on aws.

lbesnard commented 7 years ago

closing because this is being ingested in the generic timestep harvester