search

aodn / content

Tracks AODN Portal content and configuration issues
0 stars 0 forks source link

VIT - Corrupt file stops aggregations for srs_oc_chl_oc3_1d_aqua_url #417

Closed jonescc closed 5 years ago

jonescc commented 5 years ago

Refer https://github.com/aodn/issues/issues/425

A.P1D.20180106T000000Z.aust.chl_oc3.nc is corrupt and stops any aggregations which include it.

ggalibert commented 5 years ago

@lbesnard there should be a sanity check added to the relevant pipeline.

lbesnard commented 5 years ago

@jonescc what is corrupted about this file ? http://thredds.aodn.org.au/thredds/dodsC/IMOS/SRS/OC/gridded/aqua/P1D/2018/01/A.P1D.20180106T000000Z.aust.chl_oc3.nc.html

I can access it, open it, plot the data

lbesnard commented 5 years ago

@lbesnard there should be a sanity check added to the relevant pipeline.

The sanity check is automatically added via aodncore. Also, there is a CF check https://github.com/aodn/chef-private/blob/master/data_bags/imos_po_watches/SRS_OC_GRIDDED.json#L8

ggalibert commented 5 years ago

If you do an ncdump on the file you will get:

NetCDF: HDF error
Location: file /build/netcdf-StLR0y/netcdf-4.4.0/ncdump/vardata.c; line 473

Maybe the sanity check in aodncore could be improved to better pick up corrupted files?

lbesnard commented 5 years ago

nothing wrong with the pipeline. This is a file prior to v2

lbesnard commented 5 years ago

If you do an ncdump on the file you will get:

that what i did, and then it outputs all the data. Same with ncBrowse, I can plot everything

ggalibert commented 5 years ago

I ran ncdump on 10-aws.

lbesnard commented 5 years ago

maybe i have a more up ti date version of the netcdf lib

jonescc commented 5 years ago

Yeah, I was wondering that. Just rechecking locally and elsewhere.

ghost commented 5 years ago

Just to confirm, if no compliance checks are defined, Pipeline 2 does a minimal NetCDF validation, so blatantly corrupt files shouldn't make it through the pipeline.

jonescc commented 5 years ago

Have you read all the data from the file? One chunk is probably corrupt.

mhidas commented 5 years ago

We've encountered this problem before: https://github.com/aodn/issues/issues/231#issuecomment-402320120 - see the discussion in that thread. If only a data chunk is corrupt, doing ncdump -h or opening the file as a netCDF4.Dataset, or even running CF & IMOS checks may not show this problem.

We might need a separate data integrity check: https://github.com/aodn/cc-plugin-imos/issues/17

lbesnard commented 5 years ago

30/04/19 Edward King contacted

lbesnard commented 5 years ago

fixed. I ran the following aggregation to make sure of it https://processes.aodn.org.au/wps/jobStatus?jobId=d409ee6b-a218-4de1-90ff-79cc26828fe2&format=HTML