search

aodn / content

Tracks AODN Portal content and configuration issues
0 stars 0 forks source link

Problematic ACORN DM files #459

Closed ocehugo closed 3 years ago

ocehugo commented 4 years ago

The following ACORN DM files are errors and we don't have them yet:

date time filename
2020-03-28 09:14:06.024959630 IMOS_ACORN_RV_20180105T144000Z_CWI_FV01_radial.nc
2020-03-27 19:38:31.132059356 IMOS_ACORN_RV_20180914T154500Z_CSP_FV01_radial.nc
2020-04-15 18:10:10.747370215 IMOS_ACORN_RV_20190111T095500Z_FRE_FV01_radial.nc
2019-12-11 22:00:08.720899711 IMOS_ACORN_RV_20191014T020000Z_DONG_FV01_radial.nc

The first file (CWI) appears OK, but the rest are failing at db ingestion/talend (date related). @lbesnard - any ideas?

The file below failed because of an incomplete (or no) global attributes. The file is currently published. @scosoli

date time filename
2019-06-04 14:48:36.120441805 IMOS_ACORN_V_20090327T090000Z_TURQ_FV00_1-hour-avg.nc

The files below failed for the same reason above, but they are not published. @scosoli

date time filename
2019-09-26 16:20:13.654942707 IMOS_ACORN_V_20091116T060000Z_TURQ_FV01_1-hour-avg.nc
2019-10-07 15:06:57.929240358 IMOS_ACORN_V_20091116T060000Z_TURQ_FV01_1-hour-avg.nc
scosoli commented 4 years ago

@yashahoo can you check CWI CSP FRE DONG FV01_radials? I will check TURQ data

yashahoo commented 4 years ago

i will look into it.

On 3 Jun 2020, at 3:49 pm, scosoli notifications@github.com wrote:

@yashahoo https://github.com/yashahoo can you check CWI CSP FRE DONG FV01_radials? I will check TURQ data

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aodn/content/issues/459#issuecomment-638024744, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIA7YXUDXBKX4SRAVKALDSDRUX57BANCNFSM4NPTRIOA.

scosoli commented 4 years ago

@ocehugo, I don't see how IMOS_ACORN_V_20090327T090000Z_TURQ_FV00_1-hour-avg.nc has global attributes different from or missing in comparison to other files. Say, compare for instance with IMOS_ACORN_V_20090327T100000Z_TURQ_FV00_1-hour-avg.nc or IMOS_ACORN_V_20090327T110000Z_TURQ_FV00_1-hour-avg.nc. can you provide more details?

scosoli commented 4 years ago

if I ncdump -h the suspect and the following file, the only differences I get are the following: A diff between the two ncdump of the two files reports:

< title: IMOS ACORN Turqoise Coast (TURQ), one hour averaged current RT-QC data, 2009-03-27T09:00:00Z\

title: IMOS ACORN Turqoise Coast (TURQ), one hour averaged current RT-QC data, 2009-03-27T10:00:00Z\ 15,16c15,16 < date_created: 2019-06-04T09:38:57Z\ < abstract: Sea State data, CODAR Ocean Sensors. SeaSonde sea surface radar located at Turquoise Coast (WA), for time 2009-03-27T09:00:00Z. Sea State represents the surface sea water state obtained by merging the radial surface sea water state components from two or more radar stations. Data produced by Australian Coastal Ocean Radar Network, Integrated Marine Observing System.\

date_created: 2019-06-04T09:38:59Z\ abstract: Sea State data, CODAR Ocean Sensors. SeaSonde sea surface radar located at Turquoise Coast (WA), for time 2009-03-27T10:00:00Z. Sea State represents the surface sea water state obtained by merging the radial surface sea water state components from two or more radar stations. Data produced by Australian Coastal Ocean Radar Network, Integrated Marine Observing System.\ 33,34c33,34 < time_coverage_start: 2009-03-27T09:00:00Z\ < time_coverage_end: 2009-03-27T09:00:00Z\

time_coverage_start: 2009-03-27T10:00:00Z\ time_coverage_end: 2009-03-27T10:00:00Z\

that is, differences between global attributes is in the creation time stamps, and in the time coverage. I do not see any major differences other than those -expected- reported above

yashahoo commented 4 years ago

@ocehugo I don't see a problem with IMOS_ACORN_RV_20180105T144000Z_CWI_FV01_radial.nc either and have re-uploaded the file I have locally. These 2 files : IMOS_ACORN_RV_20180914T154500Z_CSP_FV01_radial.nc IMOS_ACORN_RV_20190111T095500Z_FRE_FV01_radial.nc did not get uploaded correctly when they were recently reprocessed due to an error on our end and thus were bad files with a much older time stamp compared to adjacent files in AODN, so perhaps this caused the problem? I have uploaded the new corrected versions today. So hopefully this resolves the issue.

@scosoli, could you check the DONG file? I think you created that radial file.

ocehugo commented 4 years ago

Thanks guys, the only missing now is the DONG one.

@ocehugo, I don't see how IMOS_ACORN_V_20090327T090000Z_TURQ_FV00_1-hour-avg.nc has global attributes different from or missing in comparison to other files. Say, compare for instance with IMOS_ACORN_V_20090327T100000Z_TURQ_FV00_1-hour-avg.nc or IMOS_ACORN_V_20090327T110000Z_TURQ_FV00_1-hour-avg.nc. can you provide more details?

sorry, I should have been more specific - this one was just for a double-check since the published file was OK and slightly newer than the errored one.

I don't see a problem with IMOS_ACORN_RV_20180105T144000Z_CWI_FV01_radial.nc either and have re-uploaded the file I have locally.

As above - just a double-check. I think this one was a fail on our side.

ocehugo commented 3 years ago

Update the list in here: https://github.com/aodn/content/issues/474