Boundary point in ACORN causes download issues

[kereid]

We had an error email come through from a user, and after some investigation @gsatimos found that there is a boundary point where the data dimensions change, so we have closed the issue in the issues repo, and logged it here.

More info see: https://github.com/aodn/issues/issues/162

[ggalibert]

Yes, I can confirm @gsatimos 's diagnostic. The TURQ site antennas have been moved and on the 15-12-2012 when operations were back to normal the data was collected on a different grid (hence the data dimensions change).

I can see that the error is happening for FV00 files, it would be interesting to confirm this but I'm pretty sure there wouldn't be any issue with the FV01 files. What happened is that historical data before 15-12-2012 has been reprocessed on the new grid for consistency and at that time the re-processing has been done on the FV01 files (higher value added product than FV00) already produced. It looks like this hasn't been performed on FV00 files.

Not sure if we should ask ACORN to re-process the FV00 files since as soon as a FV01 file exists, it makes the FV00 equivalent file obsolete. This is a question for @smancini .

It would be good to contact the impacted users and tell them to download FV01 files when they exist instead of FV00.

[ggalibert]

@scosoli I don't know if you intend to re-process TURQ FV00 data files on a consistent grid like it's been done on FV01 files but please note this is causing problems for when users request an aggregated/subset product via the portal over the time period the antenna configuration changed.

[scosoli]

I think I can reprocess everything on a consistent grid if that helps - and I can reprocess all historical data since the very beginning converting them in FV01 if preferred. But I would like to do that from the raw range-gated data (raw level), generate QC radials, create new vectors on the final grid and run the full QC tests on vectors. the reason I have not done that yet is that I am not confident with what previous management did and not enough information is available. it is however on my to-do-list but not on my priority list. I can escalate it if deemed necessary.

[scosoli]

I have gridded 2009-2012 data on the final grid for internal consistency. I have run the QC tests described in the manual and I am uploading the updated FV01 netcdf data. will do the same for the period 2012-2015 and all data should be consistent and aggregation should no longer fail

[ggalibert]

@scosoli we need the FV00 files processed on a consistent grid re-uploaded (not the FV01). The RT collection is the one causing problem not the DM.

[scosoli]

so what do you need then?regridded data withouth running QC? if so, will do that this week

[ggalibert]

regridded data withouth running QC

Yes, to produce FV00 files, that would be great thank you!

[scosoli]

I have set up a couple of matlab script for the purpose and files are being generated

[ggalibert]

Thank you @scosoli , I believe the error with your files are that some global attributes are missing and yet required for our pipeline to ingest them successfully. @lbesnard will show you the difference and you will be able to add the missing global attributes.

[lbesnard]

@scosoli it seems like your files which are landing in the error directory don't have all the global attributes required, especially the time_coverage_start and end ones.

See for example the list of global attributes in a good file http://thredds.aodn.org.au/thredds/dodsC/IMOS/ACORN/gridded_1h-avg-current-map_non-QC/TURQ/2019/06/04/IMOS_ACORN_V_20190604T000000Z_TURQ_FV00_1-hour-avg.nc.info

However your new ones only have:

                :project = "Integrated Marine Observing System (IMOS)" ;
                :Conventions = "CF-1.6,IMOS-1.4" ;
                :institution = "Australian Coastal Ocean Radar Network (ACORN)" ;
                :title = "IMOS ACORN Turqoise Coast (TURQ), one hour averaged current RT-QC data, 2009-03-27T09:00:00Z" ;
                :instrument = "CODAR Ocean Sensors/SeaSonde" ;
                :site_code = "TURQ, Turqoise Coast" ;
                :ssr_Stations = "Seabird (SBRD), Cervantes (CRVT)" ;
                :date_created = "2019-06-04T04:34:55Z" ;

[scosoli]

This is weird as I am using exactly the same scripts I am using for the most recent RT files.

[scosoli]

found error on my side - all working properly now.I tested upload to the portal and the file is showing up already

[scosoli]

now, however, file still seems to have the wrong grid dimensions (55x57 instead of 59x60). global attributes have been updated with respect to previous version, but not variable dimensions. the file on my mac is on the opposite showing the correct grid size

[lbesnard]

now, however, file still seems to have the wrong grid dimensions (55x57 instead of 59x60). global attributes have been updated with respect to previous version, but not variable dimensions. the file on my mac is on the opposite showing the correct grid size

this is extremely bizarre since we don't modify the files. Can you point me to the file you're talking about ?

[scosoli]

that seem to have gone. now I have another problem with a few files that I need to fix. sorry about that, will do today

[scosoli]

for instance, compare IMOS_ACORN_V_20121027T010000Z_TURQ_FV00_1-hour-avg.nc and IMOS_ACORN_V_20121027T020000Z_TURQ_FV00_1-hour-avg.nc; something has gone wrong during the gridding and seem to affect only data after IMOS_ACORN_V_20121027T020000Z_TURQ_FV00_1-hour-avg.nc

[scosoli]

it looks like the grid size was randomly swapped from (I=55, J=57) to (I=57, J=55) without any apparent motivation and this has added some complication to the regridding process. examples of the resulting surface current vector maps are attached. I need to manually find the affected times and convert the files properly

[scosoli]

this is is found in the file IMOS_aggregation_20190528T071811Z.nc and also on the thredds portal. there is little to no indication of what may have happened but the SSS-UWA for the TURQ coast has a different set of combine grids. A list of time instants affected is given below, for a total of 440 files / times:

Date index (in the IMOS aggregated file)

2012-10-21 2200 25145 2012-10-22 1900 25166 2012-10-25 0900 25226 2012-10-26 0100 25241 2012-10-26 2100 25261 2012-10-26 2300 25263 2012-10-27 0200 25266 2012-10-27 0300 25267 2012-10-27 0400 25268 2012-10-27 0500 25269 -- and up to 2012-12-07 1500 25584

Dates after 2012-12-07 1500 do not seem to be affected by this problem.

It is difficult to figure out where the problem is originating from, and all attempts to remap the files on the proper grid failed. Combining the vectors again from radials is complicated by the following: 1, only IDEAL pattern files are available for this time period for CRVT station in the sss-uwa archives. 2, the total grid was changed on 20121030 3, there are licensing - incompatibility problems with the SSCombine software

vector maps created merging the IDEAL and MEASURED radials available on the acorn archives (using a dedicated matlab package), compared with files available on the portal, a, do match suggesting that JCU merged the two types of data; and b, correct the reshaping issue and can be used for the task

However, data available on backup drive are NOT complete and there are gaps in the resulting vector data. As such, I need to implement the least-squares fit directly from the netcdf radial files that are uploaded on the portal.

Tests with the following files evidenced however further problems: /test_dev/sbrd/IMOS_ACORN_RV_20121211T040000Z_SBRD_FV00_radial.nc /test_dev/crvt/IMOS_ACORN_RV_20121211T040000Z_CRVT_FV00_radial.nc -- this file is empty netcdf IMOS_ACORN_RV_20121211T040000Z_CRVT_FV00_radial {

Indeed, from the ncdump -h output: dimensions: SEASONDE_HEADER_SIZE = 7701 ; POSITION = UNLIMITED ; // (0 currently) DATE = 20 ; SEASONDE_RADS_TIME = 5 ; SEASONDE_RCVR_TIME = 15 ;

and from the comments in the global attributes: :comment = "CODAR was unable to detect radial surface currents. This file contains no surface current data. Only metadata are stored in this file." ;

So, it looks like there is a bunch of empty radial files on the thredds portal

[scosoli]

I have uploaded the regridded files to the portal- to the best of my knowledge, this issue can be closed

[ggalibert]

@kereid could you please check that an aggregation job performed over the whole time period of the TURQ RT collection is completed successfully? Then close if successful.

[kereid]

Will do :-)

[kereid]

Download succeeded so closing