aodn / content

Tracks AODN Portal content and configuration issues
0 stars 0 forks source link

Argo - new simplified profiles break pipeline #435

Closed lbesnard closed 4 years ago

lbesnard commented 4 years ago

email sent to support@argo.net after corrupted pipeline.

New Simplified profiles type of files were received without mention of them in the documention. These files are also fairly different breaking the Talend harvester. (example dac/aoml/2900114/profiles/SD2900114_214.nc)

In the meantime, I will change the rsync script so we do not download those SD files

 I noticed some new type of files (Argo simplified profile). According to the global attributes of the file, the conventions are
"Conventions = "Argo-3.1 CF-1.6"

However, in the 3.2 version (https://archimer.ifremer.fr/doc/00187/29825/40575.pdf), which I assume would inherit of the 3.1 version, I can't find any mention of simplified type profiles. It doesn't seem to be any mention of files starting with "SD****.nc" such as this one from AOML
dac/aoml/2900114/profiles/SD2900114_214.nc

Also, these new SD files seem to be fairly different in terms of dimensions. For example, they are lacking the following dimension "N_HISTORY", which is not consistent with the rest of the dataset.

Would you be able to point me to the right documentation for this new type of profiles?
lbesnard commented 4 years ago

https://github.com/aodn/data-services/pull/944