search

aodn / python-aodntools

Repository for templates and code relating to generating standard NetCDF files for the Australia Ocean Data Network
GNU Lesser General Public License v3.0
10 stars 3 forks source link

velocity hourly timeseries product inconsistent with deployment file #157

Closed evacougnon closed 2 years ago

evacougnon commented 2 years ago

This issue was first raised by the Ocean Observer group.

After some investigation I reproduced the same inconsistency and therefore the velocity_hourly_timeseries.py needs to be checked. The issue may come from the way the ragged array is set.

Here is the script (using jupyter notebook) I used to reproduce the error: hourlytimeseries_vs_deployment.zip

Below is an email describing the issue:

Instead, I have revisited the problem and both I and another programmer (Tom) have separately extracted and compared the depth averaged velocities that are output from the LTSP with deployment file depth averaged velocities.

We get the same result. Tom codes in Python but didn’t use Eduardo’s code because it was a little too complex and difficult to interpret

Attached is Tom: LTSP vs NRSYON (NEMO) top plot zoomed in for deployment, bottom plot full LTPS time period Same plot but zoomed in for September 9-11 Me: LTSP vs NRSYON for all of September Same again but zoomed in for a few days Sep 8-11

The names of the files I used are in the legend.

Tom’s deployment file velocities are hourly averaged whereas mine are not time averaged.

It looks very much to me like the LTSP is incorrectly coded.

Could you or Marti please produce a similar comparison plot between the LTSP and a deployment file? Preferably for deployment NRSYON-2006 ? Kind regards, Madeleine figures.zip

mhidas commented 2 years ago

The issue was that the timestamps were being written to the hourly product in random order (the averaged velocity values were otherwise fine).

Fixed in #158!

image

mhidas commented 2 years ago

Thanks for all your work on this @evacougnon ! The notebook was helpful for doing a final check too.

mhidas commented 2 years ago

Just for the record, the root of the issue was that before Python 3.6 (I think...) dictionary-like objects were generally not ordered, and the original code used such a structure to set the timestamps for the hourly product. And we didn't have unittests for that code until now...