SOOP TMV NRT issues - Githubissues

aodn / content

Tracks AODN Portal content and configuration issues

0 stars 0 forks source link

SOOP TMV NRT issues #434

Closed ggalibert closed 4 years ago

ggalibert commented 5 years ago

[x] metadata record should include chlorophyl parameter.
[x] link to THREDDS resource in metadata record is not correct. Should be http://thredds.aodn.org.au/thredds/catalog/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/catalog.html with l at the end.
[x] some netcdf 1 sec files still have CPHL and TURB in count while metadata says engineering units: see IMOS_SOOP-TMV_TSUB_20180805T220408Z_VLST_FV00_transect-M2S_END-20180806T212840Z.nc . Data should all be in engineering units.
[x] all netcdf 1 sec should include the calibration_formula, calibration_blank and calibration_scale variable attributes for TURB and CPHL with:
```
calibration_formula = "value_engineering_units = (counts - calibration_blank) x calibration_scale" ;
CPHL:calibration_blank = 40. ;
CPHL:calibration_scale = 0.0123 ;
TURB:calibration_blank = 50. ;
TURB:calibration_scale = 0.006 ;
```
See http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/transect/2014/01/IMOS_SOOP-TMV_KSTUZ_20140101T082410Z_VLST_FV02_transect-M2D_END-20140101T183810Z.nc.html as an example.

lbesnard commented 5 years ago

* some netcdf 1 sec files still have CPHL and TURB in count while metadata says engineering units: see IMOS_SOOP-TMV_TSUB_20180805T220408Z_VLST_FV00_transect-M2S_END-20180806T212840Z.nc . Data should all be in engineering units.

I don't understand why yet, but this is because those files were not reprocessed. Their date is "2019-03-07T00:10:03Z" I need to investigate, but most likely due to the fact that maybe the log files were lost (?) and not re-pushed.

all netcdf 1 sec should include the calibration_formula, calibration_blank and calibration_scale variable attributes for TURB and CPHL with:

This is the case for ALL new realtime files re-processed in mid-June. But linked to the reason written above

lbesnard commented 4 years ago

affected files:

 select url from soop_tmv.indexed_file where last_indexed < '2019-06-01' and url like 'IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/%' order by url;                                                                                
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                             url                                                                             │
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2017/IMOS_SOOP-TMV_TSUB_20170201T184741Z_VLST_FV00_mooring-DEV_END-20170201T184741Z.nc   │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2017/IMOS_SOOP-TMV_TSUB_20170210T182231Z_VLST_FV00_mooring-DEV_END-20170210T182231Z.nc   │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2018/IMOS_SOOP-TMV_TSUB_20180806T212841Z_VLST_FV00_mooring-SYD_END-20180812T223550Z.nc   │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190226T184251Z_VLST_FV00_mooring-DEV_END-20190227T085550Z.nc   │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190227T184803Z_VLST_FV00_mooring-MEL_END-20190228T090420Z.nc   │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190228T184441Z_VLST_FV00_mooring-DEV_END-20190301T085120Z.nc   │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190301T182711Z_VLST_FV00_mooring-MEL_END-20190301T222600Z.nc   │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190302T074641Z_VLST_FV00_mooring-DEV_END-20190302T105320Z.nc   │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190302T202041Z_VLST_FV00_mooring-MEL_END-20190303T000000Z.nc   │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190303T091831Z_VLST_FV00_mooring-DEV_END-20190303T115850Z.nc   │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190303T211703Z_VLST_FV00_mooring-MEL_END-20190304T083340Z.nc   │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190304T184511Z_VLST_FV00_mooring-DEV_END-20190305T085820Z.nc   │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170201T090721Z_VLST_FV00_transect-M2D_END-20170201T090721Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170202T083511Z_VLST_FV00_transect-D2M_END-20170202T083511Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170203T082021Z_VLST_FV00_transect-M2D_END-20170203T082021Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170203T214321Z_VLST_FV00_transect-D2M_END-20170203T214321Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170204T094231Z_VLST_FV00_transect-M2D_END-20170204T094231Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170205T214411Z_VLST_FV00_transect-D2M_END-20170205T214411Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170207T085231Z_VLST_FV00_transect-D2M_END-20170207T085231Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170209T083911Z_VLST_FV00_transect-D2M_END-20170209T083911Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170210T214511Z_VLST_FV00_transect-D2M_END-20170210T214511Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170211T214511Z_VLST_FV00_transect-D2M_END-20170211T214511Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170212T215241Z_VLST_FV00_transect-D2M_END-20170212T215241Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2018/IMOS_SOOP-TMV_TSUB_20180805T220351Z_VLST_FV00_transect-M2D_END-20180805T220351Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2018/IMOS_SOOP-TMV_TSUB_20180805T220408Z_VLST_FV00_transect-M2S_END-20180806T212840Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2018/IMOS_SOOP-TMV_TSUB_20180812T223551Z_VLST_FV00_transect-S2M_END-20180813T235020Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2018/IMOS_SOOP-TMV_TSUB_20181031T182731Z_VLST_FV00_transect-M2S_END-20181031T191900Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2018/IMOS_SOOP-TMV_TSUB_20181117T194731Z_VLST_FV00_transect-M2S_END-20181117T203200Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190209T195211Z_VLST_FV00_transect-M2S_END-20190209T202130Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190226T082911Z_VLST_FV00_transect-M2D_END-20190226T184250Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190227T085551Z_VLST_FV00_transect-D2M_END-20190227T184800Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190228T090421Z_VLST_FV00_transect-M2D_END-20190228T184440Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190301T085121Z_VLST_FV00_transect-D2M_END-20190301T182710Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190301T222602Z_VLST_FV00_transect-M2D_END-20190302T074640Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190302T105321Z_VLST_FV00_transect-D2M_END-20190302T202040Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190303T000003Z_VLST_FV00_transect-M2D_END-20190303T091830Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190303T115851Z_VLST_FV00_transect-D2M_END-20190303T211700Z.nc │
│ IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190304T083341Z_VLST_FV00_transect-M2D_END-20190304T184510Z.nc │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
(38 rows)

need to find the associated log files now

lbesnard commented 4 years ago

Some files with a time dimension equal to one must be deleted. For the rest, I need to find the log files

ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2017/IMOS_SOOP-TMV_TSUB_20170201T184741Z_VLST_FV00_mooring-DEV_END-20170201T184741Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2017/IMOS_SOOP-TMV_TSUB_20170210T182231Z_VLST_FV00_mooring-DEV_END-20170210T182231Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2018/IMOS_SOOP-TMV_TSUB_20180806T212841Z_VLST_FV00_mooring-SYD_END-20180812T223550Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190226T184251Z_VLST_FV00_mooring-DEV_END-20190227T085550Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190227T184803Z_VLST_FV00_mooring-MEL_END-20190228T090420Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190228T184441Z_VLST_FV00_mooring-DEV_END-20190301T085120Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190301T182711Z_VLST_FV00_mooring-MEL_END-20190301T222600Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190302T074641Z_VLST_FV00_mooring-DEV_END-20190302T105320Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190302T202041Z_VLST_FV00_mooring-MEL_END-20190303T000000Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190303T091831Z_VLST_FV00_mooring-DEV_END-20190303T115850Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190303T211703Z_VLST_FV00_mooring-MEL_END-20190304T083340Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/mooring/1sec/2019/IMOS_SOOP-TMV_TSUB_20190304T184511Z_VLST_FV00_mooring-DEV_END-20190305T085820Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170201T090721Z_VLST_FV00_transect-M2D_END-20170201T090721Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170202T083511Z_VLST_FV00_transect-D2M_END-20170202T083511Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170203T082021Z_VLST_FV00_transect-M2D_END-20170203T082021Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170203T214321Z_VLST_FV00_transect-D2M_END-20170203T214321Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170204T094231Z_VLST_FV00_transect-M2D_END-20170204T094231Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170205T214411Z_VLST_FV00_transect-D2M_END-20170205T214411Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170207T085231Z_VLST_FV00_transect-D2M_END-20170207T085231Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170209T083911Z_VLST_FV00_transect-D2M_END-20170209T083911Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170210T214511Z_VLST_FV00_transect-D2M_END-20170210T214511Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170211T214511Z_VLST_FV00_transect-D2M_END-20170211T214511Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2017/IMOS_SOOP-TMV_TSUB_20170212T215241Z_VLST_FV00_transect-D2M_END-20170212T215241Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2018/IMOS_SOOP-TMV_TSUB_20180805T220351Z_VLST_FV00_transect-M2D_END-20180805T220351Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2018/IMOS_SOOP-TMV_TSUB_20180805T220408Z_VLST_FV00_transect-M2S_END-20180806T212840Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2018/IMOS_SOOP-TMV_TSUB_20180812T223551Z_VLST_FV00_transect-S2M_END-20180813T235020Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2018/IMOS_SOOP-TMV_TSUB_20181031T182731Z_VLST_FV00_transect-M2S_END-20181031T191900Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2018/IMOS_SOOP-TMV_TSUB_20181117T194731Z_VLST_FV00_transect-M2S_END-20181117T203200Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190209T195211Z_VLST_FV00_transect-M2S_END-20190209T202130Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190226T082911Z_VLST_FV00_transect-M2D_END-20190226T184250Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190227T085551Z_VLST_FV00_transect-D2M_END-20190227T184800Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190228T090421Z_VLST_FV00_transect-M2D_END-20190228T184440Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190301T085121Z_VLST_FV00_transect-D2M_END-20190301T182710Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190301T222602Z_VLST_FV00_transect-M2D_END-20190302T074640Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190302T105321Z_VLST_FV00_transect-D2M_END-20190302T202040Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190303T000003Z_VLST_FV00_transect-M2D_END-20190303T091830Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190303T115851Z_VLST_FV00_transect-D2M_END-20190303T211700Z.nc | grep "TIME = "
ncdump -h http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190304T083341Z_VLST_FV00_transect-M2D_END-20190304T184510Z.nc | grep "TIME = "

ggalibert commented 4 years ago

all netcdf 1 sec should include the calibration_formula, calibration_blank and calibration_scale variable attributes for TURB and CPHL with:

This is the case for ALL new realtime files re-processed in mid-June. But linked to the reason written above

I haven't been able to confirm this. Please see http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/realtime/transect/1sec/2019/IMOS_SOOP-TMV_TSUB_20190620T093451Z_VLST_FV00_transect-M2D_END-20190620T194830Z.nc.html, the variable attributes mentionned above are missing for a file processed 2019-06-21T09:39:52Z

lbesnard commented 4 years ago

sorry my bad fix here: https://github.com/aodn/python-aodndata/pull/147

lbesnard commented 4 years ago

The smallest count of CPHL across the current dataset is 19 (−0.4428÷0.0123)+55 = 19, so unless we use a calibration_blank < 19, we will always have negative values. but at the same time, maybe this count value is bad data anyway.

I'll change the calibration_blank to 40. Will still be an improvement

atkinsn commented 4 years ago

Metadata record has been updated to include chl parameter, and THREDS link has been fixed.

lbesnard commented 4 years ago

all files were reprocessed over the weekend. + cleaning of files over land as an example http://thredds.aodn.org.au/thredds/dodsC/IMOS/SOOP/SOOP-TMV/VLST_Spirit-of-Tasmania-1/transect/2014/04/IMOS_SOOP-TMV_KSTUZ_20140401T083740Z_VLST_FV02_transect-M2D_END-20140401T184530Z.nc.html

I still have some files to delete (see comments above)