JFE Infinity logger parser do not support metadata for bursting instruments

[ggalibert]

The instrumentburst* attributes are missing from NetCDF files generated by the infinitySDLoggerParse.m (used in WA at NRSROT - see example file).

[ggalibert]

@mpsnell could you please send me an example of raw file to be processed by the toolbox?

[ggalibert]

I can average the bursts if that is better. The sample interval is 0.1s. -- @mpsnell

infinity-burst.zip

[ocehugo]

The problem here is that the CSV file got repeated entries at the same timestamp.

...
Date,Temp.[deg C],Chl-Flu.[ppb],Chl-a[ug/l],Turb. -M[FTU],Batt.[V],
2018/08/16 08:00:00,19.873,0.56,0.56,0.12,3.01,
2018/08/16 08:00:00,19.873,0.56,0.56,0.12,3.01,
2018/08/16 08:00:00,19.873,0.57,0.57,0.13,3.01,
2018/08/16 08:00:00,19.873,0.55,0.55,0.12,3.01,
2018/08/16 08:00:00,19.873,0.55,0.55,0.16,3.01,
2018/08/16 08:00:00,19.873,0.55,0.55,0.15,3.01,
2018/08/16 08:00:00,19.873,0.54,0.54,0.13,3.01,
2018/08/16 08:00:00,19.873,0.56,0.56,0.13,3.01,
2018/08/16 08:00:00,19.873,0.54,0.54,0.13,3.01,
2018/08/16 08:00:00,19.873,0.57,0.57,0.13,3.01,
2018/08/16 08:00:01,19.873,0.60,0.60,0.12,3.01,
2018/08/16 08:00:01,19.873,0.58,0.58,0.12,3.01,
2018/08/16 08:00:01,19.873,0.58,0.58,0.11,3.01,
2018/08/16 08:00:01,19.873,0.58,0.58,0.11,3.01,
2018/08/16 08:00:01,19.873,0.58,0.58,0.11,3.01,
2018/08/16 08:00:01,19.873,0.59,0.59,0.11,3.01,
2018/08/16 08:00:01,19.873,0.58,0.58,0.11,3.01,
2018/08/16 08:00:01,19.873,0.57,0.57,0.11,3.01,

this maps the median to zero, since the number of repeated entries is larger than the actual different measurements (line 62 on the reader):

sample_data.meta.instrument_sample_interval = median(diff(data.TIME.values*24*3600));

Which in turn, led to sampling_interval == 0.

The fix is simple - just mask the repeated entries and apply the median afterwards. However, I'm wondering what is the policy here:

1. Simply raise error or
2. Try to overcome the repeated entries and continue (e.g. do average->calculate diffs-> (diffs==0 -> continue) | (diffs>0 -> raise error))

Also, there should be an assertion for sampling_interval == 0. The GUI cannot fix this automatically and keep going after reading the file. It raises a tip only at finalization phase:

Info : 413964 TIME first duplicate values (and their corresponding measurements) had to be discarded in /home/holiveira/tmp/infinity/JFE.csv. Try to re-play and fix this dataset when possible using the manufacturer's software before processing it with the toolbox.

I also could not continue on the GUI because I need to:

please document site_nominal_depth or site_depth_at_deployment and either instrument_nominal_height or instrument_nominal_depth global attributes so that an actual depth can be computed from 1 or 2 other pressure sensors in the mooring

However, there is no option to set the depth in the GUI. I don't know where. Any quick way of specifing the nominal_depth on the UI I can quickly test/reproduce?

[ggalibert]

Any quick way of specifing the nominal_depth on the UI I can quickly test/reproduce?

The best way to overcome this is to run a copy of the deployment database that has a reference to this dataset.

[ocehugo]

@mpsnell,

Are you aware of the case below - same time, two different CHLA/TURBF readings?

2018/08/16 08:00:00,19.873,0.56,0.56,0.12,3.01,
2018/08/16 08:00:00,19.873,0.57,0.57,0.13,3.01,

This is standard? Should I (a) keep as is, (b) average/median the duplicated set, (c) keep first or (d) keep last in the set ?

[mpsnell]

G’day Hugo

This instrument has been set to 0.1s sampling in the burst. I can average each burst if that is preferred.

Cheers Mark

From: Hugo Oliveira notifications@github.com Sent: Tuesday, 30 July 2019 3:49 PM To: aodn/imos-toolbox imos-toolbox@noreply.github.com Cc: Snell, Mark (O&A, IOMRC Crawley) Mark.Snell@csiro.au; Mention mention@noreply.github.com Subject: Re: [aodn/imos-toolbox] JFE Infinity logger parser do not support metadata for bursting instruments, depends on #532 (#536)

@mpsnellhttps://github.com/mpsnell,

Are you aware of the case below - same time, two different CHLA/TURBF readings?

2018/08/16 08:00:00,19.873,0.56,0.56,0.12,3.01,

2018/08/16 08:00:00,19.873,0.57,0.57,0.13,3.01,

This is standard? Should I (a) keep as is, (b) average/median the duplicated set, (c) keep first or (d) keep last in the set ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/aodn/imos-toolbox/issues/536?email_source=notifications&email_token=AEAIZFVPXMQHCXVIVSR7FVTQB7W7DA5CNFSM4HB5TEKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3DDMXI#issuecomment-516306525, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEAIZFQWV4HK7EMJSMQBL2LQB7W7DANCNFSM4HB5TEKA.

[ggalibert]

I can average each burst if that is preferred

@mpsnell, this is more up to your team to tell us what you would like us to make available via the portal: bursting data or burst average?

If you think that the average of each burst is what 99% of the users will be looking for for their research then please provide the burst averaged data. Otherwise if the bursting data has significant scientific value to be available to AODN portal users then we should find a way to handle this dataset.

I have the feeling that in this case the burst averaged data is what should be available through the portal but please confirm.

[ocehugo]

@mpsnell,

I already included support to make the median of these measurements, but the times should be appropriately defined anyway. The file should be self-describing. In this case, the time resolution is higher than what is reported. It is better to fix this in your software (higher time precsion reporting). If this is an instrument software limitation, let me know so I can document it accordingly.

[ocehugo]

@mpsnell, after discussing this with @ggalibert , I cannot proceed with averaging/median/etc this since this will have to go to FV00 - relative "raw" data.

Hence, the "raw" data need to be corrected - ie, the time variable needs more precision (if this is the time resolution you want to distribute).

sorry about the confusion

[ggalibert]

Mark, to clarify Hugo's comment above, all you need to do at this stage is to tell us what you would like us to make available via the portal: bursting data at 0.1s sampling interval or a burst average at a greater sampling interval?

[mpsnell]

G’day Hugo

The instrument has been set to 0.1s sampling in bursts but logs in seconds. I can average each burst before applying the toolbox. I am waiting for a response from our local CPHL guru regarding sampling rate and burst averaging. I will let you know.

Cheers Mark

From: Hugo Oliveira notifications@github.com Sent: Thursday, 1 August 2019 12:50 PM To: aodn/imos-toolbox imos-toolbox@noreply.github.com Cc: Snell, Mark (O&A, IOMRC Crawley) Mark.Snell@csiro.au; Mention mention@noreply.github.com Subject: Re: [aodn/imos-toolbox] JFE Infinity logger parser do not support metadata for bursting instruments, depends on #532 (#536)

@mpsnellhttps://github.com/mpsnell,

I already included support to make the median of these measurements, but the times should be appropriately defined anyway. The file should be self-describing. In this case, the time resolution is higher than what is reported. It is better to fix this in your software (higher time precsion reporting). If this is an instrument software limitation, let me know so I can document it accordingly.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/aodn/imos-toolbox/issues/536?email_source=notifications&email_token=AEAIZFS6TDDZDTTAGHQLPSLQCJTQRA5CNFSM4HB5TEKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3JJQFQ#issuecomment-517117974, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEAIZFSZV35HPCBF4R6RWLLQCJTQRANCNFSM4HB5TEKA.

[mpsnell]

G’day Guillaume

The instrument has been set to 0.1s sampling in bursts but logs in seconds. I can average each burst before applying the toolbox. I am waiting for a response from our local CPHL guru regarding sampling rate and burst averaging. I will let you know.

Cheers Mark

From: Guillaume Galibert notifications@github.com Sent: Thursday, 1 August 2019 3:19 PM To: aodn/imos-toolbox imos-toolbox@noreply.github.com Cc: Snell, Mark (O&A, IOMRC Crawley) Mark.Snell@csiro.au; Mention mention@noreply.github.com Subject: Re: [aodn/imos-toolbox] JFE Infinity logger parser do not support metadata for bursting instruments, depends on #532 (#536)

Mark, to clarify Hugo's comment above, all you need to do at this stage is to tell us what you would like us to make available via the portal: bursting data at 0.1s sampling rate or a burst average at a lower sampling rate?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/aodn/imos-toolbox/issues/536?email_source=notifications&email_token=AEAIZFXT3TP2WAAMUHKLDDTQCKE6DA5CNFSM4HB5TEKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3JSP6A#issuecomment-517154808, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEAIZFXS7HO3HHHQOBSBZX3QCKE6DANCNFSM4HB5TEKA.

[mpsnell]

Okay, I will chase up our local guy for some direction on this.

From: Hugo Oliveira notifications@github.com Sent: Thursday, 1 August 2019 2:41 PM To: aodn/imos-toolbox imos-toolbox@noreply.github.com Cc: Snell, Mark (O&A, IOMRC Crawley) Mark.Snell@csiro.au; Mention mention@noreply.github.com Subject: Re: [aodn/imos-toolbox] JFE Infinity logger parser do not support metadata for bursting instruments, depends on #532 (#536)

@mpsnellhttps://github.com/mpsnell, after discussing this with @ggaliberthttps://github.com/ggalibert , I cannot proceed with averaging/median/etc this since this will have to go to FV00 - relative "raw" data.

Hence, the "raw" data need to be corrected - ie, the time variable needs more precision (if this is the time resolution you want to distribute).

sorry about the confusion

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/aodn/imos-toolbox/issues/536?email_source=notifications&email_token=AEAIZFUBHEMRCJIBUE62F5DQCKAOPA5CNFSM4HB5TEKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3JPQEQ#issuecomment-517142546, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEAIZFUDYEANHY5ZPLWYHKLQCKAOPANCNFSM4HB5TEKA.

[ocehugo]

Requirements:

• [x] JFE data timestamps are corrected to millisecond measurements when reading.
• [x] Timestamps are corrected based on the number of repeated timestamp entries.