tristan-sebens / metl

An R package developed by and for the AFSC MESA group for extracting and standardizing data from telemetry tags.
GNU General Public License v3.0
0 stars 0 forks source link

depth data from Microwave Telemetry X Tags not uploading? #86

Closed kechave closed 4 months ago

kechave commented 4 months ago

It doesn't seem like the depth data is being uploaded from the Microwave Telemetry X Tags? At least my example tag: LL 128305

tristan-sebens commented 4 months ago

It appears that ~17% of the output records do not contain depth data.

I believe that this is a result of the tag's programming, as some tags cycle through periods of recording and not recording certain sensor data. I will ask Cindy T to weigh in on this.

The other potential explanation is that MT Xtags record their temperature and depth data in separate data sheets. This data is joined together into a single data.frame upon extraction. The join uses the timestamps associated with the data in each sheet to join them together. If two nearly simultaneous records have slightly disparate timestamps, then they would be joined as two separate records, each with one sensor value, and one NULL value.

tristan-sebens commented 4 months ago

After taking a closer look, I've identified the cause.

Like the temperature and depth data, the GPS location data is also stored in a separate sheet, and joined into a single dataframe with the sensor data. The GPS data is collected independently from the sensor data, so when it is joined into the sensor data there is typically no sensor data which was also collected at that exact instant in time, so the sensor data fields are filled in with NULL values.

Here's an example of what I mean, taken from tag 128305:

Image

CindyTribuzio-NOAA commented 4 months ago

As you note, this is likely one of the fun quirks of the way MTX tags compress data for transmission. Note that latitude and longitude in the 'Lat&Long' sheet are a daily calculations based on data in the 'Sunrise and Sunset Times' sheet. Thus, the timestamp for that doesn't have a time, only date. Theoretically, they are lot and long at the local noon. Thus, shouldn't be multiple values for Sept 1.

It looks like the LATITUDE and LONGITUDE values shown in this example are from the 'ARGOS Data' transmissions (also in the 128305a.txt file), which is when the tag is drifting at the surface and no longer attached.

Can you verify that the correct latitude and longitude values are being imported?

CindyTribuzio-NOAA commented 4 months ago

Also note that this tag transmitted 100% of it's compressed data. I would expect there to be mismatches in timestamps still, but there shouldn't be too many gaps in the timeseries.

New question (and since you said you deleted the testing data, I didn't want to muck around in the db while you were doing things): can you confirm that both the archived and real-time values are being uploaded? For example: Tag 128305 'Temp Data'.

And yes, we can soundly swear off of MTX tags forever into the future, they are a next level challenge to deal with