Trouble uploading Desert Star sablefish tags

[kechave]

I get the following error when trying to upload Desert Star data. A side question- there are several csv files in this directory that is being pointed to. Is this okay? I removed all but one file and got a different error. So still didn't work...

Error in rbind(deparse.level, ...) : numbers of columns of arguments do not match

[tristan-sebens]

Could you please provide the files you were uploading when you experienced this issue?

Can you also please provide the code you executed?

[kechave]

The data is is what you assigned me for testing.

# Set this variable to the root of your tag data directory: Desert Star
tag_data_directory = here::here('C:/Users/katy.echave/Desktop/Desert Star/SeaTag MOD/Sablefish')

if(!dir.exists(tag_data_directory)) stop("Directory does not exist")

# Decoder to detect Tag Type
decoder = metl::decoders$Decoder_DesertStar_SeaTagMOD

meta =
  data.frame(
    tag_num = c("106175"), # Tag ID number
    tag_type = c("AR"), # Tag type, as specified in ABLTAG DB
    species_code = 20510, # Species code, as specified in ABLTAG DB
    seq_num = c(1) # Sequence number. Check DB doc for more details.
  )

head(meta)

decoder$decode_to_db(d = tag_data_directory, con = db_conn, meta = meta)

[tristan-sebens]

I believe that this issue was caused because the package was using the tag serial number field (Tag SN) as the value for TAG_NUM, when the expected value is actually in the ARGOS data under the PTT field.

Package has been updated so that the TAG_NUM field is not populated from the PTT field in the ARGOS packets. ARGOS packets are associated with SPDT_MODSN2 packets based on order in the datafile. SPDT_MODSN2 packets are paired with the SDPT_ARGOS packets that immediately precede them in the data file.

Feels like a rather brittle solution to me, but it's working for now.

Data still cannot be inserted into the DB, but this is due to the corresponding records being missing from the RECOVERY table. Data is being extracted from data files into data.frames as expected.

[kechave]

This tag is in the Recovery Table and still the same error message.

As an additional comment - the tag is in the database with the PTT as the Prim. Tag Num, and the serial number as the Sec. Tag Num. I've tried both ways and same error message.

On Thu, Aug 22, 2024 at 3:14 PM Tristan N G Sebens @.***> wrote:

I believe that this issue was caused because the package was using the tag serial number field (Tag SN) as the value for TAG_NUM, when the expected value is actually in the ARGOS data under the PTT field.

Package has been updated so that the TAG_NUM field is not populated from the PTT field in the ARGOS packets. ARGOS packets are associated with SPDT_MODSN2 packets based on order in the datafile. SPDT_MODSN2 packets are paired with the SDPT_ARGOS packets that immediately precede them in the data file.

Feels like a rather brittle solution to me, but it's working for now.

Data still cannot be inserted into the DB, but this is due to the corresponding records being missing from the RECOVERY table. Data is being extracted from data files into data.frames as expected.

— Reply to this email directly, view it on GitHub https://github.com/tristan-sebens/metl/issues/106#issuecomment-2305899615, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEI3G4IXEVIGPW45XNK7U7DZSZWDRAVCNFSM6AAAAABMI7Q7SOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBVHA4TSNRRGU . You are receiving this because you authored the thread.Message ID: @.***>

-- Katy B. Echave

Sablefish Tag Program NOAA/NMFS/Auke Bay Laboratories Ted Stevens Marine Research Institute 17109 Pt. Lena Loop Rd. Juneau, AK 99801

907.789.6006

Alaska Groundfish Tagging Map | NOAA Fisheries https://www.fisheries.noaa.gov/resource/map/alaska-groundfish-tagging-map

[tristan-sebens]

Alright, I think I understand the problem now. The issue is the mismatch of two separate operations.

1. In order to match tag metadata (really just tag type) to the sensor data extracted from the directory, we provide the meta parameter, which is then used by the Decoder as a lookup table. The sensor data contains a tag ID, which in this case appears to be the SN, and this is used to add the appropriate tag type value to that data.
2. In order to ensure that data is only uploaded if it has matching records in the RELEASE and RECOVERY tables, the DB enforces constraints that check to make sure that TAG_NUM, TAG_TYPE, REL/REC_SEC, and SPECIES_CODE all match at least one record in both RECOVERY and RELEASE.

So essentially we end up trying to do two things at once with the TAG_TYPE and TAG_NUM fields. When we pass the values in through the meta parameter, we can either give it:

meta = 
    data.frame(
        #...
        tag_num = "106175",
        tag_type = "AR",
       #....
    )

or we can pass in:

meta =
  data.frame(
    #...
    tag_num = c("314"), 
    tag_type = c("SN")
    #...
  )

In the first case, we correctly match the tag ID values contained in the raw data, but don't match with the TAG_NUM and TAG_TYPE fields in the RELEASE and RECOVERY tables.

In the second case, our values match with the RELEASE and RECOVERY tables, but not with the tag id values in the sensor data on disk, so our output data.frame can never be properly constructed.

Unfortunately, at this moment I can only see two solutions, neither of them easy:

1. The records for this tag are re-uploaded to the RELEASE and RECOVERY tables such that the TAG_NUM and TAG_TYPE fields (not the SEC_TAG_NUM/SEC_TAG_TYPE fields) match the values in the sensor data (106175)
2. We remove the constraints on the ELECTRONIC_TAG_METADATA table. This will allow us to add the records, but will mean that data can be uploaded even if the provided metadata does not have corresponding records in the RELEASE and RECOVERY tables.

[kechave]

This tag is entered in both Release and Recovery as follows: Tag Type/Tag Numb: SN 314 Sec Tag Type/Sec Tag Numb: AR 106175

On Wed, Sep 4, 2024 at 5:45 PM Tristan N G Sebens @.***> wrote:

I'm not able to locate tag # 106175 in the RECOVERY table, but it is present in RELEASE. Is it possible that the tag was written into the RECOVERY table with an incorrect value?

— Reply to this email directly, view it on GitHub https://github.com/tristan-sebens/metl/issues/106#issuecomment-2330438921, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEI3G4MMZZ6EQWA4SQ4OEK3ZU6ZUZAVCNFSM6AAAAABMI7Q7SOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZQGQZTQOJSGE . You are receiving this because you authored the thread.Message ID: @.***>

-- Katy B. Echave

Sablefish Tag Program NOAA/NMFS/Auke Bay Laboratories Ted Stevens Marine Research Institute 17109 Pt. Lena Loop Rd. Juneau, AK 99801

907.789.6006

Alaska Groundfish Tagging Map | NOAA Fisheries https://www.fisheries.noaa.gov/resource/map/alaska-groundfish-tagging-map

[tristan-sebens]

Yes, exactly. And with most of the tags we would be able to set the TAG_NUM/TAG_TYPE fields to whatever we liked (in this case, SN 314), and it would simply be bound to the outgoing data, but the DesertStar tags are different. The raw data is already associated with a tag id #, in this case 106175.

Additionally, because the DesertStar data can have the data from multiple tags mixed up in the same file, we have to use the tag id # that the data is already associated with in order to distinguish which data came from which tags. Essentially, we cannot change the tag numbers until after the data has already been extracted, and the code does not currently support a secondary transformation, because no other tag requires this.

Finally, this is an instance where it is by far the easier solution to simply swap the primary and secondary TAG_NUM/TAG_TYPE values for these records in the DB. To create an application and a data model which supports this kind of cross-reference would be very difficult, as it's essentially creating an inconsistent primary key, which is sometimes the primary TAG_NUM/TAG_TYPE, and sometimes the secondary TAG_NUM/TAG_TYPE.