search

StoXProject / RstoxFDA

Fisheries Dependent Analysis with Rstox
https://stoxproject.github.io/RstoxFDA
GNU Lesser General Public License v3.0
0 stars 1 forks source link

Missing individuals when converting from Biotic to StoxBiotic #81

Closed BergenCalling closed 1 year ago

BergenCalling commented 1 year ago

Going from 12367 aged individuals to 5067 individuals when converting from Biotic to StoxBiotic. Error message: There are more than one 'serialnumber' (HaulKey in StoxBioticData) for 113 out of 243 'station'(StationKey in StoxBioticData) in the NMDBiotic data. In DefineBioticAssignment() it is currently only possible to asssing all hauls of a station in the map (manual assignment). If certain Hauls should be exclcuded, use FilterStoxBiotic(). Duplicated serialnumber for the following cruise/station (of the fishstation table of the BioticData): NA/1 NA/10 NA/102 NA/105 NA/108 NA/11 NA/110 NA/111 etc

edvinf commented 1 year ago

Thank you for reporting.

The warning message about duplicate station ids means that some stations (fishstations) have the same station number. In StoxBiotic these will be represented as one station with several hauls. That comes with some caveats, but is not a hard error, and it should not result in any loss of individuals. The warning message is unclear, which we will follow up in https://github.com/StoXProject/RstoxData/issues/267 . You may add any comment to that message there.

In order to pin down what may be causing the loss of individuals, we will need a reproducible example. Preferably as small as possible, but an entire StoX-project will do. If it contains confidential data, or data sizes are very large. Please communicate it via internal IMR channels (shared disks or e-mail).

edvinf commented 1 year ago

Thanks for the example files (https://github.com/StoXProject/RstoxData/issues/267#issuecomment-1346491471). It seems that the missing ages are from stations that are not included in the converted data, but some of them are also from stations probably filtered in earlier steps (like missiontype 4). It is possible that this relates to a recently resolved bug in RstoxData (https://github.com/StoXProject/RstoxData/issues/241). #241 was closed after testing on a development version, so I will have to check if this applies to the version of RstoxData you are using.

If you can provide your version of RstoxData, I will run a test and get back to you if the development version provides a possible fix.

BergenCalling commented 1 year ago

RstoxData 1.7.5

From: Edvin Fuglebakk @.> Sent: mandag 12. desember 2022 15:23 To: StoXProject/RstoxFDA @.> Cc: Seim, Silje Elisabeth @.>; Author @.> Subject: Re: [StoXProject/RstoxFDA] Missing individuals when converting from Biotic to StoxBiotic (Issue #81)

Thanks for the example files (StoXProject/RstoxData#267 (comment)https://github.com/StoXProject/RstoxData/issues/267#issuecomment-1346491471). It seems that the missing ages are from stations that are not included in the converted data, but some of them are also from stations probably filtered in earlier steps (like missiontype 4). It is possible that this relates to a recently resolved bug in RstoxData (StoXProject/RstoxData#241https://github.com/StoXProject/RstoxData/issues/241). #241 was closed after testing on a development version, so I will have to check if this applies to the version of RstoxData you are using.

If you can provide your version of RstoxData, I will run a test and get back to you if the development version provides a possible fix.

— Reply to this email directly, view it on GitHubhttps://github.com/StoXProject/RstoxFDA/issues/81#issuecomment-1346596113, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AP3CV77I5EXKQ5IFAIOFUNDWM4YNBANCNFSM6AAAAAAS3Z7RPQ. You are receiving this because you authored the thread.Message ID: @.**@.>>

edvinf commented 1 year ago

Thanks again for reporting. If it is indeed the issue in #241 you are experiencing, it turns out that there is no available fix yet, and #241 is re-opened. You can likely proceed without a fix for StoX-Reca workflows, which I will try to explain below.

What is happening is that StoX fails to remove some age-readings when corresponding individuals are filtered out in a FilterBiotic-process. This seems to happen for data files which also contain prey-samples (stomach-samples). In the conversion to StoxBiotic, these age-readings does not match any individuals and are dropped.

If this is all that is going on, the conversion to StoxBiotic actually gives you the result you would have expected earlier in the project and you can continue using its output without a fix, as these age readings lost belong to individuals that have been filtered out of the data. Inspecting the files you submitted earlier, it does indeed look like all age-readings that are lost in the conversion have missiontype 4, and are probably not desired for Reca runs.