kstierhoff / estimATM

A tool for planning, execution, analysis, and reporting of CPS biomass from surveys using the acoustic-trawl method (ATM)
Other
2 stars 0 forks source link

Update calibration processing #2

Open jrenfree opened 1 year ago

jrenfree commented 1 year ago

Currently, to process calibration results in reportSurvey, either 'extractCalAll.R' is run, which accesses a SQL database to load all historical calibrations, or an existing 'cal_results_all.Rdata' files is loaded which contains the historical data. Therefore, calibration results of the current survey are dependent on having historical results for all surveys.

Although including historical results might be nice, getting calibration results for the current survey should not be contingent on previous surveys.

Additionally, Kevin created 'processCal_2107.R' which processes calibration data from the 2021 survey for the non-Lasker vessels (LBC, LM, SD, and JCF). This script is nice in that it can replace repeated code chunks in the reportSurvey and reportBiomass scripts, however it's still disconnected from the Lasker cal data.

Instead, I propose that there be a simple function which processes all calibration files in a given directory, then outputs those results to a user-define file. Then, that script can be called repeatedly, for each vessel, as necessary (e.g., from reportSurvey and/or reportBiomass). It will then be more flexible for future surveys.

Lastly, the historical calibration processing can still be an option, but should be handled independently.

jrenfree commented 1 year ago

Code in reportSurvey has been updated so that it automatedly cycles through each survey vessel and parses out the calibration info to be used in the tables and plots. This removed any hard-coding and repeated code chunks.

Still to be done is updating the extraction code to perhaps read .ecs files. This is because calibrations are mostly being processed in Echoview going forward, and thus pertinent xml or txt files are not available.

jrenfree commented 1 year ago

reportSurvey was updated so that the .xml files are still read to obtain the echosounder information (serial #s, pulse lengths, etc.) and single-target data. An .ecs file is then read to obtain the calibration parameters for each frequency. These calibration results then overwrite those that were obtained from the .xml files, and are what's reported in the report tables.

The main issue that still persists is that, at least for the Lasker, the polar plots of the single target detections are made using the "incorrect" calibration parameters from the .xml files, and thus are not entirely accurate. The solution may be to, when processing the calibration data in Echoview, export the calibrated single-target detections to a CSV file, which reportSurvey reads instead of the .xml file.

jrenfree commented 9 months ago

To do this properly, the plan should be to:

This will involve more manual work preparing the .ecs files to be configured for the survey report, but it is the more proper way to get the most accurate single-target detection results that are using the survey calibration parameters.