Data processing decisions we need to make/update/confirm when we have SB19plus info/update .psa:

[coleary-noaa]

1. data processing: output variables we want - currently set to conductivity (S/m), density (kg/m^3), depth (salt water, meters, lat 0), salinity practical (PSU), temperatures ITS-90 deg C, pressure
2. filtering: filter type A or B
3. filtering: desired filter time constants
4. align CTD: advance values (e.g., If 0 seconds is entered, alignment relative to pressure (and time) remains unchanged for that variable.) - help file has suggested temperature and conductivity alignment suggestions for each model (e.g., 19plus is +0.5)
5. cell thermal mass: need alpha and beta entries (thermal anomaly amplitude & time constant): typical values in table above and equation in Formulas in help file (e.g., SBE 19plus or 19plus V2 with TC duct and 2000 rpm pump: alpha = 0.04, 1/beta = 8.0 )
6. window filters: specify which variables to run window filter on and the filters used (boxcar, cosine, gaussian, median, triangles) , & window size[scans]: default is set to 5
7. Derive TEOS-10 options: Select TEOS-10 variables (i selected absolute salinity, conservative temperature, and density for the test)
8. Derive TEOS-10 options: instrument type (Other, SBE 21 Seacat Thermosalinograph, SBE 45 MicroTSG Thermosalinograph) & sources for Lat/Lon data when NMEA data not present, source for pressure data when pressure column is not present
9. Derive options: need to select derived variables - selected depth (salt water, m), density (kg/m^3), potential temp (ITS-90 deg C), salinity (PSU) for now

[sean-rohan-NOAA]

Made a couple of modifications to the batch processing SBE19(non-plus) files for the .psa after some additional reading.

• In DataCnv: Only output strain gauge pressure, conductivity, and temperature.
• Changed the time windows for the low-pass filter.

[sean-rohan-NOAA]

2-3: The different filtering options are just for setting up two different filters. The default Filter.psa options are for the 911plus. Recommendation for 19plus V2 is pressure +1, temperature and conductivity +0.5 if there are no outliers. I set up the 19plus Filter.psa with the recommended settings.

4: I set up the AlignCTD.psa file with temperature +0.5 and no lag for conductivity. May need to revisit this decision (Sea-Bird's recommendation is +0.0-0.1).

[sean-rohan-NOAA]

Variable descent and ascent rates and relatively low resolution 4 Hz scan interval make it challenging to find a single alignment that works for all of the casts. Comparing multiple alignment values and selecting the best one seems to work reasonably well in the EBS/NBS:

1. Setup processing directory and move .hex, .psa, .xmlcon, and .bat files.
2. Convert hex files to cnv with the following channels as outputs: -- Temperature -- Conductivity -- Pressure
3. Window filter T, C, pressure (median, 5 scan window)
4. Low-pass filter (temperature = 0.5, salinity = 0.5, pressure = 1.0)
5. Align (try 13 combinations of alignments; see .xlsx attachment)
6. Cell thermal mass correction (defaults: ɑ = 0.04, τ[CTM] = 8)
7. Loop edit (minimum speed = 0.05 m/s)
8. Derive EOS-80
9. Calculate bottom temperature, bottom salinity, bottom sound speed
10. Split upcasts and downcasts using custom method
11. Bin by depth (1 m)
12. Select the ‘best’ alignment based on total path distance for salinity profiles
13. Flag density inversions and N2 instability
14. Manually review profiles and flag any remaining bad data. Replace bad data with interpolated values.
15. Review and select final profiles.
16. Calculate surface temperature, surface salinity, average sound speed.

processing_methods.xlsx

Examples ('best' alignment is bolded):

[sean-rohan-NOAA]

Method in #31 seems promising for replacing step 5.

[sean-rohan-NOAA]

Method in #31 seems promising for replacing step 5.

Process using 'typical' parameters:

1. Setup processing directory and move .hex, .psa, .xmlcon, and .bat files.
2. Convert hex files to cnv with the following channels as outputs: Temperature (T), Conductivity (C), Pressure
3. Window filter T, C, pressure (median, 5 scan window)
4. Low-pass filter (temperature = 0.5, salinity = 0.5, pressure = 1.0)
5. Align (temperature = +0.5)
6. Cell thermal mass correction (ɑ = 0.04, τ[CTM] = 8)
7. Loop edit (minimum speed = 0.05 m/s)
8. Derive EOS-80
9. Calculate bottom temperature, bottom salinity, bottom sound speed
10. Split upcasts and downcasts using custom method
11. Calculate 1-m bin averages.

Process using estimated align and celltm parameters:

1. Setup processing directory and move .hex, .psa, .xmlcon, and .bat files.
2. Convert hex files to cnv with outputs: temperature (T), conductivity (C), pressure
3. Window filter T, C, pressure (median, 5 scan window)
4. Low-pass filter (temperature = 0.5, salinity = 0.5, pressure = 1.0)
5. Loop edit (minimum speed = 0.05 m/s)
6. Derive EOS-80 and TEOS-10
7. Calculate bottom temperature, bottom salinity, bottom sound speed.
8. Split upcasts and downcasts using custom method.
9. Calculate 1-m bin averages.
10. Estimate temperature alignment using the adjustment that maximizes the squared Spearman rank correlation between temperature and conductivity channels, similar to Ullman and Hebert (2014).
11. Align temperature for downcasts and upcasts using output of step 8.
12. Derive EOS-80 and TEOS-10
13. Split upcasts and downcasts using custom method.
14. Calculate 1-m bin averages.
15. Estimate cell thermal mass correction parameters that minimize the area between 1-m binned temperature-salinity profiles, following Garau et al. (2011). The optimization uses BFGS. Starting parameters are set to CellTM recommended defaults (ɑ = 0.04, τ[CTM] = 8).
16. Cell thermal mass correction on downcasts and upcasts using output of step 14.
17. Derive EOS-80 and TEOS-10.
18. Split upcasts and downcasts using custom method.
19. Calculate 1-m bin averages.

Select and clean profiles generated using the 'typical' or estimated method:

1. Select the best profile for each cast based on path distance of salinity profiles. Profiles compared: binned profiles processed using 'typical' module values, binned output after step 11, binned output after step 16, binned output after step 21.
2. Flag density inversions and interpolate flagged values based on N2 inversion.
3. Manually review profiles and flag any remaining bad data. Replace bad data with interpolated values.
4. Review profiles.
5. Calculate surface temperature, surface salinity, average sound speed.

References: Garau, B., Ruiz, S., Zhang, W.G., Pascual, A., Heslop, E., Kerfoot, J., Tintoré, J., 2011. Thermal lag correction on slocum CTD glider data. J. Atmos. Ocean. Technol. 28, 1065–1071. https://doi.org/10.1175/JTECH-D-10-05030.1

Ullman, D.S., Hebert, D., 2014. Processing of underway CTD data. J. Atmos. Ocean. Technol. 31, 984–998. https://doi.org/10.1175/JTECH-D-13-00200.1

Example salinity profiles:

[sean-rohan-NOAA]

# Run using 'typical' parameters
gapctd::run_method(vessel = vessel,
                   year = year,
                   region = region,
                   channel = channel,
                   processing_method = "sbe19plus_v2",
                   storage_directory = here::here("output", "sbe19plus_v2"),
                   ctd_dir = ctd_dir)

# Process without alignment or cell thermal mass correction
gapctd::run_method(vessel = vessel,
                   year = year,
                   region = region,
                   channel = channel,
                   processing_method = try_method,
                   storage_directory = here::here("output", try_method),
                   ctd_dir = ctd_dir)

# Estimate alignment parameters
alignment_df <-
  gapctd::run_alignment_calcs(
    profile_files = sort(c(
      list.files(
        here::here("output", try_method),
        full.names = TRUE,
        pattern = "downcast.cnv"
      ),
      list.files(
        here::here("output", try_method),
        full.names = TRUE,
        pattern = "upcast.cnv"
      )
    )),
    make_diagnostic_plots = TRUE,
    min_pressure = 4,
    cor_method = "spearman",
  )

saveRDS(object = alignment_df, here::here("output", paste0(region, "_", year, "_", vessel, "_align_pars.rds")))

# Align profile data

gapctd::run_method(vessel = vessel,
                   year = year,
                   region = region,
                   channel = channel,
                   processing_method = try_method,
                   storage_directory = here::here("output", paste0(try_method, "_align")),
                   ctd_dir = ctd_dir,
                   alignment_df = readRDS(file = here::here("output", paste0(region, "_", year, "_", vessel, "_align_pars.rds"))))

# Estimate cell thermal mass correction parameters for aligned data

ctm_adjust_df <- gapctd::run_ctm_adjust_tsarea(profile_files = sort(c(list.files(here::here("output", "sbe19plus_v0_align"),
                                                                                 full.names = TRUE, pattern = "downcast.cnv"),
                                                                      list.files(here::here("output", "sbe19plus_v0_align"),
                                                                                 full.names = TRUE, pattern = "upcast.cnv"))),
                                               optim_method = "SANN",
                                               min_pressure = 4,
                                               start_alpha = 0.04,
                                               start_tau = 8,
                                               obj_method = "area")

saveRDS(ctm_adjust_df, file = here::here("output", paste0(region, "_", year, "_", vessel, "_ctm_adjust_pars.rds")))

# Apply cell thermal mass correction
gapctd::run_method(vessel = vessel,
                   year = year,
                   region = region,
                   channel = channel,
                   processing_method = try_method,
                   storage_directory = here::here("output", paste0(try_method, "_ctm_adjust")),
                   ctd_dir = ctd_dir,
                   ctm_df = readRDS(file = here::here("output", paste0(region, "_", year, "_", vessel, "_ctm_adjust_pars.rds"))))

# Compare results
compare_df <- gapctd::compare_methods(prefix = "/binavg/align_ctd_method",
                                      method_labels = c("Estimated (Raw)", "Estimated (Aligned)", "Estimated (CTM-Area)", "Typical method"),
                                      processing_method = c("sbe19plus_v0", "sbe19plus_v0_align", "sbe19plus_v0_ctm_adjust", "sbe19plus_v2"),
                                      return_output = TRUE,
                                      pattern_downcast = "downcast_binavg.cnv",
                                      pattern_upcast = "upcast_binavg.cnv",
                                      scale_vars = FALSE,
                                      area_method = "ts")

##### MOVE FINAL FILES TO CNV DIR #####
gapctd::move_to_final_cnv(compare_df$best_df)

# Run density inversion check and correction
gapctd::correct_density_inversion(threshold = -1e-5, 
                                  threshold_method = "bv", 
                                  correct_inversion = TRUE, 
                                  cnv_dir = here::here("output", "sbe19plus_v0_ctm"), # Update to final_cnv
                                  pattern = "binavg")

# Manually review and flag bad data. Remove and interpolate bad data.
gapctd::manual_flag_interpolate()

# Review final data
gapctd::manual_review()

# Package files into a .zip
gapctd::make_ctd_zip(region = region, vessel = vessel, year = year)

# Package data product
### Separate code