2023 To-do list

[Klankers]

To-do

This is more or less a clone of #20 to track developmental changes. Much of this will not get done in 2023. Ideally something that can roll over following annual data analyst review.

Codebase cleanup and review

• [ ] Alternatives to LGTM coding for GitHub
• [ ] Fitting review post #26 - do we want a legacy module?
• [ ] Check all code for TODO markers
• [ ] Consolidate settings/cruise configurations (#25 ?)
• [ ] Working folders assessment (issue #48 )

Review/refactor/write tests

• [ ] convert.py (refactor_convert branch)
• [ ] oxy_fitting.py
• [ ] process_bottle.py
• [ ] process_ctd.py
• [ ] rinko.py
• [ ] sbe_reader.py

---

CLI tools

• [ ] ctdcal import (import data by file extension)
• [ ] ctdcal convert (convert/export engineering data in different formats)
• [ ] ctdcal qc (launch interactive QC)

---

Documentation

• [ ] NumPy docstring for all functions
• [ ] Add/expand example notebooks to ReadTheDocs
• [ ] Develop CTDCAL manual for science party sharing

---

Improve robustness

• [ ] add handling for missing/unused sensors (e.g. Rinko, fluorometer, transmissometer)
• [ ] make codebase less specific to ODF
• [ ] reduce scope of functions
• [ ] allow users to change settings with mutable types (e.g. dict?)
• [ ] Scan discrete files for errors for safe closures
• [ ] Mypy type checker

Testing

• [ ] data-driven testing?
• [ ] static type testing

Package integration

• [ ] test Python 3.10 (last tried 2/3/21, failing due to numpy it seems)
• [ ] mypy (starting with #39)

---

Decrease processing time

• [ ] Speed up oxygen and rinko modules (density matching)
• [ ] Writing sbeReader for xarray
• [ ] netCDF rather than pkl intermediate steps (is the data size worth it)?
• [ ] Srray manipulations in numpy, rather than pandas
• [ ] save fit parameters to file (json?), make option to re-use coefs instead of re-calculating (e.g. slow, iterative oxy fits)
• [ ] parallelization

---

Features

Processing

• [ ] Process as group (ODF, PMEL, CSIRO, etc.)
• [ ] Pass in manual flags for bottle (mistrips), refT, autosal, winkler O2 to avoid using in fit routines

  Fitting

• [ ] Fit as a function of time to remove drift (e.g., conductivity cell drift)
• [ ] Identify/write out fitting statistics for human assessment
• [ ] Identify/write out flag statistics

  Plotting

• [ ] pyDIVA section plots (objective mapping) or interpolated surfaces
• [ ] pcolor diff plots (vs. previous years?)
• [ ] TS plots
• [ ] Bottle/CTD flagging plots (confusion matrix before/after)

  Cruise report and post-cruise

• [ ] Function for sanitizing bad unicode in .rst files (carriage return, bad greek letters, etc.)
• [ ] Function to prep data for server storage (using STS folder architecture)
• [ ] Run comparisons

---

Quality control

• [ ] Confirm data_qc.py and flagging functions are operating correctly
• [ ] Document data_qc.py review process
• [ ] Add residuals plot to Bokeh
• [ ] Post-qc rerun step
• [ ] Absolute precision? or what other metric for good/bad flagging (#49 )?
• [ ] Useful fitting statistics, written out, possibly iteratively

---

Miscellaneous

• [ ] standardize flag names with directory names (oxy vs. oxygen, cond vs. conductivity, etc.) (standardize on WOCE parameter names/relationships where possible, but in a long form)
• [ ] re-reckon all module names to something better
• [ ] separate all conversions (salts, oxygen) from merging/calibration routines
• [ ] conda packaging
• [ ] multi-location logging (e.g., all messages to log file, only messages >= WARNING to stdout)
• [ ] quickstart command to prompt for cruise name, etc. (a la sphinx-quickstart)
• [ ] Inbox/outbox?
• [ ] machine learning (Controlled Abstention Networks (CAN)?)
• [ ] binary search algorithm for fitting T/C

  Binary search algorithm when fitting T/C - start looking for 2nd degree fit at 1000 db, then try upwards with min=500 to start. minimum 50 db bins. Optimize based on residuals on all points and on residuals against y=0 (differences are 0) for pressure range (1000:6000) possibly fold in some of the data above the pressure range, just to make sure it's not overly throwing out the top)