Plot metrics

[AndyMcAliley]

This PR adds the phase 5_visualize. It allows RMS error and bias to be plotted by lake, by depth, or by day of year.

The plots rely on the interpolated predictions csv. Each row of that csv has both predictions and observations, which is convenient for computing residuals and plotting.

How to run the code

• Install ltls conda environment from environment_local.yaml because additional plotting packages have been added. mamba is much faster:

  mamba env create -f environment_local.yaml

  but conda should work

  conda env create -f environment_local.yaml

• Create all validation plots.

  snakemake -c all -r -s 5_visualize.smk 5_visualize/out/model_prep/initial/cpu_a/plot_all_valid.dummy

NOTE: The pipeline commands above require a compiled dataset and a trained model each of which takes several hours to obtain, plus a set of observed temperatures and metadata. If you want, copy the following files from Tallgrass, preserving their directory structure relative to the repo root directory:

• 2_process/out/model_prep/train.npz
• 2_process/out/model_prep/valid.npz
• 2_process/tmp/model_prep/temperature_observations_interpolated.csv
• 2_process/tmp/model_prep/lake_metadata_augmented.csv
• 3_train/out/model_prep/initial/cpu_a_metadata.npz
• 3_train/out/model_prep/initial/cpu_a_weights.npz

The prediction/interpolation takes 20-30 minutes. To skip that as well, you should only need the following files:

• 2_process/tmp/model_prep/lake_metadata_augmented.csv
• 4_evaluate/out/model_prep/initial/cpu_a/interpolated_predictions_valid.csv
• 4_evaluate/out/model_prep/initial/cpu_a/interpolated_predictions_train.csv

How to review this PR

• Are there any obvious errors?
• Suggestions for additional plots for metrics are welcome

Issues that will be addressed in upcoming PRs (so don't worry about them yet)

• Change to a different plotting library - probably seaborn

[jdiaz4302]

No obvious errors stood out to me. The use and flow of operations looked good and the final plots do not look suspicious.

The bias/rmse for interpolated depth has that obvious spike at around 90 meters that kind of begs context. I assume that may be a single lake that is only responsible for observations that depth? It may be worth seeing something like a corresponding histogram or decile ticks (i.e., 10th, 20th, 30th, etc... percentiles) along the x-axis to get a better idea of data density/sparsity.

Some other potentially interesting dimensions to view error/bias along might include elevation and lake area.