search

DOI-USGS / lake-temperature-model-prep

Pipeline #1
Other
6 stars 13 forks source link

Fix wqp file name bug in 7b_temp_munge step #363

Closed padilla410 closed 1 year ago

padilla410 commented 1 year ago

Overview

Our metadata reviewer for the lstm data release found a bug in this pipeline when she ID'd many NA values in the output of 7b_temp_merge/out/temp_data_with_sources.feather.

There were two issues going on : (1) I needed to get back into the metadata file and add info for some cooperators; and (2) there was no way to partial match file names for wqp tagged data. This PR fixes that bug in add_source_ids (within 7b_temp_merge/src/munge_source_ids.

All of the additional files came along for the ride when I re-ran the pipeline.

Closes USGS-R/lake-temp-lstm-static-data-release#37

Verification

Running this script should result in zero rows in rows in na_temp_dat-with_src

library(arrow)
library(tidyverse)

# Checkout the offending file - data with missing `source_id` values
temp_dat_with_src <- read_feather("7b_temp_merge/out/temp_data_with_sources.feather")
na_temp_dat_with_src <- temp_dat_with_src |> filter(is.na(source_id))