DOI-USGS / lake-temperature-model-prep

Pipeline #1
Other
6 stars 13 forks source link

Add more from 320 #338

Closed padilla410 closed 2 years ago

padilla410 commented 2 years ago

Overview

This PR includes parsers for the following datasets from #320:

MASTER_mnlakedata_ historicalfiles_manualentry_ template.xlsx was going to be in this PR but the DOW numbers are highly problematic and would need manual verification.

I also cleaned up lake-temperature-model-prep > 6_temp_coop_fetch > data_queue by moving the datasets we are going to skip into ~ > data_archive > not_pursuing

This PR closes #320. The other three files listed as parser candidates were tackled in #337.

Results from scmake("8_viz")

A numbers snapshot from 8_viz/out/lakes_summary_fig.html (same as #333 and #337): image

Verification that the expected data is in the final data

> tmp <- arrow::read_feather(
+   '7a_temp_coop_munge/out/all_coop_dat_linked.feather'
+   )
> 
> new_data <- paste0('7a_temp_coop_munge/tmp/',
+                    c('Long_29016100.rds', 'temp_DO_PCA.rds',
+                      'NorfolkReservoir_hwy62_AR_monthlyTempDO_2016_2020.rds',
+                      'NorfolkReservoir_dam_AR_monthlyTempDO_2016_2020.rds'))
> 
> unique(tmp$source) %>% 
+   as.data.frame() %>% 
+   filter(. %in% new_data)
                                                                             .
1                                     7a_temp_coop_munge/tmp/Long_29016100.rds
2                                       7a_temp_coop_munge/tmp/temp_DO_PCA.rds
3   7a_temp_coop_munge/tmp/NorfolkReservoir_dam_AR_monthlyTempDO_2016_2020.rds
4 7a_temp_coop_munge/tmp/NorfolkReservoir_hwy62_AR_monthlyTempDO_2016_2020.rds
lindsayplatt commented 2 years ago

This PR closes https://github.com/USGS-R/lake-temperature-model-prep/issues/320.

WOW! Going to review this morning. So exciting 🎉

padilla410 commented 2 years ago

Every time I try to build locally, something happens. I tried to pull down and do a local build of your branch, but mine shows a different hash for 7a_temp_coop_munge/src/data_parsers/parse_mndnr_files.R and starts to rebuild more than it should. Let's just trust what you have!

I think you see the different hash because Long 29016100 and temp_DO_PCA have their parsers in the parse_mndnr_files.R file (they're from MNDNR so it seemed reasonable to drop them in there. I assume you're seeing a lot of files rebuild because the pipeline is rebuilding all the MNDNR files (everything that depends on parse_mndnr_files(). So, in retrospect, maybe I should have dropped these two additional MNDNR parsers into their own file.