USGS-R / nawqa-pesticide-nsp

Workflow Updates for the NAWQA Pesticide National Synthesis Project
0 stars 3 forks source link

Refine fetch pattern #3

Closed padilla410 closed 2 years ago

padilla410 commented 2 years ago

This PR completes three tasks related to 1_fetch and one task related to 2_process:

The next step will be using the targets generated in p1 to create the individual pesticide rasters. The next portion of the pipeline will use dynamic branching and will map over the filet targets created in p1.

I did a bit of manual testing to verify that modifying values in a file (e.g., 1_fetch/in/maps/dbf19/H_24D.dbf) will result in a target rebuild.

Issues and questions

  1. In 1_fetch I am using tar_files to make sure that the 4 types of input files are tracked and can then be branched over to load the data. There is one issue with this: when running tar_visnetwork these targets always appear "outdated". This is a known issue. As suggested by the doc, I tried switching from tar_files to tar_files_input. That prevented the pipeline from working when it tried to produce the targets that depend on p1_pest_of_interest (p1_pest_label_csv and p1_pest_bin_csv). I was not able to determine why it failed, outside of the ambiguous error message:
Error in purrr::map_chr(., ~grep(paste("(\\.", .x, "\\.)", sep = ""),  : 
  object 'p1_pest_of_interest' not found
Error in `tar_throw_run()`:
! callr subprocess failed: object 'p1_pest_of_interest' not found
Visit https://books.ropensci.org/targets/debugging.html for debugging advice.
Run `rlang::last_error()` to see where the error occurred.

I did verify that the internal components of list_pest_csv() work as intended.

  1. The next PR will be creating the final maps, I'm not quite sure how to link up the file targets I created at the end of 2_process to the inputs of 3_viz
padilla410 commented 2 years ago

Thank you for the review! In addition to tackling your comments, I also made a few updates to 2_process and to the description of this PR. I intend to talk about some of the issues outlined above under issues and questions at our meeting later today.