search

DOI-USGS / nawqa_wqp

Scripts/workflow for Water Quality Portal pulls for NAWQA trends and networks analyses.
Other
4 stars 3 forks source link

Split inventory tasktable #14

Closed lindsayplatt closed 6 years ago

lindsayplatt commented 6 years ago

I think I am really close. Currently stuck on an error about duplicate target names, but not sure how to overcome it. I wanted to include targets from a previous makefile in subsequent ones. The way I'm doing it now is apparently not correct (I want to reference wqp_pull_folders from 1_wqpdata.yml in tasks_1_wqp.yml, and wqp_dates from 1_wqpdata.yml in tasks_1_wqp_nitrate.yml.

This starts with 1_wqpdata.yml and creates different inventory task makefile called tasks_1_wqp_inventory.yml which has different inventory calls for each constituent.

Then 1_wqpdata.yml creates a task makefile with a different task for each constituent. Nested inside of this makefile are the instructions to use the partitions created in the inventory to create another makefile per constituent which will ultimately pull the data.

Here's the error I get with the current implementation:

In read_remake_file(obj$file) :
  tasks_1_wqp_inventory.yml contains duplicate targets 1_wqpdata, wq_dates, wqp_states, wqp_codes,
   wqp_state_codes, wqp_pull, 1_wqpdata/tmp/data, 1_wqpdata/out/inventory, 1_wqpdata/out/data, 
   1_wqpdata/log, wqp_pull_folders, inventory_plan, tasks_1_wqp_inventory.yml, 
   1_wqpdata/log/tasks_1_wqp_inventory.ind, wqp_pull_plan, tasks_1_wqp.yml, 
   1_wqpdata/log/tasks_1_wqp.ind, lib, lib/cfg, lib/cfg/gd_config.yml
aappling-usgs commented 6 years ago

Hi @lindsaycarr , I'm checking back in on this open PR - do you intend to add more commits and/or respond to some of the above comments?

@abock80 and I also noticed that the .ind files from your recent data pulls aren't yet git committed - it'd be nice to get those up to github when you can.

lindsayplatt commented 6 years ago

Yes, sorry - getting back into this now! It is still not running without errors, so I've got some work to do before it is merged.

And sorry about the missing .ind files. See this PR: https://github.com/USGS-R/nawqa_wqp/pull/16

aappling-usgs commented 6 years ago

No worries, I know you've been gone + managing other big priorities. Welcome back!

aappling-usgs commented 6 years ago

Hi @lindsaycarr , this one is back to you for debugging & merge conflicts. Lemme know when you're ready for re-review or merge.

aappling-usgs commented 6 years ago

Calling this plenty for one PR.