Control `targets` pipeline in detail

NIEHS / beethoven

BEETHOVEN is: Building an Extensible, rEproducible, Test-driven, Harmonized, Open-source, Versioned, ENsemble model for air quality

https://niehs.github.io/beethoven/

Other

4 stars 0 forks source link

Control `targets` pipeline in detail #294

Closed sigmafelix closed 4 months ago

sigmafelix commented 7 months ago

[ ] ~~Check if targets supports a "tolerance" option for ignoring minute changes in all or selected nodes~~
[x] Make _targets.R succinct
[x] Split _targets.R at base learner fitting (following parts will restored later)

kyle-messier commented 7 months ago

@sigmafelix What kind of code or data tasks may be needed for a "tolerance"?

There is a lot to learn about targets including archetypes, branching, target_factories, that will hopefully provide us the nuance and detail we need. Lots to dig into though!

kyle-messier commented 7 months ago

@sigmafelix I do think our solution will be a combination of branching (I think static, but not positive yet) and function from archetypes, which does things like splitting, map(), etc. For example, we could treat each covariate calculation as a branch of the large calc_covariates.R -> buffer sizes could even be a branch (perhaps dynamic there)

sigmafelix commented 7 months ago

@Spatiotemporal-Exposures-and-Toxicology Yes, there are lots of avenues I need to explore at. The pipeline now is just for a demonstration purposes, and we need to prune a group of links that are already controlled by targets, archetypes, and others.

sigmafelix commented 6 months ago

Download is disabled; covariate calculation at a small subset of sites is run on one day. Currently seeing an error at the table join due to unit and name differences in the temporal fields across calculated covariates. This pipeline example relies on the most recent version of amadeus available in my dev branch.

sigmafelix commented 6 months ago

[x] ./tools is dedicated to educational materials
[ ] Move ./tools/pipeline to ./inst/targets
[x] Add punchcard.csv manual to README.md in main
[x] Change punchcard.csv name into the other
[ ] Add tests for pipeline helper functions, then put the functions in ./R
[x] _targets.R will be in ./inst/targets (if this name is kept); installation script may be necessary
[x] Complete deprecation of input/Rinput anytime soon
[x] Scale up the calculation on HPC and monitor stability, runtime, etc.

Checked items are currently available at tidying-dependency branch.

mitchellmanware commented 6 months ago

@sigmafelix Can you share the error? I have changes on amadeus branch which return $time for all time-oriented columns.

Is the pipeline example using the amadeus testdata or just functions?

sigmafelix commented 6 months ago

@mitchellmanware Thank you for checking in this issue. I found calc_hms in main returned a time column named "date". I have a custom function for the pipeline to fix this, but it will be great to fix it in amadeus. By the way, I have several changes (i.e., adding ellipsis arguments in all process_* and calc_* function to enable argument injection) in amadeus functions as well. Could we push all changes to main in a day or two?

mitchellmanware commented 6 months ago

Yes, next few days is a good timeline for me. I just mentioned in another issue dialogue (https://github.com/NIEHS/amadeus/issues/45#issuecomment-2029690409) that new gridMET and TerraClimate functions and calc_*modularization are complete, but I am waiting until after PR to start OpenLandMap data source.