ploomber / projects

Sample projects using Ploomber.
Apache License 2.0
83 stars 25 forks source link

adding cookbook to pass variables from one task to the next one #43

Open edublancas opened 2 years ago

edublancas commented 2 years ago

A few times on Slack, we've been asked how to pass variables from one task to the next one. Typical use cases are runtime parameters (for example, doing feature selection and then training on that subset of features).

Our recommended solution is to serialize the parameters into a JSON file (or similar). However, this involves users learning how to use the JSON library. We should add a cookbook that shows how to do it, something like this:

tasks:
  # show an function task example
  - source: functions.some_function
     product:
       variable: products/variable.json

 # show a script/notebook example
   - source: scripts/some-script.py
     product:
       nb: products/report.html
       variable: products/another-variable.json

  # load  products/variable.json and products/another-variable.json her
   - source: scripts/last.py
      product: products/final.html

@idomic - tagging you to keep you in the loop. this isn't urgent, but if you need more good issues for the interns, this is a good one

idomic commented 2 years ago

Created a few tags for us across the repos: high/med/low priority