etsy / boundary-layer

Builds Airflow DAGs from configuration files. Powers all DAGs on the Etsy Data Platform
Apache License 2.0
262 stars 58 forks source link

Upstream dependencies with generator #98

Open m-lce opened 3 years ago

m-lce commented 3 years ago

Hello,

I need to define the upstream dependencies from a key upstream from the items of the generators. Something like this

generators:
  - name: bq-jobs
    type: list_object_generator
    target: bq-job
    properties:
      items:
        - name: job1
          upstream: []
        - name: job2
          upstream:
            - job1
---
name: bq-job

operators:   
- name: << item['name'] >>
  type: kubernetes
  upstream_dependencies: << item['upstream']>>

When I try this I have this error

Found errors in sub dag: {'operators': {0: {'upstream_dependencies': ['Not a valid list.']}}}

Do you know how can I make it work ?

Thanks !