search

coderxio / sagerx

Open drug data pipelines curated by pharmacists.
https://coderx.io/sagerx
Other
49 stars 13 forks source link

Change transform to list #313

Closed jrlegrand closed 2 months ago

jrlegrand commented 4 months ago

Explanation

Changed the common transform task to handle multiple levels of transformation (staging alone or staging and intermediate). I still need to figure out the best way to handle marts because we use the + character in front of the model - I believe to indicate we want dbt to run all dependent models for these(?). Either way, I left the build_marts tasks alone for now - hardcoded into the task.

Rationale

Currently, you can only transform data into staging models because we didn't account for intermediate models. This allows flexibility to do either or both.

Tests

Tested running DAGs that only had staging models and also ones that had both staging and intermediate models.

jrlegrand commented 4 months ago

Should also update the fda_enforcement DAG - specifically this part:

    transform_staging_task = transform.override(task_id='transform-staging')(dag_id)
    transform_intermediate_task = transform.override(task_id='transform-intermediate')(dag_id,'intermediate')

    extract_task >> load_task >> transform_staging_task  >> transform_intermediate_task