Closed FridayPush closed 6 months ago
Hi @FridayPush you are correct that in this instance the union feature is what is causing the disconnected sources. Since the source.yml does not allow for a dynamic code approach (where one source.yml can represent multiple sources) we run into the situation where the sources end up not being connected.
We did find a resolution for this that we documented in our Fivetran Utils package for the underlying union_data macro. Would you be able to attempt the steps outlined here?
If this works for your lineage to connect the sources I would recommend my team updated the README of this package to point to this corresponding documentation so others are aware of how to connect the sources for the union feature. Let me know if this works for you!
This has been addressed in a previous release of the Shopify package. As such, closing out this issue.
Is there an existing issue for this?
Describe the issue
When using the DBT graph view for lineage the Sources from Shopify do not connect to the rest of the pipeline. For example we're putting a clean layer over the Shopify Source layer and in lineage with
-s @stg_shopify__order_line
which should pull the source upstream it doesn't.The sources for Shopify do exist, but they're not connected to any models. This creates a really 'blown out' main view since we have 50 shopify tables or so and then 50 sources that appear below them with no downstreams.
Relevant error log or model output
No response
Expected behavior
Source nodes should provide a lineage link into the next models.
dbt Project configurations
shopify_union_schemas: ['our_shopify_schema','schema_two'] shopify_using_fulfillment_event: True shopify_using_customer_metafields: True shopify_using_product_metafields: True . This is in a dev environment which uses an alternative generate_alias_name to have all development schemas in the same top level schema. Eg we flatten models to
<destination_schema>__<table_name>
so if we write our shopify data to schemashopify
the table for a dbt user looks likeshopify__stg_shopify__order_lines
Package versions
[">=0.8.0", "<0.9.0"]
What database are you using dbt with?
redshift
dbt Version
1.4.6
Additional Context
Shopify Source allows users to provide an Array of shopify source schemas to union together. We're using that feature and I'm guessing the lineage for the 'made up' table is causing problems.
Are you willing to open a PR to help address this issue?