[Bug] Lineage for Sources in DBT graph is broken

[FridayPush]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the issue

When using the DBT graph view for lineage the Sources from Shopify do not connect to the rest of the pipeline. For example we're putting a clean layer over the Shopify Source layer and in lineage with -s @stg_shopify__order_line which should pull the source upstream it doesn't.

The sources for Shopify do exist, but they're not connected to any models. This creates a really 'blown out' main view since we have 50 shopify tables or so and then 50 sources that appear below them with no downstreams.

Relevant error log or model output

No response

Expected behavior

Source nodes should provide a lineage link into the next models.

dbt Project configurations

shopify_union_schemas: ['our_shopify_schema','schema_two'] shopify_using_fulfillment_event: True shopify_using_customer_metafields: True shopify_using_product_metafields: True . This is in a dev environment which uses an alternative generate_alias_name to have all development schemas in the same top level schema. Eg we flatten models to <destination_schema>__<table_name> so if we write our shopify data to schema shopify the table for a dbt user looks like shopify__stg_shopify__order_lines

Package versions

[">=0.8.0", "<0.9.0"]

What database are you using dbt with?

redshift

dbt Version

1.4.6

Additional Context

Shopify Source allows users to provide an Array of shopify source schemas to union together. We're using that feature and I'm guessing the lineage for the 'made up' table is causing problems.

Are you willing to open a PR to help address this issue?

• [ ] Yes.
• [X] Yes, but I will need assistance and will schedule time during our office hours for guidance
• [ ] No.

[fivetran-joemarkiewicz]

Hi @FridayPush you are correct that in this instance the union feature is what is causing the disconnected sources. Since the source.yml does not allow for a dynamic code approach (where one source.yml can represent multiple sources) we run into the situation where the sources end up not being connected.

We did find a resolution for this that we documented in our Fivetran Utils package for the underlying union_data macro. Would you be able to attempt the steps outlined here?

If this works for your lineage to connect the sources I would recommend my team updated the README of this package to point to this corresponding documentation so others are aware of how to connect the sources for the union feature. Let me know if this works for you!

[fivetran-joemarkiewicz]

This has been addressed in a previous release of the Shopify package. As such, closing out this issue.