Open yamamoto-yuta opened 1 month ago
I was informed by @syou6162 that this is not a bug but rather an intended behavior.
dbt-osmosis determines the propagation source based on the distance of nodes, and in the case of the same distance, it cannot make an exact determination.
Based on that, I have changed the issue title from a bug to a feature request.
Overview
When using the
--add-progenitor-to-meta
option, the resultingosmosis_progenitor:
values can sometimes be incorrect.Reproduction Steps
While I am currently investigating the exact scenarios that cause this issue, one example is provided below.
Consider the following set of models in a lineage graph (the letters
A
~C
next tofct_item_shops
denote the order of JOINs).You can view the actual code in the following repository:
https://github.com/yamamoto-yuta/dbt-osmosis-inheritance-check
In this case, the propagation sources for each column in
fct_item_shops
should be as follows:item_key
...raw_items
item_code
...raw_shop_items
shop_id
...raw_shop_items
However, the actual result is as follows, where the source of
item_code
is incorrectly identified asraw_items
instead ofraw_shop_items
.item_key
...raw_items
item_code
...raw_items
← Should beraw_shop_items
shop_id
...raw_shop_items
dbt-osmosis Execution Result
```yaml version: 2 models: - name: stg_shops columns: - name: shop_id description: '' data_type: INT64 meta: osmosis_progenitor: source.my_dbt_project.tmp_dbt_osmosis_test.raw_shops - name: item_key description: '' data_type: STRING meta: osmosis_progenitor: source.my_dbt_project.tmp_dbt_osmosis_test.raw_shops - name: stg_item_shops columns: - name: item_code description: '' data_type: STRING meta: osmosis_progenitor: source.my_dbt_project.tmp_dbt_osmosis_test.raw_item_shops - name: shop_id description: '' data_type: INT64 meta: osmosis_progenitor: source.my_dbt_project.tmp_dbt_osmosis_test.raw_item_shops - name: stg_items columns: - name: item_key description: '' data_type: STRING meta: osmosis_progenitor: source.my_dbt_project.tmp_dbt_osmosis_test.raw_items - name: item_code description: '' data_type: STRING meta: osmosis_progenitor: source.my_dbt_project.tmp_dbt_osmosis_test.raw_items - name: fct_item_shops columns: - name: item_key description: '' meta: osmosis_progenitor: source.my_dbt_project.tmp_dbt_osmosis_test.raw_items data_type: STRING - name: item_code description: '' meta: osmosis_progenitor: source.my_dbt_project.tmp_dbt_osmosis_test.raw_items data_type: STRING - name: shop_id description: '' meta: osmosis_progenitor: source.my_dbt_project.tmp_dbt_osmosis_test.raw_shops data_type: INT64 ```Execution Environment