[Bug] Issue With Fee Unique Columns Test

[jmussitsch]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the issue

The test dbt_utils_unique_combination_of_columns_stg_stripe__fee_balance_transaction_id__source_relation is defined as:

- name: stg_stripe__fee
    description: The details of a fee associated with a balance transaction
    tests:
      - dbt_utils.unique_combination_of_columns:
          combination_of_columns:
            - balance_transaction_id
            - source_relation

However, the Fee table has an additional column index that is part of the composite primary key as described here:

https://docs.google.com/presentation/d/1zyxgbaOjgBt3NsY0OfsiGsWDIefcBc-R1lHWlMltCYU/edit#slide=id.g14afa1361c6_3_0

So this test is giving false positives

Relevant error log or model output

No response

Expected behavior

The expected behavior would be for this test to not give false positives and include the index column as part of the unique column set.

dbt Project configurations

name: 'chabi'
version: '1.0.0'
config-version: 2
profile: "{{ env_var('DBT_PROFILE', '') }}"
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
target-path: "target"
clean-targets:
  - "target"
  - "dbt_packages"
  - 
models:
  stripe:
    +schema:
    intermediate:
      +schema: stg
  stripe_source:
    +schema: stg

vars:
  stripe_schema: stripe
  stripe_timezone: Australia/Sydney
  stripe__using_credit_notes: true

Package versions

packages:
  - package: dbt-labs/dbt_utils
    version: [">=1.0.0", "<2.0.0"]
  - package: dbt-labs/codegen
    version: 0.9.0
  - package: fivetran/stripe
    version: 0.9.0

What database are you using dbt with?

snowflake

dbt Version

Core:
  - installed: 1.4.1
  - latest:    1.4.5 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - snowflake: 1.4.0 - Update available!

  At least one plugin is out of date or incompatible with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Additional Context

No response

Are you willing to open a PR to help address this issue?

• [X] Yes.
• [ ] Yes, but I will need assistance and will schedule time during our office hours for guidance
• [ ] No.

[fivetran-joemarkiewicz]

Hi @jmussitsch thanks so much for opening this bug report. I would completely agree with you that it seems the index field was omitted from the staging model and the corresponding unique combination of columns test. It looks like this was missed on our end because our fee data only ever has a single index, which is why the test didn't fail on our end.

That being said, we should integrate this update into our package. I see you are open to creating a PR to address the issue. Are you still interested in contributing? If so there are a few changes we will want to apply:

1. Confirm the field index exists in the get_fee_columns macro. It looks like it does, so we are good there!
2. Add the index field to the stg_stripe__fee model. In particular within the final cte.
3. Add the index field name and description to the src_stripe.yml for the fee source table.
4. Finally, add the index field name and description along with the update to the unique combination of columns test to the stg_stripe.yml for the fee staging model.

After making those updates and confirming they work on your end, you should be good to open the PR and we can review and integrate into the next patch release! Otherwise, my team can tackle this bug fix in our upcoming sprint in two weeks time. Let me know if you have any questions!

[jmussitsch]

@fivetran-joemarkiewicz thanks for getting back to me on this. I created a PR:

https://github.com/fivetran/dbt_stripe_source/pull/64

[fivetran-joemarkiewicz]

@jmussitsch thanks again for opening this issue and providing the fix! I am happy to say that this should now be live in the v0.9.2 version of the stripe_source package!

As such, I will close this issue. Please feel free to reopen if you notice the issue still persists.

[jmussitsch]

@fivetran-joemarkiewicz thank you!