search

DataRecce / recce

The dbt data-validation toolkit for teams that care about building better data
https://datarecce.io
Apache License 2.0
223 stars 4 forks source link

[DRC-575] [Bug] Advanced Diffs Return Errors in dbt-athena #392

Closed tkrishna-equip closed 1 week ago

tkrishna-equip commented 1 month ago

Current Behavior

All Advanced Diffs run for a long time and return errors

Expected Behavior

Advanced Diffs should run quickly and without errors

Steps To Reproduce

Configure this environment: dbt core: version 1.6.15 dbt-athena: version 1.6.3 python: 3.9.6

Select Advanced Diffs and try running

Relevant log output

No response

Environment

Additional Context

image image

DRC-575

popcornylu commented 1 month ago

For value diff, it seems a issue from audit helper https://github.com/dbt-labs/dbt-audit-helper/issues/87

Please see the workaround solution a paste in the issue.

popcornylu commented 1 month ago

The problem for top-k and histogram, it is fixed in #395 and will be available at the next release. I will updated the release number here once it is releaased.

For valude diff and profile diff. I found that there are compatibility issues with audit-helper and dbt-profiler on AWS Athena.

Just created a PR in audit-helper https://github.com/dbt-labs/dbt-audit-helper/pull/108 and created a PR in dbt-profiler https://github.com/data-mie/dbt-profiler/pull/91 A current walk-around solution is to put https://github.com/DataRecce/recce/blob/main/macros/recce_athena.sql in marcros folder of your dbt project.

popcornylu commented 1 week ago

The upstreams' PRs aren't merged. I close this issue first, please use the workaround solution mentioned above.