ClickHouse / dbt-clickhouse

The Clickhouse plugin for dbt (data build tool)
Apache License 2.0
255 stars 113 forks source link

Bug/223 relationship test with limit #245

Closed BentsiLeviav closed 9 months ago

BentsiLeviav commented 10 months ago

Summary

As mentioned in issue #223, dbt fails to execute relationship tests when a limit clause is specified. The issue arises because the relationship test macro appends the join_use_nulls settings, and the limit is added at the end of the query (after the settings), causing a parsing error.

To address this limitation of adding a limit clause when the query includes settings, we developed a new macro that parses the query and inserts the limit just before the settings clause.

This is accomplished by introducing a flag within a comment, which is then used to parse the settings section accordingly.

Checklist

Delete items not relevant to your PR:

genzgd commented 9 months ago

Maybe we could always add an -- end_of_sql marker (regardless of whether there are settings), instead of a -- settings_section marker only if there are settings. That would allow finding the last "-- end_of_sql" marker and insert the limit there, regardless of any nesting that may happen in building the query.

Since there aren't any real automated dbt tests around the dbt test command, has this been validated manually?

BentsiLeviav commented 9 months ago

@genzgd Yes, -- end_of_sql is better. I changed the code. I ran the tests, and everything seems to be OK besides 8 tests (which were broken even before my changes).