duckdb / dbt-duckdb

dbt (http://getdbt.com) adapter for DuckDB (http://duckdb.org)
Apache License 2.0
790 stars 70 forks source link

model with macro union gives incorrect results with 0.10.0 #345

Closed NielsHoffmann closed 4 months ago

NielsHoffmann commented 4 months ago

I have this model with a macro to generate a union:

{% set objecttype_query %}
select distinct source_definition from table_1
order by 1
{% endset %}

{% set results = run_query(objecttype_query) %}

{% if execute %}
{# Return the first column #}
{% set results_list = results.columns[0].values() %}
{% else %}
{% set results_list = [] %}
{% endif %}

{% for source_definition in results_list %}
(select 
 ...
from {{ ref('table_2')}} as objecten , {{ ref('table_3')}} as relaties
where objecten.Definition = '{{ source_definition }}'
...
{% if not loop.last %}  
UNION
 {% endif %}

{% endfor %}

with duckdb 0.9.2 this generates a table with 1049 rows (as expected) when I upgrade to duckdb 0.10.0 I get a table with only 500 rows... No other changes. running dbt-duckdb 1.7.2 in both cases.

What could cause this behaviour? It almost smells like a commit interval or checkpoint or something set a 500 and then skipping any furhter data?

jwills commented 4 months ago

Not quite sure how to help here w/o more info on what changed...

1) Is the compiled query different from 0.9.2 to 0.10.0? 2) If the query is the same, which rows in the output are different (i.e. which ones get dropped in 0.10.0?)

NielsHoffmann commented 4 months ago

I'm having troubles getting a reproducible case. However it does seem unrelated to dbt-duckdb after all, since I have the same issues querying my database though jupyter and/or dbeaver as well.