Closed lpezet closed 1 year ago
Perhaps the following implementation for backwards compatibility:
{% macro incremental_delete(tmp_relation, target_relation, unique_key=none, statement_name="pre_main") %}
{%- if unique_key is not none -%}
delete
from {{ target_relation }}
where ({{ unique_key if unique_key is string else unique_key | join(',') }}) in (
select {{ unique_key if unique_key is string else unique_key | join(',') }}
from {{ tmp_relation }}
)
{%- endif %}
{%- endmacro %}
@moszutij Your solution is definitely better. I ran my "solution" (let's call it a quick hack) before and all tests pass. That means even the standard dbt-tests-adapter tests pass. I'll add some tests to make sure my hack would fail those. It is surprising those cases are not tested by dbt-tests-adapter (I'm referring to this: https://github.com/dbt-labs/dbt-tests-adapter/blob/main/dbt/tests/adapter/basic/test_incremental.py). Maybe I missed something or I'm looking at the wrong thing?
After examining your solution, I decided to give it a try, but later realised that it only worked for a list of column names enclosed in single quotes, and not for a string representation. However, the documentation suggests that both formats are acceptable, hence the suggested implementation. I haven't had the chance to familiarise myself with the standard dbt-tests-adapter tests yet, but it's on my to-do list 👩💻. I'm also using dbt with mysql, so appreciate the pull requests.
@moszutij Thanks for checking it out. I really appreciate. I actually just found those tests in dbt-core itself: https://github.com/dbt-labs/dbt-core/blob/1.2.latest/tests/adapter/dbt/tests/adapter/incremental/test_incremental_unique_id.py Way more thorough than what I came up with. I'll update my branch for PR to leverage that test.
There are even more tests in 1.4.latest (shameless plug for my PR #146) when we get there.
Thanks for your time and effort into this @lpezet and @moszutij ! 🏆
I'm out of pocket this week, but looking forward to collaborating with you next week to get these merged and released.
Describe the bug
When using array of columns for
unique_key
in incremental materialization, the macroincremental_delete
chokes and the following error is thrown, using MySQL 8. I'm using code from main branch.Example of model:
Example of seed:
Exception thrown when running
dbt run
:Steps To Reproduce
Expected behavior
Exception not thrown and rows from seed deleted and then re-inserted without creating duplicates.
The operating system you're using: Linux GoldenHind 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
The output of
python --version
: Python 3.8.10MySQL Server 8.0.29
Additional context
The branch
fix-incremental-composite-keys
actually contains the test to ,well, test incremental materialization with composite keys. I couldn't find the equivalent indbt-tests-adapter
code...strangely...I must have missed something. I therefore created the classtests/functional/adapter/incremental_composite.py
based on dbt-tests-adapter's very own BaseIncremental class and added a test intests/functional/adapter/test_basic.py
calledTestIncrementalCompositeKeysMySQL
.The problem, I believe, is in 2 parts:
select ( {{ unique_key }} )
to get values for thein ()
.I believe the following would fix the issue:
NB: No *GPT has been harmed, or even used!, during the creation of all this. Just lots of time spent figuring out dbt-tests-adapter and dbt-mysql, git-ing, coding, and typing this very long ticket. Just like old (?) times ;)