Open jennlw opened 2 months ago
I should also note that I removed customer model names, and added fake ones. Please reach out to me if you need more information.
@jennlw has this behavior been seen in earlier dbt versions as well? When was this behavior first seen?
Hey Colin, I've reached out and I'm asking if it was seen prior to version 1.7. I'll let you know what I find.
Hey Colin, the customer stated that the issue they experienced was only on 1.7 as this piece of code had ran only on that version.
My goal was to whittle this down to the simplest way to reproduce this error as often as possible to gather insight from those results.
First, this is a very tricky issue to reproduce. I had much, much more luck on main
than 1.7.7
or 1.7.18
.
table
orders[1-100].sql
are either view
, table
, or materialized_view
(at one point I was testing with 500 but that took too long and I observed the concurrency error occurred early if it did at all)25
threads[debug] [Thread-1 (]: Applying DROP to: "ci"."dbt_mila"."stg_customers__dbt_backup"
[debug] [Thread-1 (]: Using redshift connection "model.jaffle_shop.stg_customers"
[debug] [Thread-1 (]: On model.jaffle_shop.stg_customers: /* {"app": "dbt", "dbt_version": "1.9.0b3", "profile_name": "redshift", "target_name": "dev", "node_id": "model.jaffle_shop.stg_customers"} */
drop table if exists "ci"."dbt_mila"."stg_customers__dbt_backup" cascade
[debug] [Thread-3 (]: Redshift adapter: Redshift error: could not complete because of conflict with concurrent transaction
stg_payments
table is where the concurrency transaction failure happens most often if not every timetable
materialization has this only happen rarely, if not once out of 10+ runs trying to trigger itview
materialization never failed out on me out of 10 runsmaterialized_view
materializations have this SUPER common, independent of on_configuration_change
-- all on main
1.7.18
Why is the focal point stg_payments
, the last of the non materialized view tables?
Why is its drop causing this error?
Why materialized views as the main issue here?
Is this a new bug in dbt-redshift?
Current Behavior
Job runs sporadically hitting "conflict with concurrent transaction error." The failure is inconsistent as the customer cannot reproduce it on every execution. On one of their dbt Cloud environment (5 threads) it happened twice but on our their local DBT core instances(20+ threads) they were able to reproduce it every third execution. Their local execution has another model victim.
They checked their logs and were not able to locate any other thread running parallel execution that would cause conflict.
Expected Behavior
Job run does not fail on concurrent transactions.
Steps To Reproduce
Not yet, but customer mentioned:
I had ran this past our team internally and it was mentioned:
Relevant log output
Environment
Additional Context
Please feel free to reach out to me and we can further discuss logs. I did not add them as they are customer logs and not dbt sandbox or test logs from dbt.