Open naumovdalex opened 1 year ago
We noticed the same with our post-hook, what helped us to set autocommit: False
in profiles.yml
(more details here)
@naumovdalex sorry to hear your existing workflows is failing. As part of the migration to using the redshift_connector
connector library in 1.5
instead of psycopg2
, this adapter's approach to transactions have changed such that every query is executed at once.
Check out #463 for more context, and I'd love to see if you get the the intended behavior by doing as @jan-benisek says and adding autocommit: False
to your profiles.yml
.
I suspect that you may still encounter issues due to design decisions of the underlying connector library, for example see https://github.com/aws/amazon-redshift-python-driver/issues/162.
@naumovdalex I'm going to close this issue for now. If possible, I think this would make a great reproducible use case to rationalize https://github.com/aws/amazon-redshift-python-driver/issues/162. Perhaps open an issue there detailing the limitation?
We noticed the same with our post-hook, what helped us to set
autocommit: False
inprofiles.yml
(more details here)
I tried this, unfortunately that didn't help as well. Also, as part of our team uses dbt Cloud IDE as well, this would be a bit more complicated for us
@naumovdalex sorry to hear your existing workflows is failing. As part of the migration to using the
redshift_connector
connector library in1.5
instead ofpsycopg2
, this adapter's approach to transactions have changed such that every query is executed at once.Check out #463 for more context, and I'd love to see if you get the the intended behavior by doing as @jan-benisek says and adding
autocommit: False
to yourprofiles.yml
.I suspect that you may still encounter issues due to design decisions of the underlying connector library, for example see aws/amazon-redshift-python-driver#162.
@dataders it's not that the workflows fail, but some macros are taking more than 5 minutes to execute, instead of just a few seconds
I understand now -- that not an ideal experience. @naumovdalex can you please provide a reproducible example? This would help the engineer at Redshift work on the above linked issue on their connector library
@naumovdalex are you still experiencing this issue in the new versions of dbt-redshift?
Is this a regression in a recent version of dbt-redshift?
Current Behavior
We have a macro in dbt that runs a set of DDL statements on Redshift. We concatenate those statements in a string (
cmd
) and run with"{% do run_query(cmd) %}"
. That string looks like below;After moving to dbt 1.5, we see that those statements are executed one by one, instead of sending the statement as a whole to database. We can see it both in database logs and dbt logs.
Impact
In 1.5 and onwards, these statements take more than 5 minutes to execute. Previously in versions 1.4 and below, they would execute in only a few seconds.
Expected/Previous Behavior
The below statements are executed at once in a single transaction.
Steps To Reproduce
"{% do run_query(cmd) %}"
Relevant log output
Environment
Additional Context
No response