Open jakubro opened 3 weeks ago
Hi @jakubro ! Thank you for opening this bug report. In examining this, there are multiple threads to dig into:
dbt-redshift 1.5 --> versionless. The major change here is that we migrated from using the posgres driver to the Redshift connector (working with the Redshift team). With that change, it seems that the redshift driver does not support retries on this database issue inherently which the previous driver does.
Due to Redshift not supporting this in the connector, we will work with our Redshift counterparts to discuss this. What would help us is if you open up a AWS support ticket so that we can track this (and show the customer requirement) to the team.
dbt concurrent runs - this is not something that dbt has recommended or supported out of the back and leads to situations cases like yours. For this - I am updating this issue to call out this request so that I can track it but it's not something that we seek to support on our current roadmap.
Thank you @amychen1776 for the explanation.
Re 2) We've opened a ticket with AWS per your suggestion.
Re 3) Do you by any chance have any guidance in documentation on how to structure jobs in DBT cloud?
Awesome - thank you. And in terms of jobs, here is our communication on how to set up jobs (including how to chain jobs) https://docs.getdbt.com/docs/deploy/deploy-jobs#trigger-on-job-completion--
Is this a new bug in dbt-redshift?
Current Behavior
We're observing a high failure rate after upgrading our DBT pipelines from version 1.5 to Versionless.
Our pipelines are failing with this error:
And the failing query is this:
My understanding of querying
information_schema
is that Redshift returns an error if a table is dropped while the query runs. And, DBT creates and drops some transient tables as it runs.It's important to note that we have 17 pipelines that run simultaneously. Most of them run every 10 minutes and some every 2 minutes.
Expected Behavior
DBT commands do not fail on quering
information_schema
.I would expect DBT to retry the query as it tries to build the catalog.
Steps To Reproduce
simple_model
Relevant log output
Attaching logs from the reproduction example from both terminals at the time of failure.
Logs from
dbt --debug run -s models/simple_model.sql
:Logs from
dbt --debug docs generate -s models/simple_model.sql
:Environment
Additional Context
pip freeze: