dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
https://getdbt.com
Apache License 2.0
9.29k stars 1.55k forks source link

[Bug] infinite loop in dbt core #10220

Open tricktx opened 1 month ago

tricktx commented 1 month ago

Is this a new bug in dbt-core?

Current Behavior

I need to run a dbt run on dbt core, but when I use the relative path, there is an infinite loop. I left a simple dbt running for 50 minutes and there is no return.

Here is the command used: dbt run --select models/br_camara_dados_abertos/br_camara_dados_abertos__legislatura.sql

Expected Behavior

I expected it to run dbt run.

Steps To Reproduce

  1. pip install poetry
  2. poetry install --with=dev && pre-commit install
  3. poetry shell
  4. dbt deps
  5. dbt run --select models/br_camara_dados_abertos/br_camara_dados_abertos__legislatura.sql

Relevant log output

There is an infinite loop after the following outputs: 

19:55:18 Running with dbt=1.8.1
19:55:18 Registered adapter: bigquery=1.8.1
19:55:20 Found 497 models, 2 operations, 1795 data tests, 1259 macros
19:55:20

Environment

- OS: WSL 2.1.5.0
- Python: 3.9.17 and 3.10
- dbt: 1.5.6 and 1.8.1

Which database adapter are you using with dbt?

bigquery

Additional Context

Captura de tela 2024-05-23 172311

Captura de tela 2024-05-23 171525

dbeatty10 commented 1 month ago

Thanks for reaching out @tricktx !

Are you saying that using this syntax returns in the expected amount of time?

dbt run --select br_camara_dados_abertos__legislatura

But this syntax results in an infinite loop?

dbt run --select models/br_camara_dados_abertos/br_camara_dados_abertos__legislatura.sql

If you change the logic of the model to be the following, does it still appear to be an infinite loop when you run it? Using this simple logic should help rule out the possibility of it just being computationally expensive logic that appears to be an infinite loop.

models/br_camara_dados_abertos/br_camara_dados_abertos__legislatura.sql

select 1 as id
tricktx commented 1 month ago

Thanks for the answer, @dbeatty10 .

Regarding syntax, I tested the models mentioned:

dbt run --select br_camara_dados_abertos__legislatura

and

dbt run --select models/br_camara_dados_abertos/br_camara_dados_abertos__legislatura.sql

and the infinite loop continues.

Regarding test

select 1 as id

The infinite loop error also remained, using both syntaxes

follows attachments:

Captura de tela 2024-05-28 152352

Captura de tela 2024-05-28 152536

dbeatty10 commented 1 month ago

Thanks for trying that out @tricktx. I don't have any ideas off-hand that would explain what you are seeing, and we'll need to reproduce it on our end to do anything about it. Are you able to reproduce similar behavior in a fresh project with a small number of files, by any chance?

Does it only happen with your br_camara_dados_abertos__legislatura model, or all models?

If it is just the br_camara_dados_abertos__legislatura, do you know anything different about the configuration of that model versus other models?