dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
https://getdbt.com
Apache License 2.0
9.99k stars 1.63k forks source link

[CT-3300] [Bug] dbt core process would stuck if error happens at certain stage #8959

Open ChenyuLInx opened 1 year ago

ChenyuLInx commented 1 year ago

Is this a new bug in dbt-core?

Current Behavior

If dbt-core run into an issue at certain part of the execution, the core running process would stuck without moving forward or raise any exception.

This is painful for both

Expected Behavior

dbt-core should raise the exception in those parts of the execution and continue execution as much as possible and not get stuck.

Steps To Reproduce

  1. in any dbt project, with a mounted version of core installed(via running make dev), make sure you can run dbt run.
  2. update either print_start_line or print_result_line in ModelRunner to raise an exception( I did raise NotImplementedError("not implemented"))
  3. run dbt run again
  4. watch the running process get stuck

Relevant log output

No response

Environment

- OS:Mac
- Python:3.11
- dbt: latest main(1.8.0-a1)

Which database adapter are you using with dbt?

No response

Additional Context

No response

TianyiLi1025 commented 8 months ago

I have problems with running models with multiple threads. In my case, I have no problems to run the models with 1 thread. But once more than 2 threads are used, the model running process just get stuck forever without any error message and I even can't stop it by ctrl+c. It is so weird that 20% percent of the time, it works with multiple threads. I do not know if this is also related to the issue you mentioned here. But currently, it is really annoying

Mickychen00 commented 4 months ago

Same problem. Is there any solution now?