search

fal-ai / dbt-fal

do more with dbt. dbt-fal helps you run Python alongside dbt, so you can send Slack alerts, detect anomalies and build machine learning models.
https://fal.ai/dbt-fal
Apache License 2.0
851 stars 71 forks source link

Show line number for Python model errors in dbt run #771

Open kinghuang opened 1 year ago

kinghuang commented 1 year ago

Context

When an exception is raised in a Python model run via dbt, the error output shows the string representation of the exception, but not the error type or the line number of where the exception was raised. This makes it really hard to locate where the error occurred. The log message should include the error type and line number. And, perhaps there should be an option to show or capture an entire traceback.

Is your feature request related to a problem? Please describe.

I find it difficult to debug Python models because it is hard to pinpoint where in the code an exception was raised.

Describe the solution you'd like

The error message should include the exception class and the line number from the traceback.

So, instead of the following log message:

not enough values to unpack (expected 3, got 2)

I would like to see something like:

File "wells_index__keys.py", line 107: ValueError: not enough values to unpack (expected 3, got 2)

Describe alternatives you've considered

Run often to narrow the possible scope for an exception to occur.

Additional context

Here is an excerpted output from dbt run, showing an error occurred with the message "not enough values to unpack (expected 3, got 2)".

18:50:18  1 of 2 START python table model dbt_king_dibi_env.wells_index__keys ............ [RUN]
18:51:14  Unhandled error while executing target/run/dibi_env/models/foundations_wells/wells_index__keys.py
not enough values to unpack (expected 3, got 2)
18:51:14  1 of 2 ERROR creating python table model dbt_king_dibi_env.wells_index__keys ... [ERROR in 55.56s]
18:51:14  2 of 2 SKIP relation dbt_king_dibi_env.wells_index__full ....................... [SKIP]
18:51:15  
18:51:15  Finished running 2 table models in 0 hours 0 minutes and 59.85 seconds (59.85s).
18:51:15  
18:51:15  Completed with 1 error and 0 warnings:
18:51:15  
18:51:15  not enough values to unpack (expected 3, got 2)
18:51:15  
18:51:15  Done. PASS=0 WARN=0 ERROR=1 SKIP=1 TOTAL=2

The raised exception is a ValueError from code on line 107. But, without this information in the run output, it is very hard to figure out where the problem is in the Python model.

Is there an existing feature request for this?

HlebKrah commented 1 year ago

+1 It would be great if this functionality were implemented. Now it’s not clear where the error falls and you have to transfer the model to Jupiter notebook or put a bunch of loggers