Open joleyjol opened 1 year ago
@joleyjol this looks similar to #837 , does the fix there solve this issue as well?
It looks like this should resolve my issue as well, thanks
@joleyjol , so the issue that you raised is to wrap
exception into DbtRuntimeError
but not on the SHOW TABLE EXTENDED is not supported for v2 tables
, right?
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.
Is this a new bug in dbt-spark?
Current Behavior
A dbt-spark project using the session connection method is unable to read Iceberg tables from Glue catalog due to a
pyspark.sql.utils.AnalysisException
withdesc = SHOW TABLE EXTENDED is not supported for v2 tables
.I did some digging and I think the issue is related to the exception_handler in
connections.py
In particular, this block:
I've verified that my job is hitting the
len(exc.args) == 0
condition, probably because I'm using the session connection method, but I haven't verified that.I was able to work around this error in my local environment by raising a
DbtRuntimeError
with thedesc
from the orginal exception, instead of just re-raising the original exception itself.Is there any reason this method should ever re-raise the original error instead of a
DbtRuntimeError
?Expected Behavior
The
pyspark.sql.utils.AnalysisException
should have been wrapped in aDbtRuntimeError
, and thus handled by the existing logic that checks for this specific error message to deal with Iceberg table metadata properly.Steps To Reproduce
1) Run dbt-spark in a project configured with the session connection method 2) Run a model that reads an Iceberg table from Glue 3) Observe that the run fails due to a
pyspark.sql.utils.AnalysisException
Relevant log output
Environment
Additional Context
No response