dbt-labs / dbt-spark

dbt-spark contains all of the code enabling dbt to work with Apache Spark and Databricks
https://getdbt.com
Apache License 2.0
405 stars 227 forks source link

[ADAP-873] [Regression] `1.6`does not work with `method: thrift` due to `pyhive`'s lack of `Cursor.fetchmany()` method #885

Closed dataders closed 6 months ago

dataders commented 1 year ago

Is this a regression in a recent version of dbt-spark?

Current Behavior

reports & discussion

@sid-deshmukh originally opened https://github.com/dbt-labs/dbt-external-tables/issues/234, but I believe this issue to be with dbt-spark, not dbt-external-tables.

@timvw and @jelstongreen also reported in a #db-databricks-and-spark thread in Community Slack there were experiencing similar issues

for reference, here's our internal dbt Labs Slack thread

stacktrace

compiling fails with the following stacktrace. dbt calls .get_result_from_cursor() which calls cursor.fetchall() which in PyHive is passed to it's Cursor._fetch_more() (pyhive/hive.py#L507), where it fails.

columns = [_unwrap_column(col, col_schema[1]) for col, col_schema in
           zip(response.results.columns, schema)]
full stacktrace ```py File "/Users/user/PycharmProjects/dbt-data-pipeline/venv/lib/python3.8/site-packages/dbt/clients/jinja.py", line 302, in exception_handler yield File "/Users/user/PycharmProjects/dbt-data-pipeline/venv/lib/python3.8/site-packages/dbt/clients/jinja.py", line 257, in call_macro return macro(*args, **kwargs) File "/Users/user/PycharmProjects/dbt-data-pipeline/venv/lib/python3.8/site-packages/jinja2/runtime.py", line 763, in __call__ return self._invoke(arguments, autoescape) File "/Users/user/PycharmProjects/dbt-data-pipeline/venv/lib/python3.8/site-packages/jinja2/runtime.py", line 777, in _invoke rv = self._func(*arguments) File "