search

dbt-labs / dbt-external-tables

dbt macros to stage external sources
https://hub.getdbt.com/dbt-labs/dbt_external_tables/latest/
Apache License 2.0
288 stars 116 forks source link

Doesn't work with spark 3.4.0 #234

Closed sid-deshmukh closed 10 months ago

sid-deshmukh commented 10 months ago

Describe the bug

The external tables creation is failing with spark 3.4.0 where as it works with pervious versions

Steps to reproduce

version: 2

sources:

Expected results

External table created in Glue catelog when run from EMR

Actual results

Its fails with below Error: Encountered an error while running operation: Compilation Error 'NoneType' object is not iterable

in macro stage_external_sources (macros/common/stage_external_sources.sql) called by

Which database are you using dbt with?

dataders commented 10 months ago

@sid-deshmukh can you please share what version of dbt-core dbt-spark and dbt-external-tables you are using? was there no change to these versions besides upgrading the version of Spark itself?

can you share the full error message as well as any relevant section of logs/dbt.log?

sid-deshmukh commented 10 months ago

@dataders There was no dbt-core and dbt-spark version changes. But I have tried with both 1.5.0 and 1.6.0, I get the same error. I have attached the logs for dbt.log dbt.log

dataders commented 10 months ago

closing in favor of https://github.com/dbt-labs/dbt-spark/issues/885. @sid-deshmukh I would very much love to see if you can