dbt-labs / dbt-spark

dbt-spark contains all of the code enabling dbt to work with Apache Spark and Databricks
https://getdbt.com
Apache License 2.0
391 stars 221 forks source link

show table extended in {{relation.shema}} like * fails #1099

Open rahulgoyal2987 opened 1 week ago

rahulgoyal2987 commented 1 week ago

Is this a new bug in dbt-spark?

Current Behavior

show table extended in {{relation.shema}} like * fails if any other dbt model is running on {{relation.shema}} same time.

Expected Behavior

Please change show table extended in {{relation.schema}} like '*' with below implementation to avoid parallel running model issue in same database. for table in spark.catalog.listTables({{relation.schema}}): spark.sql("show table extended in {{relation.schema}} like '{table.name}'")

Steps To Reproduce

Run two model parallel in same database

Relevant log output

No response

Environment

- OS: AWS EMR image
- Python: 3.11
- dbt-core: 1.0.0
- dbt-spark: 1.0.0

Additional Context

No response

amychen1776 commented 1 week ago

Hello @rahulgoyal2987 , you're currently on a EOL supported dbt core version. Could you update to 1.8 for both core and adapters and let me know if you're still experiencing this issue? We have made some significant changes with our metadata queries since 1.0.