search

dbt-labs / dbt-spark

dbt-spark contains all of the code enabling dbt to work with Apache Spark and Databricks
https://getdbt.com
Apache License 2.0
403 stars 227 forks source link

show table extended in {{relation.shema}} like * fails #1099

Open rahulgoyal2987 opened 2 months ago

rahulgoyal2987 commented 2 months ago

Is this a new bug in dbt-spark?

Current Behavior

show table extended in {{relation.shema}} like * fails if any other dbt model is running on {{relation.shema}} same time.

Expected Behavior

Please change show table extended in {{relation.schema}} like '*' with below implementation to avoid parallel running model issue in same database. for table in spark.catalog.listTables({{relation.schema}}): spark.sql("show table extended in {{relation.schema}} like '{table.name}'")

Steps To Reproduce

Run two model parallel in same database

Relevant log output

No response

Environment

- OS: AWS EMR image
- Python: 3.11
- dbt-core: 1.0.0
- dbt-spark: 1.0.0

Additional Context

No response

amychen1776 commented 2 months ago

Hello @rahulgoyal2987 , you're currently on a EOL supported dbt core version. Could you update to 1.8 for both core and adapters and let me know if you're still experiencing this issue? We have made some significant changes with our metadata queries since 1.0.

luisarce83 commented 3 days ago

@amychen1776 we have dbt-core=1.8.1 and we still have the problem. Can you help us in the follow issue: https://github.com/dbt-labs/dbt-spark/issues/228. Thank you.