search

dbt-labs / dbt-athena

The athena adapter plugin for dbt (https://getdbt.com)
https://dbt-athena.github.io
Apache License 2.0
228 stars 100 forks source link

Support Athena Spark #144

Open sanga8 opened 1 year ago

sanga8 commented 1 year ago

AWS recently released Athena Spark. Being able to use it would be an awesome addition

mattiamatrix commented 1 year ago

Just announced https://aws.amazon.com/about-aws/whats-new/2023/03/amazon-athena-apache-spark-4-regions/

nicor88 commented 1 year ago

https://github.com/dbt-athena/dbt-athena/pull/188 should support this feature

sanga8 commented 1 year ago

I tried using the spark_workgroup as in https://github.com/dbt-athena/dbt-athena/pull/188 but I got "Materialization "materialization_table_athena" only supports languages ['sql']; got "python"

svdimchenko commented 1 year ago

we have provided some tests with athena-spark workgroup and were not satisfied with the results. To be short, it seems to be more expensive and at least does not provide performance boost (or even is slower then usual athena in many cases). We'll need to implement other sql dialect support for spark as well. Does this issue make sense if there is https://github.com/aws-samples/dbt-glue for spark ?