dbt-labs / dbt-spark

dbt-spark contains all of the code enabling dbt to work with Apache Spark and Databricks
https://getdbt.com
Apache License 2.0
395 stars 221 forks source link

Submit scala jobs beta #891

Closed pekapa closed 6 months ago

pekapa commented 1 year ago

Problem

Starting on version 1.3 DBT Python models became available. Another very common programming language for submitting spark jobs is Scala. This PR enables Scala models to be built with DBT

Solution

Using the structure created by Python models we can extend it to also support Scala models. The parser for Scala is not widely available so a "cheap" version was built to provide just the minimum needed for it. More complex solutions (using ANTLR, for example) might be desired in the future. Since the Python validator is missing tests we are also skipping those for now here. Same for typed annotations, it follows the precedent set by the Python models code.

Related dbt-core PR: dbt-labs/dbt-core/pull/8701

Checklist

github-actions[bot] commented 6 months ago

This PR has been marked as Stale because it has been open with no activity as of late. If you would like the PR to remain open, please comment on the PR or else it will be closed in 7 days.

github-actions[bot] commented 6 months ago

Although we are closing this PR as stale, it can still be reopened to continue development. Just add a comment to notify the maintainers.