search

sqlfluff / sqlfluff

A modular SQL linter and auto-formatter with support for multiple dialects and templated code.
https://www.sqlfluff.com
MIT License
7.89k stars 726 forks source link

Issue with Databricks absolute path tables #602

Closed Fokko closed 2 years ago

Fokko commented 3 years ago

We're using DBT with Databricks and tried to implement sqlfluff for linting. However, we're seeing that the following syntax is unsupported:

SELECT *
FROM delta.`/mnt/datalake/table`

Is this something that we want to support in sqlfluff? If so, I can try to come up with a PR.

dmateusp commented 3 years ago

hi @Fokko, glad to see you here!

sqlfluff has a modular approach to dialects, you should be able to see all the dialects here: https://github.com/sqlfluff/sqlfluff/tree/master/src/sqlfluff/core/dialects

I'm thinking you would need to implement a Spark SQL dialect (which would be awesome to have)

Related #417

R7L208 commented 3 years ago

This is not specific to Databricks but the Delta Lake OSS project & Spark DataSourceV2 classes.

The Delta dialect should probably be an extension of Spark SQL and a valid file format in DBT and HQL.

https://docs.delta.io/latest/delta-batch.html#read-a-table

https://github.com/delta-io/connectors/#create-a-hive-table https://docs.getdbt.com/reference/resource-configs/spark-configs#configuring-tables

R7L208 commented 2 years ago

@tunetheweb - Can you change the label from new dialect to spark3?

Spark docs have been updated recently. This should be supported in Spark dialect then Databricks (which is SaaS Spark) could inherit it.

https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-file.html

tunetheweb commented 2 years ago

@tunetheweb - Can you change the label from new dialect to spark3?

Done!