search

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.46k stars 1.68k forks source link

[BUG] cannot #2793

Open jelmerk opened 6 months ago

jelmerk commented 6 months ago

Bug

Which Delta project/connector is this regarding?

Describe the problem

As of Spark 3.4.0, native support for 3-layer-namespaces for tables was added into SQL API

3-layer-namespace Ticket Spark 3.4.0 release notes

However when trying to use this with DeltaTable you get an error

Steps to reproduce

DeltaTable
  .createIfNotExists(spark)
  .tableName("spark_catalog.some_database.some_table")
  .addColumn("id", "long", nullable = false)
  .execute()

Observed results

[PARSE_SYNTAX_ERROR] Syntax error at or near '.'.(line 1, pos 25)

== SQL ==
spark_catalog.some_database.some_table
-------------------------^^^

org.apache.spark.sql.catalyst.parser.ParseException: 
[PARSE_SYNTAX_ERROR] Syntax error at or near '.'.(line 1, pos 25)

== SQL ==
spark_catalog.some_database.some_table
-------------------------^^^

    at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(parsers.scala:257)
    at org.apache.spark.sql.catalyst.parser.AbstractParser.parse(parsers.scala:98)
    at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:54)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseTableIdentifier(AbstractSqlParser.scala:41)
    at io.delta.sql.parser.DeltaSqlParser.parseTableIdentifier(DeltaSqlParser.scala:146)
    at io.delta.tables.DeltaTableBuilder.$anonfun$execute$1(DeltaTableBuilder.scala:316)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900)
    at org.apache.spark.sql.delta.DeltaTableUtils$.withActiveSession(DeltaTable.scala:491)
    at io.delta.tables.DeltaTableBuilder.execute(DeltaTableBuilder.scala:306)

Expected results

For it to create a some_table table in the some_database database of the spark_catalog session catalog,

Further details

The root cause seems to be that the core spark parser does not support catalogs. this throws the same error

spark.sessionState.sqlParser.parseTableIdentifier("spark_catalog.some_database.some_table")

Environment information

Willingness to contribute

The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?

felipepessoto commented 6 months ago

Is this duplicate of #2434?