Open rosspalmer opened 8 months ago
+1 this makes it hard to create tests for code that uses multiple catalogs in production
+1
+1 Any workaround to be able to test 3-layer-namespace?
Hi! We're encountering the same problem. Have you made any progress or found a workaround to resolve this issue?
Any update or does anyone have a workaround for this ?
+1
Just confirming I am still being blocked by this. We have a "workaround" where we squish the catalog and database names together when running locally but its not pretty...
I am also facing this issue
I'd also like to see this address for unit tests
I've confirmed that (1) using Delta catalog and an Iceberg catalog works ✅ (2) using two Iceberg catalogs works ✅ (3) using two Delta catalogs fails ❌
WIP investigating why
+1 i'm facing the same problem while using 2 delta catalogs
+1
+1
+1
+1
is there any feedback on this? seems like iceberg is the only way to do this?
Bug
Which Delta project/connector is this regarding?
Describe the problem
As of Spark 3.4.0, native support for 3-layer-namespaces for tables was added into SQL API, allowing multiple catalogs to be accessed through using a full table name of the
<catalog>.<schema>.<table>
convention. Multiple catalogs can be set using thespark.sql.catalog.<catalog_name>=...
spark config.This works when using the Apache Iceberg example below, but does not work when utilizing multiple Delta catalogs. While the SparkSession is initiated with the catalog present in the session, when a second, non
spark_catalog
catalog is referenced, the following exception is thrown.[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace.
Here is a recent StackOverflow post experiencing the same issue with PySpark: https://stackoverflow.com/questions/77751057/multiple-catalogs-in-spark
Steps to reproduce
I am running this on my local machine, in client mode, using my local filesystem to host data.
Here is my SparkSession generator:
Here is a batch of testing code:
Observed results
When running with the example above, the
spark.catalog.setCurrentCatalog("catalog_b")
command works but then the followingspark.sql("create schema here")
command throws the exception below:Expected results
I would expect this to create a schema
here
in the catalogcatalog_b
and allow me to save data to it.Further details
This is an effort to create a local "delta lake" for testing which can be compatible with Databrick's three layer namepace used by their Unity Catalog.
Environment information
Willingness to contribute
The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?