Closed jstammers closed 1 month ago
Thanks for reporting @jstammers!
The fact that you can set a catalog and database (schema) via the SQL interface but not using the pyspark API is... frustrating. I can poke around at making those catalog and database calls via SQL instead.
Are you available to test out those changes if I push up a PR? It's hard for me to test because I don't have access to a cluster running in shared access mode.
Sure, I'd be happy to test any changes if that would be helpful.
Great, thanks!
Just for the papertrail, we currently have context managers that make use of the pyspark API to set the catalog and database before performing any operations like creating or loading tables (where table location is obviously important).
This issue seems to indicate that there is a mismatch in permission structure for certain clusters for operations performed via the pyspark API vs via directly using Spark SQL.
My thinking here is to swap out the use of the pyspark API and instead set location hierarchy via spark SQL instead, e.g.
USE CATALOG catalog;
USE SCHEMA database;
That makes sense to me. If we can leverage the spark SQL syntax, then that could be a good work-around. We might need to consider the fact that, as far as I'm aware, the spark SQL command sets the catalog and schema globally, which could result in unintended errors, e.g.
table = con.table("my_table") # assuming my_table exists in default catalog/schema
# assuming catalog.schema.my_table_2
new_table = con.table("my_table_2", database=("catalog", "schema"))
# error if catalog.schema.my_table doesn't exist or differs from default
table = con.table("my_table")
Yep, definitely something to be wary of.
If we follow the current pattern, we would set the catalog and schema for the current query execution, then restore it to the previous value, so it should be relatively robust, but "should" is doing a lot of work there.
What happened?
I am unable to read from a table in a databricks unity catalog when using a compute cluster with a shared access mode When I try to load the table using
I receive the following error
From a quick google search, I think this is related to the fact that certain spark functions are not enabled on clusters with a shared access mode (e.g. here).
If I manually set the catalog and schema using SQL statements
I am then able to load the table using
For further context, I have not encountered these issues when using a personal compute cluster that has a single-user access mode
What version of ibis are you using?
9.1.0
What backend(s) are you using, if any?
pyspark - 3.4.1
Relevant log output
Code of Conduct