Open stevenstetzler opened 4 years ago
I seem to have gotten it to work by adding --conf spark.sql.hive.metastore.sharedPrefixes=org.apache.derby
to the pyspark command line; want to check if that also works for you?
It looks like this fixes the issue for me on both Epyc and the cloud system, thanks @ctslater. I guess this should be a default configuration in the spark-defaults.conf
that we ship with AXS?
@ctslater and I have been investigating an issue where it seems that more than one connection is being made by Spark to the Hive metastore when backed by a local Derby database. The Derby database only allows a single connection at a time, so a crash occurs when a second connection is attempted. On epyc, we use a shared MySQL database, so it seems we are blind to this issue between version changes. This bug originally appeared on our AWS JupyterHub where each user is using a local Derby database instead of a shared one.
The following is enough to reproduce the bug with Spark 3.0.0 and to work without the bug for Spark 2.4.0:
@ctslater to reproduce this on epyc, navigate to:
/epyc/users/stevengs/spark-testing
and doand copy in the code above, which should work. Then do
and copy in the code above again, which should fail with a traceback like:
the source files simply change
SPARK_HOME
,PATH
, andSPARK_CONF_DIR
to point to the right version of Spark on epyc and also to load special configurations inhive-site.xml
that specify using a local Derby database instead of the database running on epyc.The following is the
hive-site.xml
for 2.4.0and the following is the
hive-site.xml
for 3.0.0