search

cloudera / hue

Open source SQL Query Assistant service for Databases/Warehouses
https://cloudera.com
Apache License 2.0
1.18k stars 371 forks source link

spark + yarn: The Spark session could not be created in the cluster #860

Closed digitalbull closed 5 years ago

digitalbull commented 5 years ago

we have setup cdh 6.2 in our cluster, hue config as 

[desktop]
app_blacklist=
[notebook]
show_notebooks=true

livy 0.6.0 has been installed, and OS environment export as:

export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
export HADOOP_CONF_DIR=/etc/alternatives/hadoop-conf
export LIVY_LOG_DIR=/var/log/livy

after start livy we can't see the application information in YARN's web UI.

when create notebook in hue, submit a snippet we can not see the application information in YARN's web UI too.

we googled, and changed livy.conf as 

livy.spark.master = yarn
livy.spark.deploy-mode = cluster

after restart livy server, we submit the same snippet get error

The Spark session could not be created in the cluster: at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) 
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) 
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy10.mkdirs(Unknown Source) 
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2333) ... 20 more 
19/05/13 12:27:07 INFO util.ShutdownHookManager: Shutdown hook called 19/05/13 12:27:07 INFO util.ShutdownHookManager:
Deleting directory /tmp/spark-0d045154-77a0-4e12-94b2-2df18725a4ae YARN Diagnostics:
romainr commented 5 years ago

Does your logged-in user have a home dir on HDFS? (you should see the full error in the Livy or YARN logs)

In Hue admin for you user, you can click the 'Create home' checkbox and save.

digitalbull commented 5 years ago

I used root to start the yarn, so it failed, I changed a user to start livy, it's worked.
thanks your kindly advice.