Closed ghukill closed 6 years ago
If uploading jars to remote workers, also need to whitelist those directories in livy.conf
, e.g. /usr/share/java/
These have largely been implemented for default builds. Binding addresses and hosts are generalized to 0.0.0.0
and local where possible, but are listening outside as well.
Performance wise, running as a cluster show clear advantages over local
in Livy. And, it positions it nicely for other nodes to participate. Closing.
Some preliminary testing on what kind of configurations would need to be modified to support a spark cluster, as opposed to running locally via Livy.
0.0.0.0
jdbc_url
setting inlocalsettings.py
must be IP of Combine machine, as this will be used by remote executors to connect tocombine
user must be okay on%
any IP (notlocalhost
)'jars':[ 'file:///usr/share/java/mysql.jar' ]
local
to point at spark cluster, e.g.livy.spark.master = spark://combine_ip_running_master:7077
PYSPARK_PYTHON
setting from Livy instructs workers where to find python, but might be different for different machines?Running as standalone cluster on machine appears to fairly dramatically speed up processing (leverages multiple cores), but will need some more testing and figuring to see how Combine can manage this process.