exploring spark cluster

[ghukill]

Some preliminary testing on what kind of configurations would need to be modified to support a spark cluster, as opposed to running locally via Livy.

• mysql
  • server bind must be 0.0.0.0
  • jdbc_url setting in localsettings.py must be IP of Combine machine, as this will be used by remote executors to connect to
  • combine user must be okay on % any IP (not localhost)
• livy
  • must push out jars to remote workers if they don't have them, e.g. mysql, which can be configured a la: 'jars':[ 'file:///usr/share/java/mysql.jar' ]
  • livy conf would need to change from local to point at spark cluster, e.g. livy.spark.master = spark://combine_ip_running_master:7077
  • PYSPARK_PYTHON setting from Livy instructs workers where to find python, but might be different for different machines?
• spark
  • defaults would need some tweaking to push more ram / cores at executors

Running as standalone cluster on machine appears to fairly dramatically speed up processing (leverages multiple cores), but will need some more testing and figuring to see how Combine can manage this process.

[ghukill]

If uploading jars to remote workers, also need to whitelist those directories in livy.conf, e.g. /usr/share/java/

[ghukill]

These have largely been implemented for default builds. Binding addresses and hosts are generalized to 0.0.0.0 and local where possible, but are listening outside as well.

Performance wise, running as a cluster show clear advantages over local in Livy. And, it positions it nicely for other nodes to participate. Closing.