Add Livy server to Spark

juju-solutions / layer-apache-spark

Other

2 stars 10 forks source link

Add Livy server to Spark #9

Closed merlijn-sebrechts closed 7 years ago

merlijn-sebrechts commented 8 years ago

This PR incorporates Andrew's https://github.com/juju-solutions/layer-apache-spark/pull/7 and the following changes:

Run Livy as Hue user
Use correct log directory for spark
Separate hadoop-client, Spark and Livy dist settings in layer.yaml
(mostly) separate Spark and Livy handlers
Remove use of kv to store state in Livy and Spark class.

I'm not sure why the Livy dist config was under hadoop-client. It took me a while to understand what was really happening, so I just separated it to make it clearer.

merlijn-sebrechts commented 8 years ago

I tested this with the Hue Charm. Everything seems to be working. However, I noticed that Hue/Livy maintains a constant YARN job that block one slave from being used by other jobs. This is also the case for Zeppelin and Spark-notebook. It's not a real blocker, but it would be nice if we could find a way to allow other jobs to run on that node in parallel... Not sure if this is possible..

arosales commented 7 years ago

@galgalesh

I don't know where you currently are with Livy, but I wanted to chime in on this open PR and suggest a way forward.

The current PR is to add Livy directly into Spark. At the time of the PR we only had one Spark, but now we have 2 Spark charms for Apache upstream and BigTop. Most recent work has been put into the Bigtop charms. If we wanted to get Livy there we would also need to investigate modifying the bigtop-spark layer. This prompted me to think if we should consider Livy as a subordinate that could be deployed onto Apache, Bigtop, Cloudera, or others without modifications to the spark layer.

Thus, I wanted to get your thoughts on Livy as either its own layer that gets built into charms (ie not part of the spark layer itself), or as a subordinate?

Thoughts?

merlijn-sebrechts commented 7 years ago

I have no issue with Livy as a subordinate Charm. I propose to close this PR, Livy isn't a priority for us (tengu) at the moment so it will probably be a while until I reboot this work as a subordinate Charm.