Moving to Spark cluster on Combine server, as opposed to "local" instance via Livy, prompts a few todos, and provides a good opportunity to revisit how users manage Livy sessions
provide links to Spark cluster GUI @ :8080
allows users to restart Spark cluster via supervisor processes
all would be restarting spark:, or driver at spark_driver and worker at spark_worker
rename "Livy/Spark Session" link to "Spark Cluster" or something more general?
As for Livy sessions, users should have the ability to start/stop/restart them if desired (e.g. code changes), but shouldn't have to concern themselves with them for running Jobs.
consider setting in localsettings, AUTOSTART_LIVY_SESSION that would automatically fire a Livy session if not active
refactor some of the code for confirming active livy sessions
specifically when CombineJob fires a Job, if a session is not active, it just sets as False and fails
Spark cluster mostly in place, confirmed new build on dev branch with Ubuntu 18.04. Impetus for this work was better handling Livy, looking into that now...
Moving to Spark cluster on Combine server, as opposed to "local" instance via Livy, prompts a few todos, and provides a good opportunity to revisit how users manage Livy sessions
:8080
spark:
, or driver atspark_driver
and worker atspark_worker
As for Livy sessions, users should have the ability to start/stop/restart them if desired (e.g. code changes), but shouldn't have to concern themselves with them for running Jobs.
localsettings
,AUTOSTART_LIVY_SESSION
that would automatically fire a Livy session if not activeCombineJob
fires a Job, if a session is not active, it just sets as False and fails