MI-DPLA / combine

Combine /kämˌbīn/ - Metadata Aggregator Platform
MIT License
26 stars 11 forks source link

Spark cluster and Livy sessions #236

Closed ghukill closed 6 years ago

ghukill commented 6 years ago

Moving to Spark cluster on Combine server, as opposed to "local" instance via Livy, prompts a few todos, and provides a good opportunity to revisit how users manage Livy sessions

As for Livy sessions, users should have the ability to start/stop/restart them if desired (e.g. code changes), but shouldn't have to concern themselves with them for running Jobs.

ghukill commented 6 years ago

Spark cluster mostly in place, confirmed new build on dev branch with Ubuntu 18.04. Impetus for this work was better handling Livy, looking into that now...

ghukill commented 6 years ago

Closing, in favor of more specific issues.