Until now, Combine users were responsible for starting / stopping Livy a single session to run Jobs with. This is not readily apparent to new users, nor is it really necessary for them to be concerned with. While the ability should remain for manual stopping / restarting, it would nice if a Livy sessions were handled automatically for users.
Considerations:
when user runs Spark Job, check for active Livy session
if found, use
if not found, start and use
How, then, to handle stopping Livy session, if automatically started for Jobs?
Assume that stopping is sufficient, and will remain stopped, until another automatic call to init?
Or, key off difference between stale Livy session in DB and absence of one (stale would get restarted, absence would not)?
if Livy was manually stopped, then would be on user to restart in this situation
ghukill
commented
5 years ago
Until now, Combine users were responsible for starting / stopping Livy a single session to run Jobs with. This is not readily apparent to new users, nor is it really necessary for them to be concerned with. While the ability should remain for manual stopping / restarting, it would nice if a Livy sessions were handled automatically for users.
Considerations:
How, then, to handle stopping Livy session, if automatically started for Jobs?