Caching the endpoint IP

[hanyucui]

Is your feature request related to a problem? Please describe. When I deploy a new Livy server (on a new host) and point the domain name to it, I want existing sessions to keep using the old server since all the states and running jobs are still on the old server.

Describe the solution you'd like A candidate solution is to cache the IP of the Livy server in the Endpoint class. This way, updating the DNS record does not affect any current sessions.

Describe alternatives you've considered I considered updating Livy in place on the same host and other approaches that do not entail updating the DNS record. However, caching the IP in Endpoint seems the easiest and most flexible in my case. For example, in case the new deployment is bad, I can easily roll back just updating the DNS record.

I am happy to contribute if this makes sense.

[PedroRossi]

Hi @hanyucui ! This problem is really specific, In your use case this may work to cache the server IP, but I know that when using certain authentications like Kerberos this may stop working since we need the DNS to issue the service ticket. Is it possible in your scenario to just deploy a new Livy using the same HDFS in order to not lose any running sessions?

[hanyucui]

Hi @PedroRossi , thanks for getting back to me. Yeah, it is a bit specific to my needs, and I will certain keep it off by default and only turn it on for us. Wanted to know if it is entirely out of the question. Different projects have different philosophies in accommodating such needs, and I am fine either way.

Deploying a new Livy on a different host or port would be less transparent to users than we want. But using the same host and port would make the running sessions no longer accessible, right? I am not sure how feasible it is to update a running Livy in place while keeping all the sessions. Did I miss something?

[PedroRossi]

Hi @hanyucui , you can just configure your current Livy to set the recovery mode just like the following (the configuration code)

livy.server.recovery.mode = recovery
livy.server.recovery.state-store = filesystem
livy.server.recovery.state-store.url = hdfs://hdfs:port/path-to-livy-state/

with this you will be able to deploy a new Livy without losing any sessions, but you will still need to find a gap to deploy a new Livy with this option configured.

Regarding the feature itself I think it's too specific to actually deal with the resolved IP, but the problem on dealing with running sessions and migrations is relevant! I will tag @devstein @LinanZheng @itamarst in order to bring other points of view on this matter.

[hanyucui]

Thanks, @PedroRossi . That makes sense.