handle communication with kernel launchers external to k8s

[xhumanoid]

Hi Now we have this sequence for communication:

1) egw start listening specific port 2) egw communicate with remote service for run kernel_launcher and pass address and port as parameter 3) remote service allocate resources and execute kernel_launcher script 4) script back connect to egw port and send kernel information

main problem: egw must reserve range of ports it's little difficult to support and maintain when egw deployed as service in cloud environment like kubernetes

proposal: we can use existing rest api

1) egw generate attempt id and register callback with timeout 2) egw communicate with remote service for run kernel_launcher and pass http(s) address as parameter 3) remote service allocate resources and execute kernel_launcher script 4) script back connect to egw http(s) and send kernel information 5) egw execute callback function

it's simplify devops support (you need maintain just 1 port for backconnect) and remove requirement to reserve range of ports.

potentially this also can solve problems like #86

what do you think? @kevin-bates @lresende

[kevin-bates]

@xhumanoid - thank you for the issue.

The port used for the response address is ephemeral - only lasting the duration of the kernel's startup. Once the connection information is returned, the port is destroyed. As a result, I'm not certain there's an issue here.

If you need to ensure ports are allocated from a specific range, you could use --EnterpriseGatewayApp.port_range='<lower>..<upper>' to isolate and range used. However, remember that this value automatically flows to each kernel launched, so if that range is not also applicable to kernels, you'll want to add --RemoteProcessProxy.port-range='0..0' to each kernelspec's argv parameter set to disable port ranges (or specify different values).

Passing the REST URI (and adding support for a new endpoint) is interesting, but, I believe, presents its own set of issues. For one, this would be an internal-only endpoint, and I'm not sure how that would be controlled/documented, etc. In addition, we want to support load balancers, and this endpoint would need to ensure it gets directed to the same EG server that is launching the kernel (although that could perhaps be resolved by specifying the specific node). There's also authentication that typically occurs at that layer, etc. Since the port is short-lived, I feel we're better off leaving things as they are.

Regarding #86, this is more about the fact that the jupyter framework does not plumb coroutines down through the hierarchy so we're not able to get asynchronous behavior. This results in a single-threaded startup - which gets exacerbated when dealing with remote kernels - due to their latencies to startup - that isn't evident in the default, local-only, behavior.

Can you describe further the port restrictions when using EG in k8s cloud environments? Do each of the kernel images (which create 6 ports for each kernel - 5 ZMQ ports and an EG-communication port) have the same issues?

[xhumanoid]

We have jupyter kernels in docker yarn, but EG in k8s as service. also we investigate HA configuration now (current session manager contain issue with restore from json, we push fixes soon)

so port range mostly affected EG itself, kernel images fully ok.

I know about port have short live, but it's addition configuration for mapping and security firewall configuration. We already use port range limits, because can't just provide all range.

[kevin-bates]

ok, I see. Only EG is in k8s so the ports used for the response address are not exposed to anything outside the k8s cluster and, if I recall, the IP generated for the response address will likely be the k8s cluster IP - which won't have meaning either.

Is there a reason why you are deploying EG in k8s? The design center for EG and K8s is that the kernel (pods) are also in that same cluster. I think using a rest api for this introduces more issues that it resolves, so you're probably better off moving EG out of K8s or moving your kernels into K8s.

Have you tried deploying EG in Docker or Docker Swarm? I suspect you could then expose a port range and set the port-range option to that same range when starting EG and introduce a docker network.

What process proxy class are your kernels using - YarnClusterProcessProxy?

What aspects of "docker yarn" are you using? Are you using Spark?

[xhumanoid]

We don't have DockerSwarm, but we use docker compose for deploy EG on machine and pass all env parameters (use port mapping for EnterpriseGatewayApp.port_range and etc). Also we have small k8s (deployed orchestrations services) and big hadoop cluster (computation).

We have some patches YarnClusterProcessProxy, short description: 1) we use YARN rest api for allocate resources and schedule job for execution (not default shell execution) 2) also we have modified ApplicationMaster based on distributed-hadoop-shell stored on hdfs, so can run any bash command. so we can specify cpu/mem/container for execution. as entry point ikernel_launer.py

as result we have separate EG witch work only as orchestration and kernels running in cloud environment.

and now have task to migrate EG to k8s as service, for decrease complexity of support

[kevin-bates]

Ok - thank you for the explanation. Your setup sounds interesting. Would you be willing to attach your kernel.json and any run.sh scripts corresponding to your kernel usage? I think others could benefit from seeing how you're using YARN.

It sounds like EG deployed via docker works because docker provides the ability to specify a port range, while K8s does not. Unfortunately, I don't think there's a good solution here. Any other approach to handle the kernel's connection info response in an async manner won't work because the single thread tied to tornado is busy starting the kernel and the kernel startup logic can't complete w/o the connection info. So until the framework can start kernels async, I believe there's not much we can do.

One approach that might be worth a try is to configure EG with a very small port range (5-10), then define the K8s service to expose those 5 or 10 ports. Because kernel startup is single-threaded, you probably only ever need 1, but I think allowing a few minimizes conflicts and there may be some internal (below EG) latencies when cleaning up ports prior to the same port getting re-created, etc. The issue I believe you might run into here is that the IP produced by EG is likely the internal K8s cluster IP and not anything meaningful to entities outside K8s. I suppose you could define a config option that says "produce a public IP for the response-address" in this case.

I'm curious how EG's deployment in k8s decreases complexity of support? There isn't a whole lot of differences between EG in K8s and Docker, so I suspect the support complexities are associated with something else (auto-scaling? HA?). Is there a particular feature of K8s that you're looking to take advantage of?

Other ideas from anyone?

[xhumanoid]

Ok - thank you for the explanation. Your setup sounds interesting. Would you be willing to attach your kernel.json and any run.sh scripts corresponding to your kernel usage? I think others could benefit from seeing how you're using YARN.

it's don't have sense, because we use custom version of YarnClusterProcessProxy. Current version contain list of hack (initially was PoC), so we need clean it before opensource

I'm curious how EG's deployment in k8s decreases complexity of support?

you need virtual machine for EG and on this machine you have to run docker-compose with EG. with k8s you can just run this as a service

I suspect the support complexities are associated with something else (auto-scaling? HA?)

yes, now we discuss about how we can provide HA and scaling first step was fix persistent storage in #535

[xhumanoid]

second part: provide way to explicit specify host for backconnect #536

it's how we use our EG deployment now