kevin-bates
commented
5 years ago @amangarg96 - thanks for the issue - another interesting issue from you.
Use of a reverse proxy is something we recommend, so this shouldn't be a problem. In addition, your use of the client machine as the affinity key seems fine as well.
I'm curious your EG log is indicating during this period. Perhaps there's something indicated there, or, for that matter, the kernel-specific logs maintained in YARN. Please check those for any clues. Also, have you tried a forced reconnect operation from the notebook?
Regarding the separation of duties relative to the HTTP and WS, the HTTP requests essentially invoke the various manager classes to either get, start, interrupt, etc. data relative to a specific (or all) instance(s). These requests do not (necessarily) go directly to the kernel process itself. (Of course, things like interrupt or restart will implicitly trigger interaction with the kernel process.) .
The WS request handler communicates directly with the kernel. The channel embedded in the json body of the message indicates to which ZMQ port the request should be posted. Taking a look at our gateway_client.py file might help shed some light here. Unfortunately, I'm not enough of a web developer to answer your question regarding a switch, although I'm suspicious why it wasn't done that way in the first place. A quick google regarding their differences gives good reason for why WS is used - in particular bi-directional, full-duplex and far less overhead.
One area that will make things a little difficult, should you find you need to make changes, is that the EG doesn't define any handlers - all are inherited from the Kernel Gateway and Notebook projects, so this may open a can of worms for you. That said, it would be fine to define a subclass in EG that derives from the class you need to change, assuming that a change of that magnitude is warranted and can be done in a relatively clean way.
amangarg96
I am using Jupyter Enterprise Gateway in YARN Cluster mode, with a slight modification.
An NGINX proxy is being used over multiple Jupyter Enterprise Gateway servers, which sends users to different gateway servers by hashing over the hostname of the machine. (This was done as a quick-fix for #86)
When the kernel loses connection with the Gateway server, it tries to reconnect with the previous kernel. The Notebook server logs are as follows
The Notebook server seems to be able to contact with the Kernel through REST calls, but it's not able to connect to the websockets.
On the notebook, it shows that kernel is active (in idle state), but it doesn't execute the cell.
The following step is what is missing from the reconnection attempt, right?
Does it have something to do with the NGINX proxy?
Also, what all messages are sent through Websockets? Is it possible to switch notebooks and enterprise to just HTTP (Since the REST calls are working just fine)? If so, what all functionality of notebooks would be affected? Pointers to the Documentation would also help.