Closed chiawchen closed 2 years ago
Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.
You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:
Hi @chiawchen - thank you for the kind words - we really appreciate hearing this! So you actually created your own process proxy? That is really great.
I'm pretty sure what's going on here and actually voiced this concern during the review of changes (sigh). Because of the aliasing approach, subclasses that override local methods will not have those methods called because the scope of the renaming does not apply within the existing methods. This prevents the signal code from getting into the RemoteKernelManager
- which forwards to the process proxy. As a result, the process group associated with the EG instance is getting killed rather than that of the process managed by the process proxy.
Given this, I'm hoping you can do two things prior to us moving forward...
jupyter_client >= 6.1.13
, if so, continue to the next step.jupyter_client == 6.1.12
where this aliasing does not occur.Looking at the elyra/enterprise-gateway:2.6.0
image, I see that it includes jupyter_client == 6.1.12
. Do you happen to modify that image in any way - perhaps upgrade dependencies for example?
What I'm still wondering though is why we haven't heard about this yet and I can only surmise that folks happen to be running with 6.1.12
.
On a side note: I'm not sure if you've taken a look at the kernel provisioners in jupyter_client 7.0 or not, but the integration is much better since provisioners (which originated from process-proxies) are first-class objects. Unfortunately, they are not compatible with process proxies so EG can't leverage them until its 4.0 release when we move to use them and the process proxies will go away. Converting a process proxy to a provisioner is fairly straightforward. If you are interested, some of the EG process proxies exist as provisioners here - since I needed to use them as proof of concept when implementing kernel provisioning. We plan to promote these to full-fledged provisioners once we have time.
Thanks @kevin-bates, I've verified that EG runtime is using jupyter-client==6.2.0
, and once I downgraded to 6.1.12
as you mentioned, it solved magically! Thanks so much for the help!
As for the reason why we are not using jupyter-client==6.1.12
it is because we are building own docker image with our business logic and simply install the jupyter-enterprise-gateway
in the dockerfile, where I can have more freedom on the customized source code and deps management since we are building own processproxy & launcher to fit into our in-house kubernetes.
@chiawchen - thank you for the quick update. jupyter_client==6.2.0
was yanked from PyPI, I believe for the same reasons I complained about 6.1.13
(but didn't realize that it too was yanked).
Since pip install "jupyter_client~=6.1"
installs 6.1.12
, and this is what our dependencies state, I'm not sure there's much we can do...
$ pip install "jupyter_client~=6.1"
Collecting jupyter_client~=6.1
Using cached jupyter_client-6.1.12-py3-none-any.whl (112 kB)
Requirement already satisfied: pyzmq>=13 in /opt/miniconda3/envs/eg-dev/lib/python3.9/site-packages (from jupyter_client~=6.1) (23.2.0)
Requirement already satisfied: tornado>=4.1 in /opt/miniconda3/envs/eg-dev/lib/python3.9/site-packages (from jupyter_client~=6.1) (6.1)
Requirement already satisfied: python-dateutil>=2.1 in /opt/miniconda3/envs/eg-dev/lib/python3.9/site-packages (from jupyter_client~=6.1) (2.8.2)
Requirement already satisfied: traitlets in /opt/miniconda3/envs/eg-dev/lib/python3.9/site-packages (from jupyter_client~=6.1) (5.3.0)
Requirement already satisfied: jupyter-core>=4.6.0 in /opt/miniconda3/envs/eg-dev/lib/python3.9/site-packages (from jupyter_client~=6.1) (4.10.0)
Requirement already satisfied: six>=1.5 in /opt/miniconda3/envs/eg-dev/lib/python3.9/site-packages (from python-dateutil>=2.1->jupyter_client~=6.1) (1.16.0)
Installing collected packages: jupyter_client
Successfully installed jupyter_client-6.1.12
I suppose your jupyter_client
is getting installed prior to EG. I suppose we could pin to literally <=6.1.12
although that would merely produce a warning but continue with the newer (yanked) version.
yeah, we have jupyter_client been installed as a higher version before installing EG, I just pin the version down to 6.1.12
for now, thanks again for the detailed explaination!
Thanks @chiawchen. I'm curious how you went about installing the higher version of jupyter_client
. Given that pip install jupyter_client
will install the latest release of version 7.3.4
- which EG cannot support, are you explicitly installing a version of 6 greater than 6.1.12? How did you land on jupyter_client==6.2.0
?
we have several requirments.txt
which contain tons of library used by other code, so it's hard to tell which is source of dependency coming from, and those installation all come before I install jupyter-enterprise-gateway
Since we've specified our dependency on jupyter_client
and given yanked versions are not implicitly installable, I'm not certain there's much we can do here other than ensure folks are using the supported version of jupyter_client
- closing.
Description
Thanks for building this project, it's really helpful and extendable to fit in with our in-house kubernetes infrastructure, we simply provide the customized launcher script and processproxy. Everything works out of the box (we are able to launch a pod running on in-house kubernetes and able to send command to the remote kernel running on other pod in kubernetes) except one thing which is
interrupting
/deleting
the kernel.We observed a crash on enterprise gateway server side, when there's either a
interrupt
ordelete
request come from client side and it only happens on kubernetes based remote kernel, the kernel that launched locally within enterprise gateway (i.e. ipykernel_launcher) can be interrupted or deleted successfully without crashing.Reproduce
I'm not sure if this is reproducible in native kubernetes, but I'll try to provide as much details as possible
and the corresponding ENV
interrupt
ordelete
sessionkill()
properlyExpected behavior
The expected behavior is the kernel been deleted successfully and the jupyter enterprise gateway shouldn't crash if one of the remote kernel been requested to interrupt or delete. Or one step back, server side should provide the traceback / debug logging about the interrupt / delete happened, so server side can do further debugging, perhaps the issue is on customized processproxy but we cannot see the log as for this case.
Context
Command Line Output
Browser Output