Open yves-surrel opened 1 year ago
Hi @yves-surrel - it looks like you've identified a possible issue. I'm not familiar with jupyter console
but see the consoleapp.py
file in the repo - although there is no corresponding "console-script" in the set of scripts in pyproject.toml
(or has there been in the past), so I'm not sure where jupyter console
is coming from. Could you clarify how that application is installed/exposed?
I'm curious, assuming the KernelManager
can be configured in this application (as looks to be the case in consoleapp.py
), whether you can disable the port caching on the KernelManager
prior to your invocation, and whether or not that moves you forward. Perhaps try on the CLI: --KernelManager.cache_ports = False
or in the file: c.KernelManager.cache_ports = False
? Thanks.
cc: @martinRenou for possible insights
Setting --KernelManager.cache_ports = False
does not change anything, and it's the same behavior in jupyter qtconsole
, which is no surprise as cache_ports
is not Config
-able, as stated in #492. BTW, have you an idea why this 'port caching' mechanism does exist for transport = 'tcp'?
@yves-surrel - sorry about that, didn't realize that cache_ports
was not configurable and see that I had that very question on #492 that was never answered. :smile: To test this hypothesis, you'd need to modify the boolean default.
BTW, have you an idea why this 'port caching' mechanism does exist for transport = 'tcp'?
Yes. There's an inherent race condition (primarily in local kernels) between the time that the ports are determined and actually used by the kernel (process) in which another application can use any of the ports, which leads to "port in use" failures starting the kernel.
The cache_ports
functionality essentially removes any jupyter_client
-based application from the set of applications that can inject themselves into this race condition since those applications are "good citizens" by first recording their ports in a "reservation" cache.
Until the kernel determines the ports and conveys that back to the server, this issue will exist.
(Note that some remote kernels have a "launcher" or "nanny" process that creates the ports and sends that information back to the launching server, thereby decreasing the window in which the race the condition can occur from seconds to milliseconds.)
Regarding this "jupyter console" application, could you please describe how it's installed/configured? I figured it was the consoleapp.py
, but I suspect that's not the case.
The console app/entry point is provided by https://github.com/jupyter/jupyter_console
you'd need to modify the boolean default
Actually, I replaced
def _default_cache_ports(self) -> bool:
return self.transport == "tcp"
by
def _default_cache_ports(self) -> bool:
return self.transport == "tcp" and self.shell_port != 60000
in order not to break anything else and to be able to configure ports (starting with shell_port=60000
of course). Like that, it works, but it's dirty, of course. The simplest would be to have config=True
in the definition of cache_ports
, as you suggested.
Thanks for the jupyter_console
tip @blink1073.
Looking at jupyter_console
, and given that it derives from JupyterConsoleApp
, it does seem like making KernelManager.cache_ports
configurable is the way forward here.
@martinRenou or @SylvainCorlay - is there a reason cache_ports
should not be configurable (with a default value of True
)?
Changing the shell_port, iopub_port etc. in jupyter_console_config.py does not work, neither does using
jupyter console --iopub:xxxx
in the command line.After investigation, it comes from the
cache_ports
attribute ofKernelManager
:in `manager.py' and
in
async def pre_launch
inlocal_provisioner.py
.Maybe related to #492
Jupiter_client v 7.3.5