A lightweight, multi-tenant, scalable and secure gateway that enables Jupyter Notebooks to share resources across distributed clusters such as Apache Spark, Kubernetes and others.
When trying to instantiate a docker-based kernel with the gateway I started receiving the following errors in my logs:
[D 2023-05-05 13:38:51.410 EnterpriseGatewayApp] RemoteMappingKernelManager.start_kernel: tfdocker-2.9, kernel_username: guest
[D 2023-05-05 13:38:51.411 EnterpriseGatewayApp] Instantiating kernel 'docker-TF-2.9' with process proxy: enterprise_gateway.services.processproxies.docker_swarm.DockerProcessProxy
[D 230505 13:38:51 config:21] Trying paths: ['/home/blair/.docker/config.json', '/home/blair/.dockercfg']
[D 230505 13:38:51 config:25] Found file at path: /home/blair/.docker/config.json
[D 230505 13:38:51 auth:180] Found 'auths' section
[D 230505 13:38:51 auth:140] Found entry (registry='https://index.docker.io/v1/', username='bsdz')
[D 230505 13:38:51 auth:190] Found 'credHelpers' section
[E 230505 13:38:51 web:1798] Uncaught exception POST /api/kernels (192.168.251.219)
HTTPServerRequest(protocol='http', host='192.168.251.219:8887', method='POST', uri='/api/kernels', version='HTTP/1.1', remote_ip='192.168.251.219')
Traceback (most recent call last):
File "/jl/.venv/lib/python3.10/site-packages/tornado/web.py", line 1713, in _execute
result = await result
File "/jl/.venv/lib/python3.10/site-packages/enterprise_gateway/services/kernels/handlers.py", line 102, in post
await super().post()
File "/jl/.venv/lib/python3.10/site-packages/jupyter_server/services/kernels/handlers.py", line 60, in post
kernel_id = await ensure_async(
File "/jl/.venv/lib/python3.10/site-packages/jupyter_server/utils.py", line 182, in ensure_async
result = await obj
File "/jl/.venv/lib/python3.10/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 225, in start_kernel
kernel_id = await super().start_kernel(*args, **kwargs)
File "/jl/.venv/lib/python3.10/site-packages/jupyter_server/services/kernels/kernelmanager.py", line 218, in start_kernel
kernel_id = await ensure_async(self.pinned_superclass.start_kernel(self, **kwargs))
File "/jl/.venv/lib/python3.10/site-packages/jupyter_server/utils.py", line 182, in ensure_async
result = await obj
File "/jl/.venv/lib/python3.10/site-packages/jupyter_client/multikernelmanager.py", line 483, in start_kernel
await fut
File "/jl/.venv/lib/python3.10/site-packages/jupyter_client/multikernelmanager.py", line 460, in _add_kernel_when_ready
await kernel_awaitable
File "/jl/.venv/lib/python3.10/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 501, in start_kernel
self._get_process_proxy()
File "/jl/.venv/lib/python3.10/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 757, in _get_process_proxy
self.process_proxy = process_proxy_class(
TypeError: Can't instantiate abstract class DockerProcessProxy with abstract method get_error_states
[E 230505 13:38:51 web:2271] 500 POST /api/kernels (192.168.0.2) 41.95ms
[W 2023-05-05 13:38:51.452 ServerApp] 500 POST /api/sessions?1683290331394 (::1): HTTP 500: Internal Server Error (Error attempting to connect to Gateway server url 'http://192.168.0.2:8887'. Ensure gateway url is valid and the Gateway instance is running.)
[W 2023-05-05 13:38:51.452 ServerApp] wrote error: "HTTP 500: Internal Server Error (Error attempting to connect to Gateway server url 'http://192.168.0.2:8887'. Ensure gateway url is valid and the Gateway instance is running.)"
Reproduce
One can reproduce the error by simply attempting to instantiate DockerProcessProxy in a python console, i.e.
Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import enterprise_gateway
>>> enterprise_gateway.__version__
'3.2.2'
>>> from enterprise_gateway.services.processproxies.docker_swarm import DockerProcessProxy
>>> DockerProcessProxy()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class DockerProcessProxy with abstract method get_error_states
It looks like the get_error_states method might have been missed for the class. I'm happy to raise a PR with the small change if I get some pointers on what the error states should be.
I note the sibling class DockerSwarmProcessProxy has the following initial/error states:
def get_initial_states(self) -> set:
"""Return list of states in lowercase indicating container is starting (includes running)."""
return {"preparing", "starting", "running"}
def get_error_states(self) -> set:
"""Returns the list of error states indicating container is shutting down or receiving error."""
return {"failed", "rejected", "complete", "shutdown", "orphaned", "remove"}
whereas DockerProcessProxy has these initial states (but no error states).
def get_initial_states(self) -> set:
"""Return list of states in lowercase indicating container is starting (includes running)."""
return {"created", "running"}
Description
When trying to instantiate a docker-based kernel with the gateway I started receiving the following errors in my logs:
Reproduce
One can reproduce the error by simply attempting to instantiate DockerProcessProxy in a python console, i.e.
It looks like the get_error_states method might have been missed for the class. I'm happy to raise a PR with the small change if I get some pointers on what the error states should be.
I note the sibling class DockerSwarmProcessProxy has the following initial/error states:
whereas DockerProcessProxy has these initial states (but no error states).
From the docker documentation I can see an enum of states:
So I suspect the error states should the compliment, i.e. restarting, removing, paused, exited, dead.
Expected behavior
The kernel should start using docker image.
Context
Troubleshoot Output
$PATH: /home/blair/projects/dev-settings/pyenvs/jupyter-notebook/.venv/bin /home/blair/.cargo/bin /home/blair/.poetry/bin /home/blair/scripts /home/blair/.local/bin /home/blair/bin /home/blair/.cargo/bin /home/blair/.poetry/bin /home/blair/scripts /home/blair/.local/bin /home/blair/bin /usr/local/sbin /usr/local/bin /usr/sbin /usr/bin /sbin /bin /usr/games /usr/local/games /snap/bin /snap/bin sys.path: /home/blair/projects/dev-settings/pyenvs/jupyter-notebook/.venv/bin /usr/lib/python310.zip /usr/lib/python3.10 /usr/lib/python3.10/lib-dynload /home/blair/projects/dev-settings/pyenvs/jupyter-notebook/.venv/lib/python3.10/site-packages sys.executable: /home/blair/projects/dev-settings/pyenvs/jupyter-notebook/.venv/bin/python sys.version: 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] platform.platform(): Linux-5.15.0-71-generic-x86_64-with-glibc2.35 which -a jupyter: /home/blair/projects/dev-settings/pyenvs/jupyter-notebook/.venv/bin/jupyter pip list: Package Version Editable project location --------------------------- ----------- --------------------------------------------------------------------------- aiofiles 22.1.0 aiosqlite 0.18.0 anyio 3.6.2 argon2-cffi 21.3.0 argon2-cffi-bindings 21.2.0 arrow 1.2.3 attrs 22.2.0 Babel 2.11.0 backcall 0.2.0 bcrypt 4.0.1 beautifulsoup4 4.11.1 black 22.12.0 bleach 5.0.1 bokeh 2.4.3 cachetools 5.2.0 certifi 2022.12.7 cffi 1.15.1 charset-normalizer 2.1.1 click 8.1.3 colorama 0.4.6 cryptography 39.0.0 cycler 0.11.0 debugpy 1.6.5 decorator 5.1.1 defusedxml 0.7.1 deprecation 2.1.0 docker 6.0.1 fastjsonschema 2.16.2 flake8 3.9.2 fonttools 4.38.0 fqdn 1.5.1 future 0.18.2 gitdb 4.0.10 GitPython 3.1.30 google-auth 2.15.0 idna 3.4 ipykernel 6.16.2 ipympl 0.9.3 ipython 7.34.0 ipython-genutils 0.2.0 ipywidgets 8.0.4 isoduration 20.11.0 isort 5.11.4 jedi 0.18.2 Jinja2 3.1.2 json5 0.9.11 jsonpointer 2.3 jsonschema 4.17.3 jupyter-client 6.1.12 jupyter_core 4.12.0 jupyter_enterprise_gateway 3.2.2 jupyter-events 0.6.3 jupyter_packaging 0.12.3 jupyter-server 1.23.4 jupyter_server_fileid 0.8.0 jupyter-server-mathjax 0.2.6 jupyter_server_ydoc 0.8.0 jupyter-ydoc 0.2.3 jupyterlab 3.6.2 jupyterlab-citation-data 0.1 jupyterlab-citation-manager 0.3.2 jupyterlab-code-formatter 1.5.3 jupyterlab-execute-time 2.3.1 jupyterlab-git 0.41.0 jupyterlab-mathjax3 4.3.0 /home/blair/projects/3rdparty/jupyter-renderers/packages/mathjax3-extension jupyterlab-pygments 0.2.2 jupyterlab_server 2.19.0 jupyterlab-spellchecker 0.7.3 jupyterlab-templates 0.3.2 jupyterlab-vim 0.15.1 jupyterlab-widgets 3.0.6 kiwisolver 1.4.4 kubernetes 25.3.0 Markdown 3.4.3 MarkupSafe 2.1.1 matplotlib 3.5.3 matplotlib-inline 0.1.6 mccabe 0.6.1 mistune 2.0.4 mypy-extensions 0.4.3 nbclassic 0.4.8 nbclient 0.7.2 nbconvert 7.2.7 nbdime 3.1.1 nbformat 5.7.1 nest-asyncio 1.5.6 notebook 6.5.2 notebook_shim 0.2.2 numpy 1.24.2 oauthlib 3.2.2 packaging 22.0 pandoc 2.3 pandocfilters 1.5.0 panel 0.14.4 param 1.13.0 paramiko 2.12.0 parso 0.8.3 pathspec 0.10.3 pexpect 4.8.0 pickleshare 0.7.5 Pillow 9.4.0 pip 22.0.2 platformdirs 2.6.2 plumbum 1.8.1 ply 3.11 prometheus-client 0.15.0 prompt-toolkit 3.0.36 psutil 5.9.4 ptyprocess 0.7.0 pyasn1 0.4.8 pyasn1-modules 0.2.8 pycodestyle 2.7.0 pycparser 2.21 pycryptodomex 3.16.0 pyct 0.5.0 pyflakes 2.3.1 Pygments 2.14.0 PyNaCl 1.5.0 pyparsing 3.0.9 pyrsistent 0.19.3 python-dateutil 2.8.2 python-json-logger 2.0.7 pytz 2022.7 pyviz-comms 2.2.1 PyYAML 6.0 pyzmq 24.0.1 requests 2.28.1 requests-oauthlib 1.3.1 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rope 0.17.0 rsa 4.9 Send2Trash 1.8.0 setuptools 65.6.3 six 1.16.0 smmap 5.0.0 sniffio 1.3.0 soupsieve 2.3.2.post1 terminado 0.17.1 tinycss2 1.2.1 tomli 2.0.1 tomlkit 0.11.7 tornado 6.2 tqdm 4.65.0 traitlets 5.8.0 typing_extensions 4.5.0 uri-template 1.2.0 urllib3 1.26.13 watchdog 2.2.1 wcwidth 0.2.5 webcolors 1.12 webencodings 0.5.1 websocket-client 1.4.2 wheel 0.40.0 widgetsnbextension 4.0.5 y-py 0.5.9 yarn-api-client 1.0.3 ypy-websocket 0.8.2