search

jupyter-server / jupyter-scheduler

Run Jupyter notebooks as jobs
https://jupyter-scheduler.readthedocs.io
BSD 3-Clause "New" or "Revised" License
197 stars 23 forks source link

When run in JupyterHub, Job creation fails with "Unexpected error occurred during creation of job" #339

Open andrii-i opened 1 year ago

andrii-i commented 1 year ago

Description

As reported by @rwuthric in #329, job creation fails with "Unexpected error occurred during creation of job." in JupyterHub instance.

Reproduce

  1. Install jupyter scheduler with pip install jupyter_scheduler in JupyterHub instance
  2. Click 'Create Notebook Job'
  3. Click 'Create'

Expected behavior

Job is created successfully or error message with clear information about what went wrong and how to solve it is displayed.

Context

"jupyter server extension list" output 
```
Config dir: /opt/jupyterhub/etc/jupyter
    jupyter_scheduler enabled
    - Validating jupyter_scheduler...
Package jupyter_scheduler took 0.1855s to import
      jupyter_scheduler 1.2.0 OK
    jupyter_server_fileid enabled
    - Validating jupyter_server_fileid...
Package jupyter_server_fileid took 0.0024s to import
      jupyter_server_fileid 0.6.0 OK
    jupyter_server_mathjax enabled
    - Validating jupyter_server_mathjax...
Package jupyter_server_mathjax took 0.0010s to import
      jupyter_server_mathjax  OK
    jupyter_server_terminals enabled
    - Validating jupyter_server_terminals...
Package jupyter_server_terminals took 0.0047s to import
      jupyter_server_terminals 0.4.4 OK
    jupyter_server_ydoc enabled
    - Validating jupyter_server_ydoc...
Package jupyter_server_ydoc took 0.0160s to import
      jupyter_server_ydoc 0.6.1 OK
    jupyterlab enabled
    - Validating jupyterlab...
Package jupyterlab took 0.0545s to import
      jupyterlab 3.6.1 OK
    jupyterlab_git enabled
    - Validating jupyterlab_git...
Package jupyterlab_git took 0.0652s to import
      jupyterlab_git 0.41.0 OK
    nbclassic enabled
    - Validating nbclassic...
Package nbclassic took 0.0000s to import
A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
      nbclassic 0.5.1 OK
    nbdime enabled
    - Validating nbdime...
Package nbdime took 0.0000s to import
      nbdime 3.1.1 OK
    notebook_shim enabled
    - Validating notebook_shim...
Package notebook_shim took 0.0000s to import
A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
      notebook_shim  OK
```
"jupyter --version" output 
```
>> jupyter --version
Selected Jupyter core packages...
IPython          : 8.7.0
ipykernel        : 6.19.2
ipywidgets       : 8.0.3
jupyter_client   : 7.4.8
jupyter_core     : 5.1.0
jupyter_server   : 2.0.1
jupyterlab       : 3.5.1
nbclient         : 0.7.2
nbconvert        : 7.2.6
nbformat         : 5.7.0
notebook         : 6.5.2
qtconsole        : not installed
traitlets        : 5.7.1
```
rwuthric commented 1 year ago

Here the output of jupyter server extension list

Config dir: /opt/jupyterhub/etc/jupyter
    jupyter_scheduler enabled
    - Validating jupyter_scheduler...
Package jupyter_scheduler took 0.1855s to import
      jupyter_scheduler 1.2.0 OK
    jupyter_server_fileid enabled
    - Validating jupyter_server_fileid...
Package jupyter_server_fileid took 0.0024s to import
      jupyter_server_fileid 0.6.0 OK
    jupyter_server_mathjax enabled
    - Validating jupyter_server_mathjax...
Package jupyter_server_mathjax took 0.0010s to import
      jupyter_server_mathjax  OK
    jupyter_server_terminals enabled
    - Validating jupyter_server_terminals...
Package jupyter_server_terminals took 0.0047s to import
      jupyter_server_terminals 0.4.4 OK
    jupyter_server_ydoc enabled
    - Validating jupyter_server_ydoc...
Package jupyter_server_ydoc took 0.0160s to import
      jupyter_server_ydoc 0.6.1 OK
    jupyterlab enabled
    - Validating jupyterlab...
Package jupyterlab took 0.0545s to import
      jupyterlab 3.6.1 OK
    jupyterlab_git enabled
    - Validating jupyterlab_git...
Package jupyterlab_git took 0.0652s to import
      jupyterlab_git 0.41.0 OK
    nbclassic enabled
    - Validating nbclassic...
Package nbclassic took 0.0000s to import
A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
      nbclassic 0.5.1 OK
    nbdime enabled
    - Validating nbdime...
Package nbdime took 0.0000s to import
      nbdime 3.1.1 OK
    notebook_shim enabled
    - Validating notebook_shim...
Package notebook_shim took 0.0000s to import
A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
      notebook_shim  OK

Here the output of jupyter labextension list

JupyterLab v3.6.1
/opt/jupyterhub/share/jupyter/labextensions
        jupyterlab_pygments v0.2.2 enabled OK (python, jupyterlab_pygments)
        spreadsheet-editor v0.6.1 enabled OK (python, jupyterlab-spreadsheet-editor)
        nbdime-jupyterlab v2.1.1 enabled OK
        @jupyter-widgets/jupyterlab-manager v5.0.5 enabled OK (python, jupyterlab_widgets)
        @jupyterlab/scheduler v1.2.0 enabled OK
        @jupyterlab/git v0.41.0 enabled OK (python, jupyterlab-git)

Other labextensions (built into JupyterLab)
   app dir: /opt/jupyterhub/share/jupyter/lab

Here output of jupyter --version

Selected Jupyter core packages...
IPython          : 8.10.0
ipykernel        : 6.21.1
ipywidgets       : 8.0.4
jupyter_client   : 8.0.2
jupyter_core     : 5.2.0
jupyter_server   : 2.2.1
jupyterlab       : 3.6.1
nbclient         : 0.7.2
nbconvert        : 7.2.9
nbformat         : 5.7.3
notebook         : 6.5.2
qtconsole        : not installed
traitlets        : 5.9.0
3coins commented 1 year ago

@rwuthric Thanks for providing that info. Are you running this in a jupyter hub instance?

rwuthric commented 1 year ago

Yes that is correct. On another setup, not run be me, but some admin people in our university, run it on JupyterHub and there the extension works flawlessly. We checked and could not find what differ between their setup and mine. The only difference I can see is that they use a proxy server. Could that be the reason?

3coins commented 1 year ago

Getting Jupyter Hub to work with the extension is on our roadmap, and it is expected it won't work without a specific Jupyter Hub backend. It will be educational to learn what kind of setup admins have to make this work; is this something you can help find out?

rwuthric commented 1 year ago

Sure. I have partial admin right son their server so I can check out things following your guidance. Plus can relay questions if needed.

rwuthric commented 1 year ago

Here outputs from the JupyterHub instance where extension works flawlessly:

>> jupyter server extension list
Config dir: /opt/jupyterhub/etc/jupyter
    jupyter_server_proxy enabled
    - Validating jupyter_server_proxy...
      jupyter_server_proxy  OK
    jupyter_scheduler enabled
    - Validating jupyter_scheduler...
      jupyter_scheduler 1.1.4 OK
    jupyter_server_mathjax enabled
    - Validating jupyter_server_mathjax...
      jupyter_server_mathjax  OK
    jupyterlab enabled
    - Validating jupyterlab...
      jupyterlab 3.5.2 OK
    jupyterlab_git enabled
    - Validating jupyterlab_git...
      jupyterlab_git 0.41.0 OK
    jupyterlab_spellchecker enabled
    - Validating jupyterlab_spellchecker...
      jupyterlab_spellchecker 0.7.2 OK
    nbclassic enabled
    - Validating nbclassic...
      nbclassic 0.4.8 OK
    nbdime enabled
    - Validating nbdime...
      nbdime 3.1.1 OK
    notebook_shim enabled
    - Validating notebook_shim...
      notebook_shim  OK
>> jupyter labextension list
JupyterLab v3.5.2
/opt/jupyterhub/share/jupyter/labextensions
        nbdime-jupyterlab v2.1.1 enabled OK
        jupyterlab_pygments v0.2.2 enabled OK (python, jupyterlab_pygments)
        @jupyter-widgets/jupyterlab-manager v5.0.3 enabled OK (python, jupyterlab_widgets)
        @ijmbarr/jupyterlab_spellchecker v0.7.2 enabled OK (python, jupyterlab-spellchecker)
        @jupyterlab/scheduler v1.1.4 enabled OK
        @jupyterlab/server-proxy v3.2.2 enabled OK
        @jupyterlab/git v0.41.0 enabled OK (python, jupyterlab-git)
>> jupyter --version
Selected Jupyter core packages...
IPython          : 8.7.0
ipykernel        : 6.17.1
ipywidgets       : 8.0.2
jupyter_client   : 7.4.7
jupyter_core     : 5.1.0
jupyter_server   : 1.23.3
jupyterlab       : 3.5.2
nbclient         : 0.7.2
nbconvert        : 7.2.5
nbformat         : 5.7.0
notebook         : 6.5.2
qtconsole        : not installed
traitlets        : 5.6.0
gigascake commented 1 year ago

I too am having this same problem. However, I also found a case where it works fine. In the administrator account where jupyterlab is installed and operated, jupyter-scheduler operates normally, and 'unexpected error~' occurs for other general users. It is managed as a separate sqlite db(scheduler.sqlite) in the ~/.local/share/jupyter folder, but I don't understand why this problem occurs.

gigascake commented 1 year ago

i solved, user session logout, delete ~/.local/share/jupyter/scheduler.sqlite, relogin.