Failed to list clusters: might the server extension not be installed/enabled?

[KoenBal]

Hello,

I'm having troubles to get my dashboard setup on a SageMaker notebook instance.

Tests I've done:

- Checked serverextension list:
(JupyterSystemEnv) sh-4.2$ jupyter serverextension list
config dir: /home/ec2-user/.jupyter
    jupyterlab_git  enabled 
    - Validating...
Error loading server extension jupyterlab_git
      X is jupyterlab_git importable?
config dir: /home/ec2-user/anaconda3/etc/jupyter
    dask_labextension  enabled 
    - Validating...
      dask_labextension 1.0.3 OK
    ipyparallel.nbextension  enabled 
    - Validating...
      ipyparallel.nbextension  OK
    jupyter_server_proxy  enabled 
    - Validating...
      jupyter_server_proxy  OK
    jupyterlab  enabled 
    - Validating...
      jupyterlab 0.32.1 OK
    nbdime  enabled 
    - Validating...
      nbdime 1.1.0 OK
    nb_conda  enabled 
    - Validating...
      nb_conda 2.2.1 OK

• copy-pasted the url below in the extension search box

Nothing much happens doing that.

The pop up error "Failed to list clusters: might the server extension not be installed/enabled?" show up when disabling/enabling the lab extension.

Tried: https://github.com/dask/dask-labextension/issues/61#issuecomment-515786252

Any ideas?

[KoenBal]

(JupyterSystemEnv) sh-4.2$ pip list
Package                            Version  
---------------------------------- ---------
aiohttp                            3.5.4    
alabaster                          0.7.10   
anaconda-client                    1.6.14   
anaconda-navigator                 1.8.7    
anaconda-project                   0.8.2    
asn1crypto                         0.24.0   
astroid                            1.6.3    
astropy                            3.0.2    
async-timeout                      3.0.1    
attrs                              18.1.0   
autovizwidget                      0.12.9   
awscli                             1.16.221 
Babel                              2.5.3    
backcall                           0.1.0    
backports.shutil-get-terminal-size 1.0.0    
bcrypt                             3.1.7    
beautifulsoup4                     4.6.0    
bitarray                           0.8.1    
bkcharts                           0.2      
blaze                              0.11.3   
bleach                             2.1.3    
bokeh                              1.0.4    
boto                               2.48.0   
boto3                              1.9.211  
botocore                           1.12.211 
Bottleneck                         1.2.1    
cached-property                    1.5.1    
certifi                            2019.6.16
cffi                               1.12.3   
chardet                            3.0.4    
click                              6.7      
cloudpickle                        0.5.3    
clyent                             1.2.2    
colorama                           0.3.9    
conda                              4.5.12   
conda-build                        3.10.5   
conda-verify                       2.0.0    
contextlib2                        0.5.5    
cryptography                       2.7      
cycler                             0.10.0   
Cython                             0.28.2   
cytoolz                            0.9.0.1  
dask                               2.3.0    
dask-labextension                  1.0.3    
datashape                          0.5.4    
ddt                                1.2.1    
decorator                          4.3.0    
defusedxml                         0.6.0    
distributed                        2.3.2    
docker                             3.7.3    
docker-compose                     1.24.1   
docker-pycreds                     0.4.0    
dockerpty                          0.4.1    
docopt                             0.6.2    
docutils                           0.14     
entrypoints                        0.2.3    
environment-kernels                1.1.1    
et-xmlfile                         1.0.1    
fastcache                          1.0.2    
filelock                           3.0.4    
Flask                              1.0.2    
Flask-Cors                         3.0.4    
gevent                             1.3.0    
gitdb2                             2.0.5    
GitPython                          3.0.1    
glob2                              0.6      
gmpy2                              2.0.8    
greenlet                           0.4.13   
h5py                               2.8.0    
hdijupyterutils                    0.12.9   
heapdict                           1.0.0    
html5lib                           1.0.1    
idna                               2.7      
idna-ssl                           1.1.0    
imageio                            2.3.0    
imagesize                          1.0.0    
ipykernel                          4.8.2    
ipyparallel                        6.2.2    
ipython                            6.4.0    
ipython-genutils                   0.2.0    
ipywidgets                         7.4.0    
isort                              4.3.4    
itsdangerous                       0.24     
jdcal                              1.4      
jedi                               0.12.0   
Jinja2                             2.10     
jmespath                           0.9.4    
jsonschema                         2.6.0    
jupyter                            1.0.0    
jupyter-client                     5.2.3    
jupyter-console                    5.2.0    
jupyter-core                       4.4.0    
jupyter-server-proxy               1.1.0    
jupyterlab                         0.32.1   
jupyterlab-launcher                0.10.5   
kiwisolver                         1.0.1    
lazy-object-proxy                  1.3.1    
llvmlite                           0.23.1   
locket                             0.2.0    
lxml                               4.2.1    
MarkupSafe                         1.0      
matplotlib                         2.2.2    
mccabe                             0.6.1    
mistune                            0.8.3    
mkl-fft                            1.0.0    
mkl-random                         1.0.1    
mock                               3.0.5    
more-itertools                     4.1.0    
mpmath                             1.0.0    
msgpack                            0.6.0    
msgpack-python                     0.5.6    
multidict                          4.5.2    
multipledispatch                   0.5.0    
navigator-updater                  0.2.1    
nb-conda                           2.2.1    
nb-conda-kernels                   2.2.2    
nbconvert                          5.4.1    
nbdime                             1.1.0    
nbformat                           4.4.0    
networkx                           2.1      
nltk                               3.3      
nose                               1.3.7    
notebook                           5.5.0    
numba                              0.38.0   
numexpr                            2.6.5    
numpy                              1.14.3   
numpydoc                           0.8.0    
odo                                0.5.1    
olefile                            0.45.1   
openpyxl                           2.5.3    
packaging                          17.1     
pandas                             0.24.2   
pandocfilters                      1.4.2    
paramiko                           2.6.0    
parso                              0.2.0    
partd                              0.3.8    
path.py                            11.0.1   
pathlib2                           2.3.2    
patsy                              0.5.0    
pep8                               1.7.1    
pexpect                            4.5.0    
pickleshare                        0.7.4    
pid                                2.2.4    
Pillow                             5.4.1    
pip                                19.1.1   
pkginfo                            1.4.2    
plotly                             4.0.0    
pluggy                             0.6.0    
ply                                3.11     
prompt-toolkit                     1.0.15   
protobuf                           3.7.1    
protobuf3-to-dict                  0.1.5    
psutil                             5.4.5    
psycopg2                           2.7.5    
ptyprocess                         0.5.2    
py                                 1.5.3    
py4j                               0.10.7   
pyasn1                             0.4.6    
pycodestyle                        2.4.0    
pycosat                            0.6.3    
pycparser                          2.19     
pycrypto                           2.6.1    
pycurl                             7.43.0.1 
pyflakes                           1.6.0    
pygal                              2.4.0    
Pygments                           2.2.0    
pykerberos                         1.2.1    
pylint                             1.8.4    
PyNaCl                             1.3.0    
pyodbc                             4.0.23   
pyOpenSSL                          19.0.0   
pyparsing                          2.2.0    
PySocks                            1.7.0    
pyspark                            2.3.2    
pytest                             3.5.1    
pytest-arraydiff                   0.2      
pytest-astropy                     0.3.0    
pytest-doctestplus                 0.1.3    
pytest-openfiles                   0.3.0    
pytest-remotedata                  0.2.1    
python-dateutil                    2.7.3    
pytz                               2018.4   
PyWavelets                         0.5.2    
PyYAML                             3.12     
pyzmq                              17.0.0   
QtAwesome                          0.4.4    
qtconsole                          4.3.1    
QtPy                               1.4.1    
requests                           2.20.0   
requests-kerberos                  0.12.0   
retrying                           1.3.3    
rope                               0.10.7   
rsa                                3.4.2    
ruamel-yaml                        0.15.46  
s3fs                               0.1.5    
s3transfer                         0.2.1    
sagemaker                          1.37.1   
sagemaker-nbi-agent                1.0      
sagemaker-pyspark                  1.2.4    
scikit-image                       0.13.1   
scikit-learn                       0.20.3   
scipy                              1.1.0    
seaborn                            0.8.1    
Send2Trash                         1.5.0    
setuptools                         41.0.1   
simpervisor                        0.3      
simplegeneric                      0.8.1    
singledispatch                     3.4.0.3  
six                                1.12.0   
smmap2                             2.0.5    
snowballstemmer                    1.2.1    
sortedcollections                  0.6.1    
sortedcontainers                   1.5.10   
sparkmagic                         0.12.5   
Sphinx                             1.7.4    
sphinxcontrib-websupport           1.0.1    
spyder                             3.2.8    
SQLAlchemy                         1.2.11   
statsmodels                        0.9.0    
sympy                              1.1.1    
tables                             3.4.3    
tblib                              1.3.2    
terminado                          0.8.1    
testpath                           0.3.1    
texttable                          0.9.1    
toolz                              0.9.0    
tornado                            5.0.2    
traitlets                          4.3.2    
typing                             3.6.4    
typing-extensions                  3.7.4    
unicodecsv                         0.14.1   
urllib3                            1.23     
wcwidth                            0.1.7    
webencodings                       0.5.1    
websocket-client                   0.56.0   
Werkzeug                           0.14.1   
wheel                              0.33.4   
widgetsnbextension                 3.4.2    
wrapt                              1.10.11  
xlrd                               1.1.0    
XlsxWriter                         1.0.4    
xlwt                               1.3.0    
yarl                               1.3.0    
zict                               0.1.3

[TomAugspurger]

Did you run

jupyter labextension install dask-labextension

after pip install dask_labextension?

[KoenBal]

Yes. Should I do the inverse? Why?

[TomAugspurger]

Sorry I misread the output in your original post. You've done it correctly.

Do you press enter / hit the search button after pasting the address in the box?

Is there any output in your browser's debug console?

On Thu, Sep 5, 2019 at 9:53 AM Koen Bal notifications@github.com wrote:

Yes. Should I do the inverse? Why?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dask/dask-labextension/issues/87?email_source=notifications&email_token=AAKAOIV34CEIR4LXBIIPZWDQIEMM7A5CNFSM4IT52T7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD57OGGI#issuecomment-528409369, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKAOIVOMZPHZ5TSBWF5R73QIEMM7ANCNFSM4IT52T7A .

[KoenBal]

No success with enter or search button.

Seems that some errors are popping up.

[TomAugspurger]

I'm not familiar with how sagemaker does networking, but the 127.0.0.1:8787 address may not be what you want to connect to (unless you're running that browser on the same machine as the scheduler?).

https://github.com/dask/dask-examples/blob/master/binder/start and https://github.com/dask/dask-examples/blob/master/binder/jupyterlab-workspace.json may be worth referencing.

[quasiben]

I wasn't aware (nor really expected) server extensions to work on sagemaker ? have you tried other server extensions and got the to successfully install ?

As @TomAugspurger said, localhost might not be proxied correctly. Instead, can you navigate directly to the sagemakerip:8787

Quickly searching for dash sagemaker dashboard I came across this post: https://towardsdatascience.com/serverless-distributed-data-pre-processing-using-dask-amazon-ecs-and-python-part-1-a6108c728cc4

There, the author mentions some dns issues which may be relevant for you.

[KoenBal]

Thanks for the ideas.

https://github.com/jupyterlab/jupyterlab-commenting seems to be working.

Tried all the ip addresses which "sh-4.2$ hostname -I" command returned as http://ip:8787/

The setup in the post talks about AWS ECS, im trying to do some tests locally. So I can't use the ECS Service Discovery Functionality to assign the DNS name I guess.

[KoenBal]

https://myinstance.notebook.us-east-1.sagemaker.aws/proxy/8787/status gives me Getting closer :)

But the panes stay blank. How could I fix this error?

[TomAugspurger]

Does sagemaker block websocket connections?

On Fri, Sep 6, 2019 at 5:41 AM Koen Bal notifications@github.com wrote:

https://myinstance.notebook.us-east-1.sagemaker.aws/proxy/8787/status gives me [image: image] https://user-images.githubusercontent.com/5289035/64421976-36413300-d0a3-11e9-8757-29c4f14e92e1.png Getting closer :)

But the panes stay blank. How could I fix this error? [image: image] https://user-images.githubusercontent.com/5289035/64422091-7c969200-d0a3-11e9-926d-69cb97a8231d.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/dask-labextension/issues/87?email_source=notifications&email_token=AAKAOIRPCE5752KLSQFIOA3QIIXT7A5CNFSM4IT52T7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6CO7NA#issuecomment-528805812, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKAOIW6I74TX2DWYEHYFG3QIIXT7ANCNFSM4IT52T7A .

[dschick]

Hej,

I actually get the same error Dask Server ErrorFailed to list clusters: might the server extension not be installed/enabled? when using a completely fresh install of jupyterlab on linux mint. I can connect to an existing cluster, but cannot create a new ones.

sys.executable:
    /usr/bin/python3

sys.version:
    3.6.8 (default, Oct  7 2019, 12:59:55) 
    [GCC 8.3.0]

platform.platform():
    Linux-4.15.0-69-generic-x86_64-with-LinuxMint-19.2-tina

which -a jupyter:
    /home/schick/.local/bin/jupyter

pip list:
    aiohttp (3.6.2)
    apt-clone (0.2.1)
    apturl (0.5.2)
    async-timeout (3.0.1)
    atomicwrites (1.3.0)
    attrs (19.3.0)
    backcall (0.1.0)
    beautifulsoup4 (4.6.0)
    bleach (3.1.0)
    bokeh (1.4.0)
    Brlapi (0.6.6)
    certifi (2018.1.18)
    chardet (3.0.4)
    Click (7.0)
    cloudpickle (1.2.2)
    command-not-found (0.3)
    configobj (5.0.6)
    cupshelpers (1.0)
    cycler (0.10.0)
    Cython (0.29.14)
    dask (2.7.0)
    dask-labextension (1.0.3)
    decorator (4.4.1)
    defer (1.0.6)
    defusedxml (0.6.0)
    distributed (2.7.0)
    entrypoints (0.3)
    flake8 (3.7.9)
    fsspec (0.5.2)
    h5py (2.10.0)
    HeapDict (1.0.1)
    httplib2 (0.9.2)
    idna (2.8)
    idna-ssl (1.1.0)
    importlib-metadata (0.23)
    ipykernel (5.1.3)
    ipython (7.9.0)
    ipython-genutils (0.2.0)
    jedi (0.15.1)
    Jinja2 (2.10.3)
    json5 (0.8.5)
    jsonschema (3.1.1)
    jupyter-client (5.3.4)
    jupyter-core (4.6.1)
    jupyter-server-proxy (1.1.0)
    jupyterlab (1.2.3)
    jupyterlab-hdf (0.1.0)
    jupyterlab-server (1.0.6)
    kiwisolver (1.1.0)
    locket (0.2.0)
    louis (3.5.0)
    macaroonbakery (1.1.3)
    Mako (1.0.7)
    MarkupSafe (1.1.1)
    matplotlib (3.1.1)
    mccabe (0.6.1)
    mistune (0.8.4)
    more-itertools (7.2.0)
    mpmath (1.1.0)
    msgpack (0.6.2)
    multidict (4.5.2)
    nbconvert (5.6.1)
    nbformat (4.4.0)
    nbresuse (0.3.2)
    nemo-emblems (4.2.0)
    netifaces (0.10.4)
    notebook (6.0.2)
    numpy (1.17.4)
    onboard (1.4.1)
    packaging (19.2)
    PAM (0.4.2)
    pandas (0.25.3)
    pandocfilters (1.4.2)
    parso (0.5.1)
    partd (1.0.0)
    pexpect (4.7.0)
    pickleshare (0.7.5)
    Pillow (6.2.1)
    Pint (0.9)
    pip (9.0.1)
    pluggy (0.13.0)
    prometheus-client (0.7.1)
    prompt-toolkit (2.0.10)
    protobuf (3.0.0)
    psutil (5.6.5)
    ptyprocess (0.6.0)
    py (1.8.0)
    pycairo (1.16.2)
    pycodestyle (2.5.0)
    pycrypto (2.6.1)
    pycups (1.9.73)
    pycurl (7.43.0.1)
    pyflakes (2.1.1)
    Pygments (2.4.2)
    pygobject (3.26.1)
    PyICU (1.9.8)
    pyinotify (0.9.6)
    pymacaroons (0.13.0)
    PyNaCl (1.1.2)
    pyparsing (2.4.5)
    pyRFC3339 (1.0)
    pyrsistent (0.15.5)
    pytest (5.2.2)
    python-apt (1.6.4)
    python-dateutil (2.8.1)
    python-debian (0.1.32)
    python-xapp (1.6.0)
    python-xlib (0.20)
    pytz (2019.3)
    pyxdg (0.25)
    PyYAML (5.1.2)
    pyzmq (18.1.0)
    reportlab (3.4.0)
    requests (2.18.4)
    requests-unixsocket (0.1.5)
    scipy (1.3.2)
    screen-resolution-extra (0.0.0)
    Send2Trash (1.5.0)
    sessioninstaller (0.0.0)
    setproctitle (1.1.10)
    setuptools (41.6.0)
    simpervisor (0.3)
    six (1.13.0)
    sortedcontainers (2.1.0)
    sympy (1.4)
    system-service (0.3)
    systemd-python (234)
    tabulate (0.8.5)
    tblib (1.5.0)
    terminado (0.8.3)
    testpath (0.4.4)
    tinycss (0.4)
    toolz (0.10.0)
    tornado (6.0.3)
    tqdm (4.38.0)
    traitlets (4.3.3)
    typing-extensions (3.7.4.1)
    ubuntu-drivers-common (0.0.0)
    udkm1Dsim (0.1.3)
    ufw (0.36)
    urllib3 (1.22)
    wcwidth (0.1.7)
    webencodings (0.5.1)
    wheel (0.33.6)
    xkit (0.0.0)
    yarl (1.3.0)
    zict (1.0.0)
    zipp (0.6.0)

[ian-r-rose]

@dschick Might the extension not be enabled? What do you see when entering

jupyter serverextension list

in a console?

[dschick]

@ian-r-rose , thanks for the quick reply. jupyter serverextension list gives no output at all. here ist the list of my labextension

jupyter labextension list
JupyterLab v1.2.3
Known labextensions:
   app dir: /home/schick/.local/share/jupyter/lab
        @bokeh/jupyter_bokeh v1.1.0  enabled  OK
        @jupyter-widgets/jupyterlab-manager v1.1.0  enabled  OK
        @jupyterlab/hdf5 v0.2.1  enabled  OK
        @jupyterlab/shortcutui v0.4.0  enabled  OK
        @jupyterlab/toc v2.0.0-rc.0  enabled  OK
        @wallneradam/run_all_buttons v0.1.0  enabled  OK
        dask-labextension v1.0.1  enabled  OK
        jupyterlab-flake8 v0.3.2  enabled  OK
        jupyterlab-python-file v0.3.0  enabled  OK
        jupyterlab-spreadsheet v0.2.0  enabled  OK
        jupyterlab-system-monitor v0.4.1  enabled  OK
        jupyterlab-theme-toggle v0.4.2  enabled  OK
        jupyterlab-topbar-extension v0.4.0  enabled  OK
        jupyterlab_filetree v0.2.1  enabled  OK

[dschick]

okay,

jupyter serverextension enable dask_labextension

solved the problem, although I am running jupyter notebook 6.0.2

[EnyMan]

@KoenBal did you managed to see the dashboard? I am in the same situation as you but I would really love to see the dashboard as it would allow me to see why the workers suddenly stopped doing anything.

[KoenBal]

nope, stopped trying to get it working.

[jackwellsxyz]

Hi @KoenBal I've got a sense that whatever address you put in the cluster dialog box, you have to be able to hit from the machine your web browser is from (I'm assuming it's your local machine). So try putting that URL in a tab in your web browser and see if you can access it. You can't, and that's what your devtools console screenshot is also telling you.

I've a hunch your SageMaker notebook is on a private subnet, which is good from a security standpoint. You'll want to have a bastion host on a public subnet that can reach your SageMaker instance, and then you'll need to start an ssh tunnel so you can connect to it. There are lots of tutorials on the Internet on how to do this if you haven't already set one up. Once configured, you'll execute something like this on your local machine: ssh -L 8787:myinstance.notebook.us-east-1.sagemaker.aws:8787 -i mysshkey.pem ec2-user@mybastionhost.mycompany.com. You should then be able to point a tab on your web browser to 127.0.0.1:8787/status, and if that works, plop that in your Dask labextension as well.

I'm facing the same issue, and I'm working through it right now. Best of luck.

[jacobtomlinson]

The jupyter-server-proxy package should proxy traffic to the status page so something like https://myinstance.notebook.us-east-1.sagemaker.aws/proxy/8787/status should work. You shouldn't need to do any port forwarding.

Given that the error is complaining about the websocket that suggests the browser was able to connect to the dashboard and load enough of it to know it needed to connect to a websocket too.

So perhaps it is as @TomAugspurger suggested and Sagemaker is blocking websockets for some reason.

[jacobtomlinson]

A quick note if you don't mind your dashboard being exposed to the internet on some obscure URL you can always proxy it with serveo. Bit of a hacky workaround but thought I'd share anyway.

Once you have your Dask cluster running in Sagemaker you can open a terminal and run ssh -R 80:localhost:8787 serveo.net where localhost:8787 is the host and port of your dashboard relative to the notebook server. It will probably ask you to accept some host keys but then it will print a URL where your dashboard will be available.

In this example the subdomain virgo was randomly assigned but you can also override this if you check the serveo docs.

[jacobtomlinson]

As this issue is related to the labextension not being enabled I've raised #130 to update the docs and will now close the issue.

We still haven't gotten to the bottom of the dashboard being broken on Sagemaker due to websocket issues, but we are tracking that in dask/dask#5432.

[sorenwacker]

I have the same error:

jupyter serverextension enable dask_labextension
Enabling: dask_labextension
- Writing config: /home/swacker/.jupyter
    - Validating...
Error loading server extension dask_labextension
      X is dask_labextension importable?

I found that Jupyter was confused with conda environments. For some reason it was not loading Jinja2 from the conda environment, but from the main python installation (independent of conda). So, I deactivated conda completely and ran:

jupyter serverextension enable dask_labextension
Enabling: dask_labextension
- Writing config: /home/swacker/.jupyter
    - Validating...
      dask_labextension 6.0.0 OK

But then I still have problems: