Closed KoenBal closed 4 years ago
(JupyterSystemEnv) sh-4.2$ pip list
Package Version
---------------------------------- ---------
aiohttp 3.5.4
alabaster 0.7.10
anaconda-client 1.6.14
anaconda-navigator 1.8.7
anaconda-project 0.8.2
asn1crypto 0.24.0
astroid 1.6.3
astropy 3.0.2
async-timeout 3.0.1
attrs 18.1.0
autovizwidget 0.12.9
awscli 1.16.221
Babel 2.5.3
backcall 0.1.0
backports.shutil-get-terminal-size 1.0.0
bcrypt 3.1.7
beautifulsoup4 4.6.0
bitarray 0.8.1
bkcharts 0.2
blaze 0.11.3
bleach 2.1.3
bokeh 1.0.4
boto 2.48.0
boto3 1.9.211
botocore 1.12.211
Bottleneck 1.2.1
cached-property 1.5.1
certifi 2019.6.16
cffi 1.12.3
chardet 3.0.4
click 6.7
cloudpickle 0.5.3
clyent 1.2.2
colorama 0.3.9
conda 4.5.12
conda-build 3.10.5
conda-verify 2.0.0
contextlib2 0.5.5
cryptography 2.7
cycler 0.10.0
Cython 0.28.2
cytoolz 0.9.0.1
dask 2.3.0
dask-labextension 1.0.3
datashape 0.5.4
ddt 1.2.1
decorator 4.3.0
defusedxml 0.6.0
distributed 2.3.2
docker 3.7.3
docker-compose 1.24.1
docker-pycreds 0.4.0
dockerpty 0.4.1
docopt 0.6.2
docutils 0.14
entrypoints 0.2.3
environment-kernels 1.1.1
et-xmlfile 1.0.1
fastcache 1.0.2
filelock 3.0.4
Flask 1.0.2
Flask-Cors 3.0.4
gevent 1.3.0
gitdb2 2.0.5
GitPython 3.0.1
glob2 0.6
gmpy2 2.0.8
greenlet 0.4.13
h5py 2.8.0
hdijupyterutils 0.12.9
heapdict 1.0.0
html5lib 1.0.1
idna 2.7
idna-ssl 1.1.0
imageio 2.3.0
imagesize 1.0.0
ipykernel 4.8.2
ipyparallel 6.2.2
ipython 6.4.0
ipython-genutils 0.2.0
ipywidgets 7.4.0
isort 4.3.4
itsdangerous 0.24
jdcal 1.4
jedi 0.12.0
Jinja2 2.10
jmespath 0.9.4
jsonschema 2.6.0
jupyter 1.0.0
jupyter-client 5.2.3
jupyter-console 5.2.0
jupyter-core 4.4.0
jupyter-server-proxy 1.1.0
jupyterlab 0.32.1
jupyterlab-launcher 0.10.5
kiwisolver 1.0.1
lazy-object-proxy 1.3.1
llvmlite 0.23.1
locket 0.2.0
lxml 4.2.1
MarkupSafe 1.0
matplotlib 2.2.2
mccabe 0.6.1
mistune 0.8.3
mkl-fft 1.0.0
mkl-random 1.0.1
mock 3.0.5
more-itertools 4.1.0
mpmath 1.0.0
msgpack 0.6.0
msgpack-python 0.5.6
multidict 4.5.2
multipledispatch 0.5.0
navigator-updater 0.2.1
nb-conda 2.2.1
nb-conda-kernels 2.2.2
nbconvert 5.4.1
nbdime 1.1.0
nbformat 4.4.0
networkx 2.1
nltk 3.3
nose 1.3.7
notebook 5.5.0
numba 0.38.0
numexpr 2.6.5
numpy 1.14.3
numpydoc 0.8.0
odo 0.5.1
olefile 0.45.1
openpyxl 2.5.3
packaging 17.1
pandas 0.24.2
pandocfilters 1.4.2
paramiko 2.6.0
parso 0.2.0
partd 0.3.8
path.py 11.0.1
pathlib2 2.3.2
patsy 0.5.0
pep8 1.7.1
pexpect 4.5.0
pickleshare 0.7.4
pid 2.2.4
Pillow 5.4.1
pip 19.1.1
pkginfo 1.4.2
plotly 4.0.0
pluggy 0.6.0
ply 3.11
prompt-toolkit 1.0.15
protobuf 3.7.1
protobuf3-to-dict 0.1.5
psutil 5.4.5
psycopg2 2.7.5
ptyprocess 0.5.2
py 1.5.3
py4j 0.10.7
pyasn1 0.4.6
pycodestyle 2.4.0
pycosat 0.6.3
pycparser 2.19
pycrypto 2.6.1
pycurl 7.43.0.1
pyflakes 1.6.0
pygal 2.4.0
Pygments 2.2.0
pykerberos 1.2.1
pylint 1.8.4
PyNaCl 1.3.0
pyodbc 4.0.23
pyOpenSSL 19.0.0
pyparsing 2.2.0
PySocks 1.7.0
pyspark 2.3.2
pytest 3.5.1
pytest-arraydiff 0.2
pytest-astropy 0.3.0
pytest-doctestplus 0.1.3
pytest-openfiles 0.3.0
pytest-remotedata 0.2.1
python-dateutil 2.7.3
pytz 2018.4
PyWavelets 0.5.2
PyYAML 3.12
pyzmq 17.0.0
QtAwesome 0.4.4
qtconsole 4.3.1
QtPy 1.4.1
requests 2.20.0
requests-kerberos 0.12.0
retrying 1.3.3
rope 0.10.7
rsa 3.4.2
ruamel-yaml 0.15.46
s3fs 0.1.5
s3transfer 0.2.1
sagemaker 1.37.1
sagemaker-nbi-agent 1.0
sagemaker-pyspark 1.2.4
scikit-image 0.13.1
scikit-learn 0.20.3
scipy 1.1.0
seaborn 0.8.1
Send2Trash 1.5.0
setuptools 41.0.1
simpervisor 0.3
simplegeneric 0.8.1
singledispatch 3.4.0.3
six 1.12.0
smmap2 2.0.5
snowballstemmer 1.2.1
sortedcollections 0.6.1
sortedcontainers 1.5.10
sparkmagic 0.12.5
Sphinx 1.7.4
sphinxcontrib-websupport 1.0.1
spyder 3.2.8
SQLAlchemy 1.2.11
statsmodels 0.9.0
sympy 1.1.1
tables 3.4.3
tblib 1.3.2
terminado 0.8.1
testpath 0.3.1
texttable 0.9.1
toolz 0.9.0
tornado 5.0.2
traitlets 4.3.2
typing 3.6.4
typing-extensions 3.7.4
unicodecsv 0.14.1
urllib3 1.23
wcwidth 0.1.7
webencodings 0.5.1
websocket-client 0.56.0
Werkzeug 0.14.1
wheel 0.33.4
widgetsnbextension 3.4.2
wrapt 1.10.11
xlrd 1.1.0
XlsxWriter 1.0.4
xlwt 1.3.0
yarl 1.3.0
zict 0.1.3
Did you run
jupyter labextension install dask-labextension
after pip install dask_labextension
?
Yes. Should I do the inverse? Why?
Sorry I misread the output in your original post. You've done it correctly.
Do you press enter / hit the search button after pasting the address in the box?
Is there any output in your browser's debug console?
On Thu, Sep 5, 2019 at 9:53 AM Koen Bal notifications@github.com wrote:
Yes. Should I do the inverse? Why?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dask/dask-labextension/issues/87?email_source=notifications&email_token=AAKAOIV34CEIR4LXBIIPZWDQIEMM7A5CNFSM4IT52T7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD57OGGI#issuecomment-528409369, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKAOIVOMZPHZ5TSBWF5R73QIEMM7ANCNFSM4IT52T7A .
No success with enter or search button.
Seems that some errors are popping up.
I'm not familiar with how sagemaker does networking, but the 127.0.0.1:8787
address may not be what you want to connect to (unless you're running that browser on the same machine as the scheduler?).
https://github.com/dask/dask-examples/blob/master/binder/start and https://github.com/dask/dask-examples/blob/master/binder/jupyterlab-workspace.json may be worth referencing.
I wasn't aware (nor really expected) server extensions to work on sagemaker ? have you tried other server extensions and got the to successfully install ?
As @TomAugspurger said, localhost might not be proxied correctly. Instead, can you navigate directly to the sagemakerip:8787
Quickly searching for dash sagemaker dashboard
I came across this post: https://towardsdatascience.com/serverless-distributed-data-pre-processing-using-dask-amazon-ecs-and-python-part-1-a6108c728cc4
There, the author mentions some dns issues which may be relevant for you.
Thanks for the ideas.
https://github.com/jupyterlab/jupyterlab-commenting seems to be working.
Tried all the ip addresses which "sh-4.2$ hostname -I" command returned as http://ip:8787/
The setup in the post talks about AWS ECS, im trying to do some tests locally. So I can't use the ECS Service Discovery Functionality to assign the DNS name I guess.
https://myinstance.notebook.us-east-1.sagemaker.aws/proxy/8787/status gives me Getting closer :)
But the panes stay blank. How could I fix this error?
Does sagemaker block websocket connections?
On Fri, Sep 6, 2019 at 5:41 AM Koen Bal notifications@github.com wrote:
https://myinstance.notebook.us-east-1.sagemaker.aws/proxy/8787/status gives me [image: image] https://user-images.githubusercontent.com/5289035/64421976-36413300-d0a3-11e9-8757-29c4f14e92e1.png Getting closer :)
But the panes stay blank. How could I fix this error? [image: image] https://user-images.githubusercontent.com/5289035/64422091-7c969200-d0a3-11e9-926d-69cb97a8231d.png
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/dask-labextension/issues/87?email_source=notifications&email_token=AAKAOIRPCE5752KLSQFIOA3QIIXT7A5CNFSM4IT52T7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6CO7NA#issuecomment-528805812, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKAOIW6I74TX2DWYEHYFG3QIIXT7ANCNFSM4IT52T7A .
Hej,
I actually get the same error
Dask Server ErrorFailed to list clusters: might the server extension not be installed/enabled?
when using a completely fresh install of jupyterlab on linux mint.
I can connect to an existing cluster, but cannot create a new ones.
sys.executable:
/usr/bin/python3
sys.version:
3.6.8 (default, Oct 7 2019, 12:59:55)
[GCC 8.3.0]
platform.platform():
Linux-4.15.0-69-generic-x86_64-with-LinuxMint-19.2-tina
which -a jupyter:
/home/schick/.local/bin/jupyter
pip list:
aiohttp (3.6.2)
apt-clone (0.2.1)
apturl (0.5.2)
async-timeout (3.0.1)
atomicwrites (1.3.0)
attrs (19.3.0)
backcall (0.1.0)
beautifulsoup4 (4.6.0)
bleach (3.1.0)
bokeh (1.4.0)
Brlapi (0.6.6)
certifi (2018.1.18)
chardet (3.0.4)
Click (7.0)
cloudpickle (1.2.2)
command-not-found (0.3)
configobj (5.0.6)
cupshelpers (1.0)
cycler (0.10.0)
Cython (0.29.14)
dask (2.7.0)
dask-labextension (1.0.3)
decorator (4.4.1)
defer (1.0.6)
defusedxml (0.6.0)
distributed (2.7.0)
entrypoints (0.3)
flake8 (3.7.9)
fsspec (0.5.2)
h5py (2.10.0)
HeapDict (1.0.1)
httplib2 (0.9.2)
idna (2.8)
idna-ssl (1.1.0)
importlib-metadata (0.23)
ipykernel (5.1.3)
ipython (7.9.0)
ipython-genutils (0.2.0)
jedi (0.15.1)
Jinja2 (2.10.3)
json5 (0.8.5)
jsonschema (3.1.1)
jupyter-client (5.3.4)
jupyter-core (4.6.1)
jupyter-server-proxy (1.1.0)
jupyterlab (1.2.3)
jupyterlab-hdf (0.1.0)
jupyterlab-server (1.0.6)
kiwisolver (1.1.0)
locket (0.2.0)
louis (3.5.0)
macaroonbakery (1.1.3)
Mako (1.0.7)
MarkupSafe (1.1.1)
matplotlib (3.1.1)
mccabe (0.6.1)
mistune (0.8.4)
more-itertools (7.2.0)
mpmath (1.1.0)
msgpack (0.6.2)
multidict (4.5.2)
nbconvert (5.6.1)
nbformat (4.4.0)
nbresuse (0.3.2)
nemo-emblems (4.2.0)
netifaces (0.10.4)
notebook (6.0.2)
numpy (1.17.4)
onboard (1.4.1)
packaging (19.2)
PAM (0.4.2)
pandas (0.25.3)
pandocfilters (1.4.2)
parso (0.5.1)
partd (1.0.0)
pexpect (4.7.0)
pickleshare (0.7.5)
Pillow (6.2.1)
Pint (0.9)
pip (9.0.1)
pluggy (0.13.0)
prometheus-client (0.7.1)
prompt-toolkit (2.0.10)
protobuf (3.0.0)
psutil (5.6.5)
ptyprocess (0.6.0)
py (1.8.0)
pycairo (1.16.2)
pycodestyle (2.5.0)
pycrypto (2.6.1)
pycups (1.9.73)
pycurl (7.43.0.1)
pyflakes (2.1.1)
Pygments (2.4.2)
pygobject (3.26.1)
PyICU (1.9.8)
pyinotify (0.9.6)
pymacaroons (0.13.0)
PyNaCl (1.1.2)
pyparsing (2.4.5)
pyRFC3339 (1.0)
pyrsistent (0.15.5)
pytest (5.2.2)
python-apt (1.6.4)
python-dateutil (2.8.1)
python-debian (0.1.32)
python-xapp (1.6.0)
python-xlib (0.20)
pytz (2019.3)
pyxdg (0.25)
PyYAML (5.1.2)
pyzmq (18.1.0)
reportlab (3.4.0)
requests (2.18.4)
requests-unixsocket (0.1.5)
scipy (1.3.2)
screen-resolution-extra (0.0.0)
Send2Trash (1.5.0)
sessioninstaller (0.0.0)
setproctitle (1.1.10)
setuptools (41.6.0)
simpervisor (0.3)
six (1.13.0)
sortedcontainers (2.1.0)
sympy (1.4)
system-service (0.3)
systemd-python (234)
tabulate (0.8.5)
tblib (1.5.0)
terminado (0.8.3)
testpath (0.4.4)
tinycss (0.4)
toolz (0.10.0)
tornado (6.0.3)
tqdm (4.38.0)
traitlets (4.3.3)
typing-extensions (3.7.4.1)
ubuntu-drivers-common (0.0.0)
udkm1Dsim (0.1.3)
ufw (0.36)
urllib3 (1.22)
wcwidth (0.1.7)
webencodings (0.5.1)
wheel (0.33.6)
xkit (0.0.0)
yarl (1.3.0)
zict (1.0.0)
zipp (0.6.0)
@dschick Might the extension not be enabled? What do you see when entering
jupyter serverextension list
in a console?
@ian-r-rose , thanks for the quick reply.
jupyter serverextension list
gives no output at all.
here ist the list of my labextension
jupyter labextension list
JupyterLab v1.2.3
Known labextensions:
app dir: /home/schick/.local/share/jupyter/lab
@bokeh/jupyter_bokeh v1.1.0 enabled OK
@jupyter-widgets/jupyterlab-manager v1.1.0 enabled OK
@jupyterlab/hdf5 v0.2.1 enabled OK
@jupyterlab/shortcutui v0.4.0 enabled OK
@jupyterlab/toc v2.0.0-rc.0 enabled OK
@wallneradam/run_all_buttons v0.1.0 enabled OK
dask-labextension v1.0.1 enabled OK
jupyterlab-flake8 v0.3.2 enabled OK
jupyterlab-python-file v0.3.0 enabled OK
jupyterlab-spreadsheet v0.2.0 enabled OK
jupyterlab-system-monitor v0.4.1 enabled OK
jupyterlab-theme-toggle v0.4.2 enabled OK
jupyterlab-topbar-extension v0.4.0 enabled OK
jupyterlab_filetree v0.2.1 enabled OK
okay,
jupyter serverextension enable dask_labextension
solved the problem, although I am running jupyter notebook 6.0.2
@KoenBal did you managed to see the dashboard? I am in the same situation as you but I would really love to see the dashboard as it would allow me to see why the workers suddenly stopped doing anything.
nope, stopped trying to get it working.
Hi @KoenBal I've got a sense that whatever address you put in the cluster dialog box, you have to be able to hit from the machine your web browser is from (I'm assuming it's your local machine). So try putting that URL in a tab in your web browser and see if you can access it. You can't, and that's what your devtools console screenshot is also telling you.
I've a hunch your SageMaker notebook is on a private subnet, which is good from a security standpoint. You'll want to have a bastion host on a public subnet that can reach your SageMaker instance, and then you'll need to start an ssh tunnel so you can connect to it. There are lots of tutorials on the Internet on how to do this if you haven't already set one up. Once configured, you'll execute something like this on your local machine: ssh -L 8787:myinstance.notebook.us-east-1.sagemaker.aws:8787 -i mysshkey.pem ec2-user@mybastionhost.mycompany.com
. You should then be able to point a tab on your web browser to 127.0.0.1:8787/status, and if that works, plop that in your Dask labextension as well.
I'm facing the same issue, and I'm working through it right now. Best of luck.
The jupyter-server-proxy
package should proxy traffic to the status page so something like https://myinstance.notebook.us-east-1.sagemaker.aws/proxy/8787/status should work. You shouldn't need to do any port forwarding.
Given that the error is complaining about the websocket that suggests the browser was able to connect to the dashboard and load enough of it to know it needed to connect to a websocket too.
So perhaps it is as @TomAugspurger suggested and Sagemaker is blocking websockets for some reason.
A quick note if you don't mind your dashboard being exposed to the internet on some obscure URL you can always proxy it with serveo. Bit of a hacky workaround but thought I'd share anyway.
Once you have your Dask cluster running in Sagemaker you can open a terminal and run ssh -R 80:localhost:8787 serveo.net
where localhost:8787
is the host and port of your dashboard relative to the notebook server. It will probably ask you to accept some host keys but then it will print a URL where your dashboard will be available.
In this example the subdomain virgo
was randomly assigned but you can also override this if you check the serveo docs.
As this issue is related to the labextension not being enabled I've raised #130 to update the docs and will now close the issue.
We still haven't gotten to the bottom of the dashboard being broken on Sagemaker due to websocket issues, but we are tracking that in dask/dask#5432.
I have the same error:
jupyter serverextension enable dask_labextension
Enabling: dask_labextension
- Writing config: /home/swacker/.jupyter
- Validating...
Error loading server extension dask_labextension
X is dask_labextension importable?
I found that Jupyter was confused with conda environments. For some reason it was not loading Jinja2 from the conda environment, but from the main python installation (independent of conda). So, I deactivated conda completely and ran:
jupyter serverextension enable dask_labextension
Enabling: dask_labextension
- Writing config: /home/swacker/.jupyter
- Validating...
dask_labextension 6.0.0 OK
But then I still have problems:
Hello,
I'm having troubles to get my dashboard setup on a SageMaker notebook instance.
Tests I've done:
Nothing much happens doing that.
The pop up error "Failed to list clusters: might the server extension not be installed/enabled?" show up when disabling/enabling the lab extension.
Tried: https://github.com/dask/dask-labextension/issues/61#issuecomment-515786252
Any ideas?