search

berkeley-dsep-infra / jupyterhub-k8s

[Deprecated] Data 8's deployment of JupyterHub on Kubernetes
Apache License 2.0
34 stars 17 forks source link

Generalization cleanup #1 #146

Closed yuvipanda closed 7 years ago

yuvipanda commented 7 years ago

Cleanup and optimization of the generalizaiton we did a while ago.

Mostly aimed at making it easier for other institutions to use our chart. Also helps make our lives easier :)

yuvipanda commented 7 years ago

Testing this on datahub-dev yielded strange results. Particularly, --base-url does not seem to take effect at all, leading to JS and CSS loading to fail inside the notebook.

yuvipanda commented 7 years ago

Looking at the effective running cmdlines for a working and non-working pods:

working pod

jovyan@jupyter-yuvipanda-10:~$ strings /proc/7/cmdline 
/opt/conda/bin/python
/opt/conda/bin/jupyterhub-singleuser
--port=8888
--ip=0.0.0.0
--user=yuvipanda
--cookie-name=jupyter-hub-token-yuvipanda
--base-url=/user/yuvipanda
--hub-prefix=/hub/
--hub-api-url=http://10.15.251.67:8081/hub/api

non-working pod

jovyan@jupyter-rylo-3:~$ strings /proc/1/cmdline 
/opt/conda/bin/python
/opt/conda/bin/jupyterhub-singleuser
--user="rylo"
--cookie-name="jupyter-hub-token-rylo"
--base-url="/user/rylo"
--hub-host=""
--hub-prefix="/hub/"
--hub-api-url="http://10.15.251.67:8081/hub/api"
--ip="0.0.0.0"
--port=8888

The presence of quotes in the second one but not first bothers me, and makes me feel suspicious.

yuvipanda commented 7 years ago

Update: The quotes were a complete red herring lol

yuvipanda commented 7 years ago

In a working notebook, I get:

    <script src="/user/yuvipanda/static/tree/js/main.min.js" type="text/javascript" charset="utf-8"></script>

In a notebook that doesn't,

<script src="/static/tree/js/main.min.js" type="text/javascript" charset="utf-8"></script>
yuvipanda commented 7 years ago

Both of them have:

data-base-url="/user/yuvipanda/"
data-notebook-path=""
data-terminals-available="True"

which leaves me very confused.

yuvipanda commented 7 years ago

Looks like the places in notebook using static_url are getting it wrong, while the places using baseUrl are getting it right. Quite strange!

yuvipanda commented 7 years ago

Also it seemed to work for me locally on minikube, altho it was a different image.

minrk commented 7 years ago

@yuvipanda can you pip list in the good/bad containers? And show the env as well? My guess is that one update or another caused an argument to be ignored, but it's not obvious which would cause this. It could be a mismatch between the versions of jupyterhub on the host and jupyterhub providing the jupyterhub-singleuser executable.

yuvipanda commented 7 years ago

@minrk they're both the same container tho - same hash, etc.

Working container:

env:

CONDA_DIR=/opt/conda
EMAIL=tonyyang@berkeley.edu
GIT_AUTHOR_NAME=tonyyang
GIT_COMMITTER_NAME=tonyyang
HOME=/home/jovyan
HOSTNAME=jupyter-tonyyang-51
HUB_PORT_8081_TCP_ADDR=10.15.251.67
HUB_PORT_8081_TCP_PORT=8081
HUB_PORT_8081_TCP_PROTO=tcp
HUB_PORT_8081_TCP=tcp://10.15.251.67:8081
HUB_PORT=tcp://10.15.251.67:8081
HUB_SERVICE_HOST=10.15.251.67
HUB_SERVICE_PORT=8081
JPY_API_TOKEN=<a-token>
JPY_BASE_URL=/user/tonyyang
JPY_COOKIE_NAME=jupyter-hub-token-tonyyang
JPY_HUB_API_URL=http://10.15.251.67:8081/hub/api
JPY_HUB_PREFIX=/hub/
JPY_USER=tonyyang
JUPYTERHUB_API_TOKEN=<a-token>
KUBERNETES_PORT_443_TCP_ADDR=10.15.240.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP=tcp://10.15.240.1:443
KUBERNETES_PORT=tcp://10.15.240.1:443
KUBERNETES_SERVICE_HOST=10.15.240.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_ALL=en_US.UTF-8
MEM_GUARANTEE=1073741824
MEM_LIMIT=1073741824
MPLCONFIGDIR=/var/cache/matplotlib
NB_UID=1000
NB_USER=jovyan
NLTK_DATA=/opt/conda/nltk
PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PROXY_API_PORT_8001_TCP_ADDR=10.15.255.114
PROXY_API_PORT_8001_TCP_PORT=8001
PROXY_API_PORT_8001_TCP_PROTO=tcp
PROXY_API_PORT_8001_TCP=tcp://10.15.255.114:8001
PROXY_API_PORT=tcp://10.15.255.114:8001
PROXY_API_SERVICE_HOST=10.15.255.114
PROXY_API_SERVICE_PORT=8001
PROXY_PUBLIC_PORT_80_TCP_ADDR=10.15.244.199
PROXY_PUBLIC_PORT_80_TCP_PORT=80
PROXY_PUBLIC_PORT_80_TCP_PROTO=tcp
PROXY_PUBLIC_PORT_80_TCP=tcp://10.15.244.199:80
PROXY_PUBLIC_PORT=tcp://10.15.244.199:80
PROXY_PUBLIC_SERVICE_HOST=10.15.244.199
PROXY_PUBLIC_SERVICE_PORT=80
PWD=/home/jovyan
SHELL=/bin/bash
SHLVL=1
STATSD_PORT_8125_UDP_ADDR=10.15.247.17
STATSD_PORT_8125_UDP_PORT=8125
STATSD_PORT_8125_UDP_PROTO=udp
STATSD_PORT_8125_UDP=udp://10.15.247.17:8125
STATSD_PORT=udp://10.15.247.17:8125
STATSD_SERVICE_HOST=10.15.247.17
STATSD_SERVICE_PORT=8125
_=/usr/bin/env

pip list

alabaster (0.7.10)
alembic (0.8.10)
Babel (2.3.4)
bleach (1.5.0)
bokeh (0.12.4)
boto (2.45.0)
bz2file (0.98)
click (6.7)
click-plugins (1.0.3)
cligj (0.4.0)
cloudpickle (0.1.1)
conda (4.2.12)
connectortools (0.1)
coverage (3.7.1)
coveralls (0.5)
cycler (0.10.0)
Cython (0.23.5)
datascience (0.9.5)
decorator (4.0.11)
descartes (1.1.0)
dill (0.2.6)
docopt (0.6.2)
docutils (0.13.1)
entrypoints (0.2.2)
fastcache (1.0.2)
Fiona (1.7.4)
folium (0.1.5)
GDAL (2.0.0)
gensim (0.12.4)
geopandas (0.2.1)
geopy (1.11.0)
gitdb2 (2.0.0)
GitPython (2.1.3)
h5py (2.6.0)
html5lib (0.999)
imagesize (0.7.1)
ipykernel (4.5.2)
ipython (5.2.2)
ipython-genutils (0.1.0)
ipywidgets (6.0.0)
Jinja2 (2.9.5)
jsonschema (2.5.1)
jupyter (1.0.0)
jupyter-client (5.0.0)
jupyter-console (5.1.0)
jupyter-core (4.3.0)
jupyterhub (0.7.2)
llvmlite (0.16.0)
lxml (3.7.2)
Mako (1.0.6)
MarkupSafe (0.23)
marshmallow (2.13.3)
matplotlib (2.0.0)
mistune (0.7.3)
mne (0.13.1)
mplleaflet (0.0.5)
mpmath (0.19)
munch (2.1.0)
nbconvert (5.1.1)
nbformat (4.2.0)
nbgdrive (0.8.6)
nbgrader (0.4.0rc1)
nbinteract (0.0.22)
nbresuse (0.1.0)
nbserverproxy (0.2.0)
networkx (1.11)
nibabel (2.1.0)
nltk (3.2.1)
notebook (4.4.1)
numba (0.31.0)
numexpr (2.6.1)
numpy (1.10.4)
okpy (1.9.5)
olefile (0.44)
pamela (0.3.0)
pandas (0.19.2)
pandocfilters (1.4.1)
patsy (0.4.1)
pexpect (4.2.1)
pickleshare (0.7.3)
Pillow (4.4.0)
pip (8.1.2)
prompt-toolkit (1.0.13)
psutil (5.2.0)
psycopg2 (2.6.1)
ptyprocess (0.5.1)
py (1.4.32)
pycortex (0.1.1)
pycosat (0.6.1)
pycrypto (2.6.1)
pydicom (0.9.9)
Pygments (2.2.0)
pygrowup (0.8.2)
pypandoc (1.3.3)
pyparsing (2.2.0)
pyproj (1.9.5.1)
PySAL (1.11.1)
pytest (3.0.6)
python-dateutil (2.6.0)
python-editor (1.0.3)
pytz (2016.10)
PyYAML (3.12)
pyzmq (16.0.2)
qtconsole (4.2.1)
requests (2.11.1)
Rtree (0.8.3)
ruamel-yaml (-VERSION)
scikit-image (0.12.3)
scikit-learn (0.18)
scipy (0.18.1)
seaborn (0.7.1)
setuptools (27.2.0)
Shapely (1.5.13)
simplegeneric (0.8.1)
six (1.10.0)
smart-open (1.3.5)
smmap2 (2.0.1)
snowballstemmer (1.2.1)
Sphinx (1.5.3)
SQLAlchemy (1.1.5)
statsmodels (0.8.0)
sympy (1.0)
terminado (0.6)
testpath (0.3)
toolz (0.8.2)
tornado (4.4.2)
tqdm (4.11.2)
traitlets (4.3.1)
wcwidth (0.1.7)
webargs (1.5.3)
wheel (0.29.0)
widgetsnbextension (2.0.0)

Getting output from a non-working pod shortly...

yuvipanda commented 7 years ago

For a non-working pod:

env list

jovyan@jupyter-yuvipanda-10:~$ env | sort                                                                  
CONDA_DIR=/opt/conda
EMAIL=yuvipanda@berkeley.edu
GIT_AUTHOR_NAME=yuvipanda
GIT_COMMITTER_NAME=yuvipanda
HOME=/home/jovyan
HOSTNAME=jupyter-yuvipanda-10
HUB_PORT_8081_TCP_ADDR=10.15.251.67
HUB_PORT_8081_TCP_PORT=8081
HUB_PORT_8081_TCP_PROTO=tcp
HUB_PORT_8081_TCP=tcp://10.15.251.67:8081
HUB_PORT=tcp://10.15.251.67:8081
HUB_SERVICE_HOST=10.15.251.67
HUB_SERVICE_PORT=8081
JPY_API_TOKEN=<a-token>
JUPYTERHUB_API_TOKEN=<a-token>
KUBERNETES_PORT_443_TCP_ADDR=10.15.240.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP=tcp://10.15.240.1:443
KUBERNETES_PORT=tcp://10.15.240.1:443
KUBERNETES_SERVICE_HOST=10.15.240.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_ALL=en_US.UTF-8
MEM_GUARANTEE=1073741824
MEM_LIMIT=1073741824
MPLCONFIGDIR=/var/cache/matplotlib
NB_UID=1000
NB_USER=jovyan
NLTK_DATA=/opt/conda/nltk
PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PROXY_API_PORT_8001_TCP_ADDR=10.15.255.114
PROXY_API_PORT_8001_TCP_PORT=8001
PROXY_API_PORT_8001_TCP_PROTO=tcp
PROXY_API_PORT_8001_TCP=tcp://10.15.255.114:8001
PROXY_API_PORT=tcp://10.15.255.114:8001
PROXY_API_SERVICE_HOST=10.15.255.114
PROXY_API_SERVICE_PORT=8001
PROXY_PUBLIC_PORT_80_TCP_ADDR=10.15.244.199
PROXY_PUBLIC_PORT_80_TCP_PORT=80
PROXY_PUBLIC_PORT_80_TCP_PROTO=tcp
PROXY_PUBLIC_PORT_80_TCP=tcp://10.15.244.199:80
PROXY_PUBLIC_PORT=tcp://10.15.244.199:80
PROXY_PUBLIC_SERVICE_HOST=10.15.244.199
PROXY_PUBLIC_SERVICE_PORT=80
PWD=/home/jovyan
SHELL=/bin/bash
SHLVL=1
STATSD_PORT_8125_UDP_ADDR=10.15.247.17
STATSD_PORT_8125_UDP_PORT=8125
STATSD_PORT_8125_UDP_PROTO=udp
STATSD_PORT_8125_UDP=udp://10.15.247.17:8125
STATSD_PORT=udp://10.15.247.17:8125
STATSD_SERVICE_HOST=10.15.247.17
STATSD_SERVICE_PORT=8125
_=/usr/bin/env
jovyan@jupyter-yuvipanda-10:~$

pip list:

jovyan@jupyter-yuvipanda-10:~$ pip list
alabaster (0.7.10)
alembic (0.8.10)
Babel (2.3.4)
bleach (1.5.0)
bokeh (0.12.4)
boto (2.45.0)
bz2file (0.98)
click (6.7)
click-plugins (1.0.3)
cligj (0.4.0)
cloudpickle (0.1.1)
conda (4.2.12)
connectortools (0.1)
coverage (3.7.1)
coveralls (0.5)
cycler (0.10.0)
Cython (0.23.5)
datascience (0.9.5)
decorator (4.0.11)
descartes (1.1.0)
dill (0.2.6)
docopt (0.6.2)
docutils (0.13.1)
entrypoints (0.2.2)
fastcache (1.0.2)
Fiona (1.7.4)
folium (0.1.5)
GDAL (2.0.0)
gensim (0.12.4)
geopandas (0.2.1)
geopy (1.11.0)
gitdb2 (2.0.0)
GitPython (2.1.3)
h5py (2.6.0)
html5lib (0.999)
imagesize (0.7.1)
ipykernel (4.5.2)
ipython (5.2.2)
ipython-genutils (0.1.0)
ipywidgets (6.0.0)
Jinja2 (2.9.5)
jsonschema (2.5.1)
jupyter (1.0.0)
jupyter-client (5.0.0)
jupyter-console (5.1.0)
jupyter-core (4.3.0)
jupyterhub (0.7.2)
llvmlite (0.16.0)
lxml (3.7.2)
Mako (1.0.6)
MarkupSafe (0.23)
marshmallow (2.13.3)
matplotlib (2.0.0)
mistune (0.7.3)
mne (0.13.1)
mplleaflet (0.0.5)
mpmath (0.19)
munch (2.1.0)
nbconvert (5.1.1)
nbformat (4.2.0)
nbgdrive (0.8.6)
nbgrader (0.4.0rc1)
nbinteract (0.0.22)
nbresuse (0.1.0)
nbserverproxy (0.2.0)
networkx (1.11)
nibabel (2.1.0)
nltk (3.2.1)
notebook (4.4.1)
numba (0.31.0)
numexpr (2.6.1)
numpy (1.10.4)
okpy (1.9.5)
olefile (0.44)
pamela (0.3.0)
pandas (0.19.2)
pandocfilters (1.4.1)
patsy (0.4.1)
pexpect (4.2.1)
pickleshare (0.7.3)
Pillow (4.4.0)
pip (8.1.2)
prompt-toolkit (1.0.13)
psutil (5.2.0)
psycopg2 (2.6.1)
ptyprocess (0.5.1)
py (1.4.32)
pycortex (0.1.1)
pycosat (0.6.1)
pycrypto (2.6.1)
pydicom (0.9.9)
Pygments (2.2.0)
pygrowup (0.8.2)
pypandoc (1.3.3)
pyparsing (2.2.0)
pyproj (1.9.5.1)
PySAL (1.11.1)
pytest (3.0.6)
python-dateutil (2.6.0)
python-editor (1.0.3)
pytz (2016.10)
PyYAML (3.12)
pyzmq (16.0.2)
qtconsole (4.2.1)
requests (2.11.1)
Rtree (0.8.3)
ruamel-yaml (-VERSION)
scikit-image (0.12.3)
scikit-learn (0.18)
scipy (0.18.1)
seaborn (0.7.1)
setuptools (27.2.0)
Shapely (1.5.13)
simplegeneric (0.8.1)
six (1.10.0)
smart-open (1.3.5)
smmap2 (2.0.1)
snowballstemmer (1.2.1)
Sphinx (1.5.3)
SQLAlchemy (1.1.5)
statsmodels (0.8.0)
sympy (1.0)
terminado (0.6)
testpath (0.3)
toolz (0.8.2)
tornado (4.4.2)
tqdm (4.11.2)
traitlets (4.3.1)
wcwidth (0.1.7)
webargs (1.5.3)
wheel (0.29.0)
widgetsnbextension (2.0.0)
yuvipanda commented 7 years ago

@minrk was awesome and helped me debug this. https://github.com/data-8/nbinteract/issues/34 seems to be the ultimate cause!

I'm going to temporarily fix this by having KubeSpawner set those env variables still, but will need to fix nbinteract soon.