search

mechmotum / jupyterhub-deploy-teaching

Reference deployment of JupyterHub and nbgrader on a single server
http://jupyterhub-deploy-teaching.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
0 stars 2 forks source link

Notes on trying to deploy myself #20

Open moorepants opened 5 years ago

moorepants commented 5 years ago

Got this error first time:

TASK [nginx : start the nginx service] ******************************************************************
fatal: [bicycle]: FAILED! => {"changed": false, "msg": "Unable to start service nginx: Job for nginx.service failed because the control process exited with error code. See \"systemctl status nginx.service\" and \"journalctl -xe\" for details.\n"}

RUNNING HANDLER [nginx : reload nginx] ******************************************************************
    to retry, use: --limit @/home/moorepants/Websites/bicycle.ucdavis.edu/jupyterhub-deploy-teaching/deploy.retry

PLAY RECAP **********************************************************************************************
bicycle                    : ok=30   changed=16   unreachable=0    failed=1

On the server:

root@bicycle:~# systemctl status nginx.service
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2018-10-15 15:57:09 PDT; 1min 35s ago
  Process: 32440 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status

Oct 15 15:57:09 bicycle systemd[1]: Starting A high performance web server and a reverse proxy server...
Oct 15 15:57:09 bicycle nginx[32440]: nginx: [emerg] PEM_read_bio_X509_AUX("/etc/nginx/ssl/ssl.crt") fail
Oct 15 15:57:09 bicycle nginx[32440]: nginx: configuration file /etc/nginx/nginx.conf test failed
Oct 15 15:57:09 bicycle systemd[1]: nginx.service: Control process exited, code=exited status=1
Oct 15 15:57:09 bicycle systemd[1]: Failed to start A high performance web server and a reverse proxy ser
Oct 15 15:57:09 bicycle systemd[1]: nginx.service: Unit entered failed state.
Oct 15 15:57:09 bicycle systemd[1]: nginx.service: Failed with result 'exit-code'.
moorepants commented 5 years ago

I got an email saying download the certificate. I clicked and get a bicycle_ucdavis_edu.cer file. It has four hashes that are separated by:

-----BEGIN CERTIFICATE-----
hash
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
hash
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
hash
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
hash
-----END CERTIFICATE-----

In the drive secrets/bicycle2/ fodler there is a bicycle.csr and a bicycle.key file. The csr file has a single hash:

-----BEGIN CERTIFICATE REQUEST-----
hash
-----END CERTIFICATE REQUEST-----

I tried both of these files as ssl.crt with ansible and they failed. I am not sure what order the four hashes should go in. Kenny mentioned that he had to change the order before.

moorepants commented 5 years ago

After reading Kenny's email carefully you must move the fourth hash to the top leaving the 1,2,3 hashes in the same order.

moorepants commented 5 years ago

Ansible ran and it didnt' seem jupyterhub was running. I got an nginx bad gateway at the url. I ran the supervisor command and it said it started jupyterhub but still got bad gateway.

moorepants commented 5 years ago

I run:

root@bicycle:~# supervisorctl start jupyterhub
jupyterhub: started

but don't see a jupyterhub process in htop.

moorepants commented 5 years ago

@ixjlyons Any idea what I'm missing?

moorepants commented 5 years ago

I'm getting:

Found database schema version 3ec6993fe20c != 896818069c98. Backup your database and run `jupyterhub upgrade-db` to upgrade to the latest schema.

but when I run the upgrade-db command nothing seems to happen.

moorepants commented 5 years ago

I added --upgrade-db to the startup script and it ran but I"m getting these errors:

root@bicycle:~# /srv/jupyterhub/start-jupyterhub.sh 
XDG_SESSION_ID=1
SHELL=/bin/bash
TERM=xterm-256color
SSH_CLIENT=
SSH_TTY=/dev/pts/0
USER=root
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
PATH=/opt/conda/bin:/opt/conda/bin:/opt/conda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
MAIL=/var/mail/root
PWD=/root
LANG=en_US.UTF-8
HOME=/root
SHLVL=2
LOGNAME=
SSH_CONNECTION=
LESSOPEN=| /usr/bin/lesspipe %s
XDG_RUNTIME_DIR=/run/user/0
LESSCLOSE=/usr/bin/lesspipe %s %s
_=/usr/bin/env
[W 2018-10-15 16:28:53.294 JupyterHub app:441] JupyterHub.proxy_auth_token is deprecated in JupyterHub 0.8, use ConfigurableHTTPProxy.auth_token
[I 2018-10-15 16:28:53.294 JupyterHub app:1673] Using Authenticator: jupyterhub.auth.PAMAuthenticator-0.9.4
[I 2018-10-15 16:28:53.294 JupyterHub app:1673] Using Spawner: systemdspawner.systemdspawner.SystemdSpawner
[I 2018-10-15 16:28:53.298 JupyterHub app:1016] Loading cookie_secret from /srv/jupyterhub/cookie_secret
[I 2018-10-15 16:28:53.327 JupyterHub dbutil:125] Upgrading sqlite:////srv/jupyterhub/jupyterhub.sqlite
[I 2018-10-15 16:28:53.328 JupyterHub dbutil:105] Backing up /srv/jupyterhub/jupyterhub.sqlite => /srv/jupyterhub/jupyterhub.sqlite.2018-10-15-162853
[I 2018-10-15 16:28:54.027 alembic.runtime.migration migration:117] Context impl SQLiteImpl.
[I 2018-10-15 16:28:54.028 alembic.runtime.migration migration:122] Will assume non-transactional DDL.
[I 2018-10-15 16:28:54.034 alembic.runtime.migration migration:327] Running upgrade 3ec6993fe20c -> 1cebaf56856c, Add session_id to auth tokens
[I 2018-10-15 16:28:54.049 alembic.runtime.migration migration:327] Running upgrade 1cebaf56856c -> 56cc5a70207e, token tracking
[W 2018-10-15 16:28:54.063 alembic 56cc5a70207e_token_tracking:31] sqlite cannot use ALTER TABLE to create foreign keys. Upgrade will be incomplete.
[I 2018-10-15 16:28:54.065 alembic.runtime.migration migration:327] Running upgrade 56cc5a70207e -> 99a28a4418e1, user.created and spawner.started
[I 2018-10-15 16:28:54.078 alembic.runtime.migration migration:327] Running upgrade 99a28a4418e1 -> d68c98b66cd4, client-description
[I 2018-10-15 16:28:54.083 alembic.runtime.migration migration:327] Running upgrade d68c98b66cd4 -> 896818069c98, Add APIToken.expires_at
  <jupyterhub.objects.Server object at 0x7f6adca8c080>
  <jupyterhub.objects.Server object at 0x7f6adca93320>
[W 2018-10-15 16:28:54.709 JupyterHub app:1519] Deleting OAuth client service-shared1
[W 2018-10-15 16:28:54.716 JupyterHub app:1519] Deleting OAuth client service-shared1-notebook
[W 2018-10-15 16:28:54.721 JupyterHub app:1519] Deleting OAuth client service-eng122f16
[W 2018-10-15 16:28:54.733 JupyterHub app:1519] Deleting OAuth client service-eng122f17
[W 2018-10-15 16:28:54.746 JupyterHub app:1519] Deleting OAuth client user-sjangra
[W 2018-10-15 16:28:54.758 JupyterHub app:1519] Deleting OAuth client user-mmllanes
[W 2018-10-15 16:28:54.766 JupyterHub app:1519] Deleting OAuth client user-mbuonarati
[W 2018-10-15 16:28:54.779 JupyterHub app:1519] Deleting OAuth client user-nabachus
[W 2018-10-15 16:28:54.793 JupyterHub app:1519] Deleting OAuth client user-nodkim
[W 2018-10-15 16:28:54.811 JupyterHub app:1519] Deleting OAuth client user-hsgothard
[W 2018-10-15 16:28:54.829 JupyterHub app:1519] Deleting OAuth client user-tipeev
[W 2018-10-15 16:28:54.855 JupyterHub app:1519] Deleting OAuth client user-nama
[W 2018-10-15 16:28:54.884 JupyterHub app:1519] Deleting OAuth client user-ggchen
[W 2018-10-15 16:28:54.979 JupyterHub app:1519] Deleting OAuth client user-kfngissah
[W 2018-10-15 16:28:55.056 JupyterHub app:1519] Deleting OAuth client user-vikli
[W 2018-10-15 16:28:55.166 JupyterHub app:1519] Deleting OAuth client user-jwday
[W 2018-10-15 16:28:55.321 JupyterHub app:1519] Deleting OAuth client user-sadunn
[W 2018-10-15 16:28:55.425 JupyterHub app:1519] Deleting OAuth client user-dplucas
[W 2018-10-15 16:28:55.597 JupyterHub app:1519] Deleting OAuth client user-tmadden
[W 2018-10-15 16:28:55.656 JupyterHub app:1519] Deleting OAuth client user-spgentry
[W 2018-10-15 16:28:55.690 JupyterHub app:1519] Deleting OAuth client user-jhkchoi
[W 2018-10-15 16:28:55.697 JupyterHub app:1519] Deleting OAuth client user-hilbert
[W 2018-10-15 16:28:55.704 JupyterHub app:1519] Deleting OAuth client user-mhubbard
[W 2018-10-15 16:28:55.710 JupyterHub app:1519] Deleting OAuth client user-jrector
[W 2018-10-15 16:28:55.724 JupyterHub app:1519] Deleting OAuth client user-jsmullin
[W 2018-10-15 16:28:55.730 JupyterHub app:1519] Deleting OAuth client user-domishchuk
[W 2018-10-15 16:28:55.736 JupyterHub app:1519] Deleting OAuth client user-mpmodera
[W 2018-10-15 16:28:55.744 JupyterHub app:1519] Deleting OAuth client user-irdonisgon
[W 2018-10-15 16:28:55.750 JupyterHub app:1519] Deleting OAuth client user-srdas
[W 2018-10-15 16:28:55.757 JupyterHub app:1519] Deleting OAuth client user-mrhill
[W 2018-10-15 16:28:55.763 JupyterHub app:1519] Deleting OAuth client user-mjculshawmaurer
[W 2018-10-15 16:28:55.771 JupyterHub app:1519] Deleting OAuth client user-nawais
[W 2018-10-15 16:28:55.784 JupyterHub app:1519] Deleting OAuth client user-essin
[W 2018-10-15 16:28:55.916 JupyterHub app:1519] Deleting OAuth client user-swkresie
[W 2018-10-15 16:28:55.926 JupyterHub app:1519] Deleting OAuth client user-plreynolds
[W 2018-10-15 16:28:55.994 JupyterHub app:1519] Deleting OAuth client user-bwmargolis
[W 2018-10-15 16:28:56.004 JupyterHub app:1519] Deleting OAuth client user-crchighizola
[W 2018-10-15 16:28:56.023 JupyterHub app:1519] Deleting OAuth client user-zbhays
[W 2018-10-15 16:28:56.126 JupyterHub app:1519] Deleting OAuth client user-sppatankar
[W 2018-10-15 16:28:56.137 JupyterHub app:1519] Deleting OAuth client user-msdassonville
[W 2018-10-15 16:28:56.265 JupyterHub app:1519] Deleting OAuth client user-mishi
[W 2018-10-15 16:28:56.331 JupyterHub app:1519] Deleting OAuth client user-abmckay
[W 2018-10-15 16:28:56.452 JupyterHub app:1519] Deleting OAuth client user-krlyons
[W 2018-10-15 16:28:56.478 JupyterHub app:1519] Deleting OAuth client user-ndmaharaj
[W 2018-10-15 16:28:56.586 JupyterHub app:1519] Deleting OAuth client user-qlai
[W 2018-10-15 16:28:56.752 JupyterHub app:1519] Deleting OAuth client user-haiwu
[W 2018-10-15 16:28:56.814 JupyterHub app:1519] Deleting OAuth client user-bmgatten
[W 2018-10-15 16:28:56.937 JupyterHub app:1519] Deleting OAuth client user-adaliu
[W 2018-10-15 16:28:57.074 JupyterHub app:1519] Deleting OAuth client user-vermorales
[W 2018-10-15 16:28:57.193 JupyterHub app:1519] Deleting OAuth client user-hdagnew
[W 2018-10-15 16:28:57.199 JupyterHub app:1519] Deleting OAuth client user-nrandhawa
[W 2018-10-15 16:28:57.205 JupyterHub app:1519] Deleting OAuth client user-chwchuang
[W 2018-10-15 16:28:57.216 JupyterHub app:1519] Deleting OAuth client user-jmdill
[W 2018-10-15 16:28:57.339 JupyterHub app:1519] Deleting OAuth client user-jrmoses
[W 2018-10-15 16:28:57.420 JupyterHub app:1519] Deleting OAuth client user-jasonmoore
[W 2018-10-15 16:28:57.431 JupyterHub app:1519] Deleting OAuth client user-lfilipozzi
[I 2018-10-15 16:28:57.586 JupyterHub app:1855] Hub API listening on http://127.0.0.1:8081/hub/
[W 2018-10-15 16:28:57.588 JupyterHub proxy:565] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2018-10-15 16:28:57.588 JupyterHub proxy:567] Starting proxy @ http://127.0.0.1:8000/
16:28:57.829 - info: [ConfigProxy] Proxying http://127.0.0.1:8000 to (no default)
16:28:57.834 - info: [ConfigProxy] Proxy API at http://127.0.0.1:8001/api/routes
[I 2018-10-15 16:28:57.852 JupyterHub app:1876] Starting managed service eng122 at http://127.0.0.1:5000
[I 2018-10-15 16:28:57.854 JupyterHub service:302] Starting service 'eng122': ['jupyterhub-singleuser', '--group=eng122', '--allow-root']
[I 2018-10-15 16:28:57.860 JupyterHub service:114] Spawning jupyterhub-singleuser --group=eng122 --allow-root
Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 3, in <module>
    from jupyterhub.singleuser import main
  File "/opt/conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 34, in <module>
    from notebook.notebookapp import (
  File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 43, in <module>
    ioloop.install()
  File "/opt/conda/lib/python3.6/site-packages/zmq/eventloop/ioloop.py", line 210, in install
    assert (not ioloop.IOLoop.initialized()) or \
AttributeError: type object 'IOLoop' has no attribute 'initialized'
[E 2018-10-15 16:28:58.938 JupyterHub app:1894] Service eng122 exited with status 1
[I 2018-10-15 16:28:58.939 JupyterHub app:1876] Starting managed service mae223 at http://127.0.0.1:5001
[I 2018-10-15 16:28:58.939 JupyterHub service:302] Starting service 'mae223': ['jupyterhub-singleuser', '--group=mae223', '--allow-root']
[I 2018-10-15 16:28:58.945 JupyterHub service:114] Spawning jupyterhub-singleuser --group=mae223 --allow-root
Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 3, in <module>
    from jupyterhub.singleuser import main
  File "/opt/conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 34, in <module>
    from notebook.notebookapp import (
  File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 43, in <module>
    ioloop.install()
  File "/opt/conda/lib/python3.6/site-packages/zmq/eventloop/ioloop.py", line 210, in install
    assert (not ioloop.IOLoop.initialized()) or \
AttributeError: type object 'IOLoop' has no attribute 'initialized'
[E 2018-10-15 16:28:59.933 JupyterHub app:1894] Service mae223 exited with status 1
[I 2018-10-15 16:28:59.933 JupyterHub app:1876] Starting managed service cull_idle_servers
[I 2018-10-15 16:28:59.933 JupyterHub service:302] Starting service 'cull_idle_servers': ['/opt/conda/bin/python3', '/srv/jupyterhub/cull_idle_servers.py', '--cull-every=600', '--timeout=7200']
[I 2018-10-15 16:28:59.938 JupyterHub service:114] Spawning /opt/conda/bin/python3 /srv/jupyterhub/cull_idle_servers.py --cull-every=600 --timeout=7200
[I 2018-10-15 16:28:59.956 JupyterHub proxy:301] Checking routes
[I 2018-10-15 16:28:59.957 JupyterHub proxy:370] Adding default route for Hub: / => http://127.0.0.1:8081
16:28:59.957 - info: [ConfigProxy] 200 GET /api/routes 
[W 2018-10-15 16:28:59.961 JupyterHub proxy:349] Adding missing route for eng122 (<jupyterhub.objects.Server object at 0x7f6adc107d30>)
[W 2018-10-15 16:28:59.964 JupyterHub proxy:349] Adding missing route for mae223 (<jupyterhub.objects.Server object at 0x7f6adc107d30>)
[I 2018-10-15 16:28:59.968 JupyterHub proxy:224] Adding service eng122 to proxy /services/eng122/ => http://127.0.0.1:5000
[I 2018-10-15 16:28:59.974 JupyterHub proxy:224] Adding service mae223 to proxy /services/mae223/ => http://127.0.0.1:5001
16:28:59.980 - info: [ConfigProxy] Adding route /services/mae223 -> http://127.0.0.1:5001
16:28:59.982 - info: [ConfigProxy] 201 POST /api/routes/services/mae223 
16:28:59.983 - info: [ConfigProxy] Adding route /services/eng122 -> http://127.0.0.1:5000
16:28:59.983 - info: [ConfigProxy] 201 POST /api/routes/services/eng122 
16:28:59.983 - info: [ConfigProxy] Adding route / -> http://127.0.0.1:8081
16:28:59.984 - info: [ConfigProxy] 201 POST /api/routes/ 
[I 2018-10-15 16:28:59.984 JupyterHub app:1912] JupyterHub is now running at http://127.0.0.1:8000/
[I 2018-10-15 16:29:00.348 JupyterHub log:158] 200 GET /hub/api/users (cull_idle_servers@127.0.0.1) 160.93ms
[E 2018-10-15 16:29:27.866 JupyterHub service:342] Service eng122 exited with status 1
[I 2018-10-15 16:29:27.866 JupyterHub service:302] Starting service 'eng122': ['jupyterhub-singleuser', '--group=eng122', '--allow-root']
[I 2018-10-15 16:29:27.872 JupyterHub service:114] Spawning jupyterhub-singleuser --group=eng122 --allow-root
Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 3, in <module>
    from jupyterhub.singleuser import main
  File "/opt/conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 34, in <module>
    from notebook.notebookapp import (
  File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 43, in <module>
    ioloop.install()
  File "/opt/conda/lib/python3.6/site-packages/zmq/eventloop/ioloop.py", line 210, in install
    assert (not ioloop.IOLoop.initialized()) or \
AttributeError: type object 'IOLoop' has no attribute 'initialized'
[E 2018-10-15 16:29:28.951 JupyterHub service:342] Service mae223 exited with status 1
[I 2018-10-15 16:29:28.952 JupyterHub service:302] Starting service 'mae223': ['jupyterhub-singleuser', '--group=mae223', '--allow-root']
[I 2018-10-15 16:29:28.958 JupyterHub service:114] Spawning jupyterhub-singleuser --group=mae223 --allow-root
Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 3, in <module>
    from jupyterhub.singleuser import main
  File "/opt/conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 34, in <module>
    from notebook.notebookapp import (
  File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 43, in <module>
    ioloop.install()
  File "/opt/conda/lib/python3.6/site-packages/zmq/eventloop/ioloop.py", line 210, in install
    assert (not ioloop.IOLoop.initialized()) or \
AttributeError: type object 'IOLoop' has no attribute 'initialized'
[E 2018-10-15 16:29:57.877 JupyterHub service:342] Service eng122 exited with status 1
[I 2018-10-15 16:29:57.877 JupyterHub service:302] Starting service 'eng122': ['jupyterhub-singleuser', '--group=eng122', '--allow-root']
[I 2018-10-15 16:29:57.883 JupyterHub service:114] Spawning jupyterhub-singleuser --group=eng122 --allow-root
Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 3, in <module>
    from jupyterhub.singleuser import main
  File "/opt/conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 34, in <module>
    from notebook.notebookapp import (
  File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 43, in <module>
    ioloop.install()
  File "/opt/conda/lib/python3.6/site-packages/zmq/eventloop/ioloop.py", line 210, in install
    assert (not ioloop.IOLoop.initialized()) or \
AttributeError: type object 'IOLoop' has no attribute 'initialized'
[E 2018-10-15 16:29:58.964 JupyterHub service:342] Service mae223 exited with status 1
[I 2018-10-15 16:29:58.964 JupyterHub service:302] Starting service 'mae223': ['jupyterhub-singleuser', '--group=mae223', '--allow-root']
[I 2018-10-15 16:29:58.971 JupyterHub service:114] Spawning jupyterhub-singleuser --group=mae223 --allow-root
Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 3, in <module>
    from jupyterhub.singleuser import main
  File "/opt/conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 34, in <module>
    from notebook.notebookapp import (
  File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 43, in <module>
    ioloop.install()
  File "/opt/conda/lib/python3.6/site-packages/zmq/eventloop/ioloop.py", line 210, in install
    assert (not ioloop.IOLoop.initialized()) or \
AttributeError: type object 'IOLoop' has no attribute 'initialized'
[W 2018-10-15 16:30:00.967 JupyterHub app:1417] Cannot connect to managed service eng122 at http://127.0.0.1:5000
[W 2018-10-15 16:30:01.872 JupyterHub app:1417] Cannot connect to managed service mae223 at http://127.0.0.1:5001
moorepants commented 5 years ago

I don't seem to be getting the oauth we had. selection_298

moorepants commented 5 years ago

Got it! Had to set "use_oath" to true. But it might have deleted all the users...

moorepants commented 5 years ago

Now can login with ucd login, but getting "failed to spawn" when trying to start a server.

API request failed (500): Spawner failed to start [status=3]. The logs for jasonmoore may contain details.
moorepants commented 5 years ago

Some errors in the logs:

selection_299

moorepants commented 5 years ago

pyzmq was a 16 and should have been at 17, conda updat pyzmq solved it! https://github.com/jupyter/notebook/issues/3595

It now seems to work again!

moorepants commented 5 years ago

It doesn't seem like ansible installs the latest versions of all the conda packages.

moorepants commented 5 years ago

Now I fucked it up doing a conda update --all:

root@bicycle:~# conda list
# packages in environment at /opt/conda:
#
# Name                    Version                   Build  Channel
alembic                   0.9.5                    py36_0    conda-forge
alembic                   0.9.5                     <pip>
ampl-mp                   3.1.0                         0    conda-forge
asn1crypto                0.22.0                   py36_0    conda-forge
async_generator           1.10                       py_0    conda-forge
backports                 1.0                      py36_1    conda-forge
backports.functools_lru_cache 1.4                      py36_1    conda-forge
blas                      1.1                    openblas    conda-forge
bleach                    2.0.0                    py36_0    conda-forge
bleach                    2.1.1                     <pip>
bottleneck                1.2.1                    py36_1    conda-forge
ca-certificates           2018.8.24            ha4d7672_0    conda-forge
certifi                   2018.8.24             py36_1001    conda-forge
cffi                      1.10.0                   py36_0    conda-forge
chardet                   3.0.4                    py36_0    conda-forge
cma                       2.3.1                     <pip>
conda                     4.5.11                py36_1000    conda-forge
conda-env                 2.6.0                         1    conda-forge
configurable-http-proxy   3.0.0                         0    conda-forge
cryptography              2.0.3                    py36_0    conda-forge
curl                      7.52.1                        0    conda-forge
cycler                    0.10.0                   py36_0    conda-forge
cyipopt                   0.1.7                    py36_2    conda-forge
cython                    0.27.3                   py36_0    conda-forge
dbus                      1.10.22                       0    conda-forge
decorator                 4.1.2                    py36_0    conda-forge
entrypoints               0.2.3                    py36_1    conda-forge
expat                     2.2.1                         0    conda-forge
fastcache                 1.0.2                    py36_0    conda-forge
fontconfig                2.12.1                        4    conda-forge
freetype                  2.7                           1    conda-forge
future                    0.16.0                   py36_0    conda-forge
gettext                   0.19.7                        1    conda-forge
glib                      2.53.5                        0    conda-forge
gmp                       6.1.2                         0    conda-forge
gmpy2                     2.0.8                    py36_1    conda-forge
gst-plugins-base          1.8.0                         0    conda-forge
gstreamer                 1.8.0                         1    conda-forge
html5lib                  1.0b10                    <pip>
html5lib                  0.999999999              py36_0    conda-forge
icu                       58.1                          1    conda-forge
idna                      2.6                      py36_1    conda-forge
ipopt                     3.12.8          blas_openblas_0  [blas_openblas]  conda-forge
ipykernel                 4.6.1                    py36_0    conda-forge
ipyparallel               6.0.2                    py36_0    conda-forge
ipython                   6.2.1                    py36_0    conda-forge
ipython_genutils          0.2.0                    py36_0    conda-forge
ipywidgets                7.0.1                      py_2    conda-forge
jedi                      0.10.2                   py36_0    conda-forge
jedi                      0.11.0                    <pip>
jinja2                    2.9.6                    py36_0    conda-forge
jpeg                      9b                            1    conda-forge
jsonschema                2.6.0                    py36_0    conda-forge
jupyter                   1.0.0                    py36_0    conda-forge
jupyter_client            5.1.0                    py36_0    conda-forge
jupyter_console           5.1.0                    py36_0    conda-forge
jupyter_contrib_core      0.3.3                    py36_0    conda-forge
jupyter_contrib_nbextensions 0.3.1                    py36_0    conda-forge
jupyter_core              4.3.0                    py36_0    conda-forge
jupyter_highlight_selected_word 0.0.11                   py36_0    conda-forge
jupyter_latex_envs        1.3.8.2                  py36_1    conda-forge
jupyter_nbextensions_configurator 0.2.7                    py36_0    conda-forge
jupyterhub                0.8.0b4                   <pip>
jupyterhub                0.9.4                    py36_0    conda-forge
jupyterhub-systemdspawner 0.9.10                    <pip>
jupyterlab                0.27.0                   py36_1    conda-forge
jupyterlab_launcher       0.5.1                    py36_0    conda-forge
libffi                    3.2.1                         3    conda-forge
libgcc                    5.2.0                         0    defaults
libgcc-ng                 7.2.0                hdf63c60_3    conda-forge
libgfortran               3.0.0                         1    defaults
libiconv                  1.14                          4    conda-forge
libpng                    1.6.28                        1    conda-forge
libsodium                 1.0.10                        0    conda-forge
libxcb                    1.12                          1    conda-forge
libxml2                   2.9.5                         0    conda-forge
Mako                      1.0.7                     <pip>
mako                      1.0.7                    py36_0    conda-forge
markupsafe                1.0                      py36_0    conda-forge
matplotlib                2.1.0                    py36_0    conda-forge
metis                     5.1.0                         3    conda-forge
mistune                   0.7.4                    py36_0    conda-forge
mpc                       1.0.3                         4    conda-forge
mpfr                      3.1.5                         0    conda-forge
mpich                     3.2                           5    conda-forge
mpmath                    1.0.0                      py_0    conda-forge
mumps                     5.0.2           blas_openblas_205  [blas_openblas]  conda-forge
nbconvert                 5.3.1                      py_1    conda-forge
nbformat                  4.4.0                    py36_0    conda-forge
nbgrader                  0.5.3                    py36_0    conda-forge
ncurses                   5.9                          10    conda-forge
nodejs                    6.11.0                        0    conda-forge
notebook                  5.1.0                    py36_0    conda-forge
numexpr                   2.6.4                    py36_0    conda-forge
numpy                     1.13.3          py36_blas_openblas_200  [blas_openblas]  conda-forge
oauthenticator            0.6.1                     <pip>
openblas                  0.2.19                        2    conda-forge
openssl                   1.0.2p               h470a237_1    conda-forge
opty                      0.2.0                    py36_0    conda-forge
packaging                 16.8                     py36_0    conda-forge
pamela                    0.3.0                    py36_0    conda-forge
pandas                    0.20.3                   py36_1    conda-forge
pandoc                    1.19.2                        0    conda-forge
pandocfilters             1.4.1                    py36_0    conda-forge
pandocfilters             1.4.2                     <pip>
parso                     0.1.0                     <pip>
pcre                      8.39                          0    conda-forge
pexpect                   4.2.1                    py36_0    conda-forge
pickleshare               0.7.4                    py36_0    conda-forge
pip                       9.0.1                    py36_0    conda-forge
prometheus_client         0.4.2                      py_0    conda-forge
prompt_toolkit            1.0.15                   py36_0    conda-forge
ptscotch                  6.0.4                         4    conda-forge
ptyprocess                0.5.2                    py36_0    conda-forge
pycosat                   0.6.3            py36h470a237_1    conda-forge
pycparser                 2.18                     py36_0    conda-forge
pycurl                    7.43.0                   py36_2    defaults
pydy                      0.4.0                    py36_0    pydy
pygments                  2.2.0                    py36_0    conda-forge
pyopenssl                 17.2.0                   py36_0    conda-forge
pyparsing                 2.2.0                    py36_0    conda-forge
pyqt                      5.6.0                    py36_4    conda-forge
pysocks                   1.6.7                    py36_0    conda-forge
python                    3.6.2                         0    conda-forge
python-dateutil           2.6.1                    py36_0    conda-forge
python-editor             1.0.3                    py36_0    conda-forge
python-editor             1.0.3                     <pip>
python-oauth2             1.0.1                    py36_0    conda-forge
pytz                      2017.2                   py36_0    conda-forge
pyyaml                    3.12                     py36_1    conda-forge
pyzmq                     16.0.2                   py36_2    conda-forge
qt                        5.6.2                         3    conda-forge
qtconsole                 4.3.1                    py36_0    conda-forge
readline                  6.2                           0    conda-forge
requests                  2.18.4                   py36_1    conda-forge
resonance                 0.19.1                py36_1000    conda-forge
resonance                 0.19.1                    <pip>
ruamel_yaml               0.11.14                  py36_0    conda-forge
scalapack                 2.0.2                         0    conda-forge
scipy                     0.19.1          py36_blas_openblas_202  [blas_openblas]  conda-forge
scotch                    6.0.4                         3    conda-forge
setuptools                36.5.0                    <pip>
setuptools                36.3.0                   py36_0    conda-forge
simplegeneric             0.8.1                    py36_0    conda-forge
sip                       4.18                     py36_1    conda-forge
six                       1.11.0                   py36_1    conda-forge
sqlalchemy                1.1.13                   py36_0    conda-forge
sqlite                    3.13.0                        1    conda-forge
sympy                     1.1.1                    py36_0    conda-forge
terminado                 0.6                      py36_0    conda-forge
testpath                  0.3.1                    py36_0    conda-forge
tk                        8.5.19                        2    conda-forge
tornado                   5.1.1            py36h470a237_0    conda-forge
traitlets                 4.3.2                    py36_0    conda-forge
uncertainties             3.0.2                      py_0    conda-forge
urllib3                   1.22                     py36_0    conda-forge
wcwidth                   0.1.7                    py36_0    conda-forge
webencodings              0.5                      py36_0    conda-forge
webencodings              0.5.1                     <pip>
wheel                     0.30.0                     py_1    conda-forge
widgetsnbextension        3.0.3                    py36_2    conda-forge
xlrd                      1.0.0                    py36_1    conda-forge
xlwt                      1.3.0                    py36_0    conda-forge
xorg-libxau               1.0.8                         3    conda-forge
xorg-libxdmcp             1.1.2                         3    conda-forge
xz                        5.2.3                         0    conda-forge
yaml                      0.1.6                         0    conda-forge
zeromq                    4.2.1                         1    conda-forge
zlib                      1.2.8                         3    conda-forge
root@bicycle:~# conda update pyzmq
Solving environment: done

## Package Plan ##

  environment location: /opt/conda

  added / updated specs: 
    - pyzmq

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    zeromq-4.2.5               |       hfc679d8_6         939 KB  conda-forge
    pyzmq-17.1.2               |   py36hae99301_1         1.0 MB  conda-forge
    libsodium-1.0.16           |       h470a237_1         536 KB  conda-forge
    libstdcxx-ng-7.2.0         |       hdf63c60_3         2.5 MB  conda-forge
    ------------------------------------------------------------
                                           Total:         5.0 MB

The following NEW packages will be INSTALLED:

    libstdcxx-ng: 7.2.0-hdf63c60_3 conda-forge

The following packages will be UPDATED:

    libsodium:    1.0.10-0         conda-forge --> 1.0.16-h470a237_1     conda-forge
    pyzmq:        16.0.2-py36_2    conda-forge --> 17.1.2-py36hae99301_1 conda-forge
    zeromq:       4.2.1-1          conda-forge --> 4.2.5-hfc679d8_6      conda-forge

Proceed ([y]/n)? y

Downloading and Extracting Packages
zeromq-4.2.5         | 939 KB    | ######################################################################################################################################################################## | 100% 
pyzmq-17.1.2         | 1.0 MB    | ######################################################################################################################################################################## | 100% 
libsodium-1.0.16     | 536 KB    | ######################################################################################################################################################################## | 100% 
libstdcxx-ng-7.2.0   | 2.5 MB    | ######################################################################################################################################################################## | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
root@bicycle:~# supervisorctl restart jupyterhub
jupyterhub: stopped
jupyterhub: started
root@bicycle:~# conda update --all
Solving environment: done

## Package Plan ##

  environment location: /opt/conda

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    bottleneck-1.2.1           |   py36h7eb728f_1         309 KB  conda-forge
    dbus-1.13.0                |       h3a4f0e9_0         560 KB  conda-forge
    pamela-0.3.0               |          py_1001           8 KB  conda-forge
    qtconsole-4.4.2            |           py36_0         156 KB  conda-forge
    wcwidth-0.1.7              |             py_1          17 KB  conda-forge
    jupyter_console-6.0.0      |             py_0          19 KB  conda-forge
    simplegeneric-0.8.1        |             py_1           7 KB  conda-forge
    sip-4.18.1                 |   py36hfc679d8_0         266 KB  conda-forge
    pyyaml-3.13                |   py36h470a237_1         460 KB  conda-forge
    cryptography-vectors-2.3.1 |        py36_1000        30.5 MB  conda-forge
    gst-plugins-base-1.12.5    |       hde13a9d_0         2.7 MB  conda-forge
    configurable-http-proxy-3.1.0|          node8_1         247 KB  conda-forge
    jupyter_nbextensions_configurator-0.4.0|        py36_1000         468 KB  conda-forge
    pyparsing-2.2.2            |             py_0          51 KB  conda-forge
    libffi-3.2.1               |       hfc679d8_5          51 KB  conda-forge
    ampl-mp-3.1.0              |       h26a2512_1         1.5 MB  conda-forge
    chardet-3.0.4              |        py36_1003         190 KB  conda-forge
    ipykernel-5.1.0            |     pyh24bf2e0_0          77 KB  conda-forge
    markupsafe-1.0             |   py36h470a237_1          33 KB  conda-forge
    decorator-4.3.0            |             py_0          10 KB  conda-forge
    sqlite-3.25.2              |       hb1c47c0_0         1.6 MB  conda-forge
    mpich-3.2.1                |       h26a2512_5         5.3 MB  conda-forge
    qt-5.6.2                   |       hf70d934_9        44.3 MB  conda-forge
    jedi-0.13.1                |        py36_1000         228 KB  conda-forge
    pexpect-4.6.0              |        py36_1000          75 KB  conda-forge
    jupyter_client-5.2.3       |             py_1          59 KB  conda-forge
    html5lib-1.0.1             |             py_0          89 KB  conda-forge
    gstreamer-1.12.5           |       h5856ed1_0         2.1 MB  conda-forge
    matplotlib-3.0.0           |                2           5 KB  conda-forge
    expat-2.2.5                |       hfc679d8_2         144 KB  conda-forge
    libcurl-7.61.1             |       hbdb9355_2         523 KB  conda-forge
    packaging-18.0             |             py_0          19 KB  conda-forge
    ipywidgets-7.4.2           |             py_0          92 KB  conda-forge
    libgfortran-ng-7.2.0       |       hdf63c60_3         1.2 MB  conda-forge
    pcre-8.41                  |       hfc679d8_3         243 KB  conda-forge
    asn1crypto-0.24.0          |        py36_1003         154 KB  conda-forge
    readline-7.0               |       haf1bffa_1         381 KB  conda-forge
    gettext-0.19.8.1           |       h5e8e0c9_1         3.5 MB  conda-forge
    metis-5.1.0                |       h470a237_3         4.1 MB  conda-forge
    ipopt-3.12.11              |blas_openblash1727795_0         2.0 MB  conda-forge
    libuuid-2.32.1             |       h470a237_2          24 KB  conda-forge
    xorg-libxau-1.0.8          |       h470a237_6          12 KB  conda-forge
    gmp-6.1.2                  |       hfc679d8_0         676 KB  conda-forge
    urllib3-1.23               |        py36_1001         152 KB  conda-forge
    traitlets-4.3.2            |        py36_1000         131 KB  conda-forge
    numpy-1.15.2               |py36_blas_openblashd3ea46f_1         9.3 MB  conda-forge
    ptscotch-6.0.6             |       h889977d_0         639 KB  conda-forge
    sqlalchemy-1.2.12          |   py36h470a237_0         1.6 MB  conda-forge
    python-dateutil-2.7.3      |             py_0         202 KB  conda-forge
    libxslt-1.1.32             |       h88dbc4e_2         511 KB  conda-forge
    jsonschema-2.6.0           |        py36_1002          62 KB  conda-forge
    pydy-0.4.0                 |           py36_0         5.0 MB  conda-forge
    libgfortran-3.0.0          |                1         281 KB  conda-forge
    mpi-1.0                    |            mpich           4 KB  conda-forge
    cffi-1.11.5                |   py36h5e8e0c9_1         404 KB  conda-forge
    mpmath-1.0.0               |             py_1         430 KB  conda-forge
    nbgrader-0.5.4             |           py36_0        12.0 MB  conda-forge
    parso-0.3.1                |             py_0          59 KB  conda-forge
    glib-2.56.2                |       h464dc38_0         4.9 MB  conda-forge
    pthread-stubs-0.4          |       h470a237_1           5 KB  conda-forge
    setuptools-40.4.3          |           py36_0         554 KB  conda-forge
    ipyparallel-6.2.2          |        py36_1000         314 KB  conda-forge
    jupyter_contrib_core-0.3.3 |             py_2          15 KB  conda-forge
    jupyter_highlight_selected_word-0.2.0|        py36_1000          13 KB  conda-forge
    terminado-0.8.1            |        py36_1001          23 KB  conda-forge
    bleach-3.0.2               |             py_0         125 KB  conda-forge
    jupyter-1.0.0              |             py_1           6 KB  conda-forge
    nbformat-4.4.0             |             py_1          91 KB  conda-forge
    pip-18.1                   |        py36_1000         1.8 MB  conda-forge
    pandas-0.23.4              |   py36hf8a1672_0        27.8 MB  conda-forge
    lxml-4.2.5                 |   py36hc9114bc_0         6.0 MB  conda-forge
    pytz-2018.5                |             py_0         193 KB  conda-forge
    openblas-0.2.20            |                8        17.0 MB  conda-forge
    libiconv-1.15              |       h470a237_3         2.0 MB  conda-forge
    uncertainties-3.0.2        |        py36_1001         117 KB  conda-forge
    entrypoints-0.2.3          |        py36_1002           9 KB  conda-forge
    backcall-0.1.0             |             py_0          13 KB  conda-forge
    webencodings-0.5.1         |             py_1          12 KB  conda-forge
    libpng-1.6.35              |       ha92aebf_2         305 KB  conda-forge
    zlib-1.2.11                |       h470a237_3          93 KB  conda-forge
    cython-0.29                |   py36hfc679d8_0         4.9 MB  conda-forge
    xlwt-1.3.0                 |             py_1          83 KB  conda-forge
    libxcb-1.13                |       h470a237_2         393 KB  conda-forge
    kiwisolver-1.0.1           |   py36h2d50403_2         924 KB  conda-forge
    libedit-3.1.20170329       |       haf1bffa_1         158 KB  conda-forge
    pygments-2.2.0             |             py_1         622 KB  conda-forge
    xorg-libxdmcp-1.1.2        |       h470a237_7          17 KB  conda-forge
    cycler-0.10.0              |             py_1           8 KB  conda-forge
    python-3.6.6               |       h5001a0f_3        20.5 MB  conda-forge
    yaml-0.1.7                 |       h470a237_1          77 KB  conda-forge
    gmpy2-2.0.8                |   py36hb705a9b_2         624 KB  conda-forge
    libssh2-1.8.0              |       h5b517e9_2         240 KB  conda-forge
    mako-1.0.7                 |             py_1          57 KB  conda-forge
    send2trash-1.5.0           |             py_0          12 KB  conda-forge
    jupyterlab_server-0.2.0    |             py_0          21 KB  conda-forge
    ruamel_yaml-0.15.71        |   py36h470a237_0         550 KB  conda-forge
    matplotlib-base-3.0.0      |   py36hc039c98_2         9.4 MB  conda-forge
    libgcc-7.2.0               |       h69d50b8_2         304 KB  conda-forge
    pickleshare-0.7.5          |        py36_1000          12 KB  conda-forge
    jinja2-2.10                |             py_1          89 KB  conda-forge
    tk-8.6.8                   |       ha92aebf_0         3.1 MB  conda-forge
    scalapack-2.0.2            |       h186ce0b_3         1.4 MB  conda-forge
    ncurses-6.1                |       hfc679d8_1         1.2 MB  conda-forge
    xlrd-1.1.0                 |             py_2         110 KB  conda-forge
    pycurl-7.43.0.2            |   py36hb7f436b_0          60 KB  defaults
    freetype-2.9.1             |       h6debe1e_4         800 KB  conda-forge
    cyipopt-0.1.9              |   py36h7eb728f_0         384 KB  conda-forge
    ptyprocess-0.6.0           |        py36_1000          22 KB  conda-forge
    pyopenssl-18.0.0           |        py36_1000          80 KB  conda-forge
    fastcache-1.0.2            |   py36h470a237_1          48 KB  conda-forge
    icu-58.2                   |       hfc679d8_0        22.8 MB  conda-forge
    pandocfilters-1.4.2        |             py_1           9 KB  conda-forge
    alembic-0.9.9              |             py_0         104 KB  conda-forge
    mpc-1.1.0                  |       hb705a9b_6         125 KB  conda-forge
    six-1.11.0                 |        py36_1001          21 KB  conda-forge
    requests-2.19.1            |        py36_1001          95 KB  conda-forge
    backports-1.0              |             py_2           4 KB  conda-forge
    scipy-1.1.0                |py36_blas_openblash7943236_201        40.3 MB  conda-forge
    ipython-7.0.1              |py36h24bf2e0_1000         1.1 MB  conda-forge
    jupyter_latex_envs-1.4.4   |        py36_1000         744 KB  conda-forge
    scotch-6.0.6               |       hc058ab9_0         1.1 MB  conda-forge
    pycparser-2.19             |             py_0          87 KB  conda-forge
    curl-7.61.1                |       h74213dd_2         134 KB  conda-forge
    jpeg-9c                    |       h470a237_1         229 KB  conda-forge
    krb5-1.16.1                |       hbb41f41_0         1.4 MB  conda-forge
    fontconfig-2.13.1          |       h65d0f4c_0         320 KB  conda-forge
    sympy-1.3                  |        py36_1000         9.4 MB  conda-forge
    testpath-0.4.2             |        py36_1000          91 KB  conda-forge
    jupyter_contrib_nbextensions-0.5.0|        py36_1000        19.4 MB  conda-forge
    jupyterlab-0.35.1          |           py36_0        10.5 MB  conda-forge
    mpfr-4.0.1                 |       h16a7912_0         602 KB  conda-forge
    nodejs-8.10.0              |                0        14.9 MB  conda-forge
    future-0.16.0              |        py36_1002         667 KB  conda-forge
    pyqt-5.6.0                 |   py36h8210e8a_7         5.4 MB  conda-forge
    cryptography-2.3.1         |   py36hdffb7b8_0         960 KB  conda-forge
    pandoc-2.3.1               |                0        20.7 MB  conda-forge
    opty-1.0.0                 |           py36_2          52 KB  conda-forge
    pysocks-1.6.8              |        py36_1002          22 KB  conda-forge
    numexpr-2.6.6              |   py36h470a237_1         419 KB  conda-forge
    widgetsnbextension-3.4.2   |        py36_1000         1.7 MB  conda-forge
    jupyterlab_launcher-0.13.1 |             py_2          20 KB  conda-forge
    ipython_genutils-0.2.0     |             py_1          21 KB  conda-forge
    libxml2-2.9.8              |       h422b904_5         1.8 MB  conda-forge
    python-editor-1.0.3        |             py_0           8 KB  conda-forge
    backports.functools_lru_cache-1.5|             py_1           6 KB  conda-forge
    notebook-5.7.0             |        py36_1000         7.3 MB  conda-forge
    prompt_toolkit-2.0.6       |             py_0         218 KB  conda-forge
    idna-2.7                   |        py36_1002         131 KB  conda-forge
    wheel-0.32.1               |           py36_0          34 KB  conda-forge
    xz-5.2.4                   |       h470a237_1         328 KB  conda-forge
    mumps-5.0.2                |blas_openblash613969f_212         2.4 MB  conda-forge
    mistune-0.8.4              |   py36h470a237_0          53 KB  conda-forge
    ------------------------------------------------------------
                                           Total:       408.1 MB

The following NEW packages will be INSTALLED:

    backcall:                          0.1.0-py_0                    conda-forge
    cryptography-vectors:              2.3.1-py36_1000               conda-forge
    jupyterlab_server:                 0.2.0-py_0                    conda-forge
    kiwisolver:                        1.0.1-py36h2d50403_2          conda-forge
    krb5:                              1.16.1-hbb41f41_0             conda-forge
    libcurl:                           7.61.1-hbdb9355_2             conda-forge
    libedit:                           3.1.20170329-haf1bffa_1       conda-forge
    libgfortran-ng:                    7.2.0-hdf63c60_3              conda-forge
    libssh2:                           1.8.0-h5b517e9_2              conda-forge
    libuuid:                           2.32.1-h470a237_2             conda-forge
    libxslt:                           1.1.32-h88dbc4e_2             conda-forge
    lxml:                              4.2.5-py36hc9114bc_0          conda-forge
    matplotlib-base:                   3.0.0-py36hc039c98_2          conda-forge
    mpi:                               1.0-mpich                     conda-forge
    parso:                             0.3.1-py_0                    conda-forge
    pthread-stubs:                     0.4-h470a237_1                conda-forge
    send2trash:                        1.5.0-py_0                    conda-forge

The following packages will be UPDATED:

    alembic:                           0.9.5-py36_0                  conda-forge --> 0.9.9-py_0                           conda-forge
    ampl-mp:                           3.1.0-0                       conda-forge --> 3.1.0-h26a2512_1                     conda-forge
    asn1crypto:                        0.22.0-py36_0                 conda-forge --> 0.24.0-py36_1003                     conda-forge
    backports:                         1.0-py36_1                    conda-forge --> 1.0-py_2                             conda-forge
    backports.functools_lru_cache:     1.4-py36_1                    conda-forge --> 1.5-py_1                             conda-forge
    bleach:                            2.0.0-py36_0                  conda-forge --> 3.0.2-py_0                           conda-forge
    bottleneck:                        1.2.1-py36_1                  conda-forge --> 1.2.1-py36h7eb728f_1                 conda-forge
    cffi:                              1.10.0-py36_0                 conda-forge --> 1.11.5-py36h5e8e0c9_1                conda-forge
    chardet:                           3.0.4-py36_0                  conda-forge --> 3.0.4-py36_1003                      conda-forge
    configurable-http-proxy:           3.0.0-0                       conda-forge --> 3.1.0-node8_1                        conda-forge
    cryptography:                      2.0.3-py36_0                  conda-forge --> 2.3.1-py36hdffb7b8_0                 conda-forge
    curl:                              7.52.1-0                      conda-forge --> 7.61.1-h74213dd_2                    conda-forge
    cycler:                            0.10.0-py36_0                 conda-forge --> 0.10.0-py_1                          conda-forge
    cyipopt:                           0.1.7-py36_2                  conda-forge --> 0.1.9-py36h7eb728f_0                 conda-forge
    cython:                            0.27.3-py36_0                 conda-forge --> 0.29-py36hfc679d8_0                  conda-forge
    dbus:                              1.10.22-0                     conda-forge --> 1.13.0-h3a4f0e9_0                    conda-forge
    decorator:                         4.1.2-py36_0                  conda-forge --> 4.3.0-py_0                           conda-forge
    entrypoints:                       0.2.3-py36_1                  conda-forge --> 0.2.3-py36_1002                      conda-forge
    expat:                             2.2.1-0                       conda-forge --> 2.2.5-hfc679d8_2                     conda-forge
    fastcache:                         1.0.2-py36_0                  conda-forge --> 1.0.2-py36h470a237_1                 conda-forge
    fontconfig:                        2.12.1-4                      conda-forge --> 2.13.1-h65d0f4c_0                    conda-forge
    freetype:                          2.7-1                         conda-forge --> 2.9.1-h6debe1e_4                     conda-forge
    future:                            0.16.0-py36_0                 conda-forge --> 0.16.0-py36_1002                     conda-forge
    gettext:                           0.19.7-1                      conda-forge --> 0.19.8.1-h5e8e0c9_1                  conda-forge
    glib:                              2.53.5-0                      conda-forge --> 2.56.2-h464dc38_0                    conda-forge
    gmp:                               6.1.2-0                       conda-forge --> 6.1.2-hfc679d8_0                     conda-forge
    gmpy2:                             2.0.8-py36_1                  conda-forge --> 2.0.8-py36hb705a9b_2                 conda-forge
    gst-plugins-base:                  1.8.0-0                       conda-forge --> 1.12.5-hde13a9d_0                    conda-forge
    gstreamer:                         1.8.0-1                       conda-forge --> 1.12.5-h5856ed1_0                    conda-forge
    html5lib:                          0.999999999-py36_0            conda-forge --> 1.0.1-py_0                           conda-forge
    icu:                               58.1-1                        conda-forge --> 58.2-hfc679d8_0                      conda-forge
    idna:                              2.6-py36_1                    conda-forge --> 2.7-py36_1002                        conda-forge
    ipopt:                             3.12.8-blas_openblas_0        conda-forge [blas_openblas] --> 3.12.11-blas_openblash1727795_0      conda-forge [blas_openblas]
    ipykernel:                         4.6.1-py36_0                  conda-forge --> 5.1.0-pyh24bf2e0_0                   conda-forge
    ipyparallel:                       6.0.2-py36_0                  conda-forge --> 6.2.2-py36_1000                      conda-forge
    ipython:                           6.2.1-py36_0                  conda-forge --> 7.0.1-py36h24bf2e0_1000              conda-forge
    ipython_genutils:                  0.2.0-py36_0                  conda-forge --> 0.2.0-py_1                           conda-forge
    ipywidgets:                        7.0.1-py_2                    conda-forge --> 7.4.2-py_0                           conda-forge
    jedi:                              0.10.2-py36_0                 conda-forge --> 0.13.1-py36_1000                     conda-forge
    jinja2:                            2.9.6-py36_0                  conda-forge --> 2.10-py_1                            conda-forge
    jpeg:                              9b-1                          conda-forge --> 9c-h470a237_1                        conda-forge
    jsonschema:                        2.6.0-py36_0                  conda-forge --> 2.6.0-py36_1002                      conda-forge
    jupyter:                           1.0.0-py36_0                  conda-forge --> 1.0.0-py_1                           conda-forge
    jupyter_client:                    5.1.0-py36_0                  conda-forge --> 5.2.3-py_1                           conda-forge
    jupyter_console:                   5.1.0-py36_0                  conda-forge --> 6.0.0-py_0                           conda-forge
    jupyter_contrib_core:              0.3.3-py36_0                  conda-forge --> 0.3.3-py_2                           conda-forge
    jupyter_contrib_nbextensions:      0.3.1-py36_0                  conda-forge --> 0.5.0-py36_1000                      conda-forge
    jupyter_core:                      4.3.0-py36_0                  conda-forge --> 4.4.0-py_0                           conda-forge
    jupyter_highlight_selected_word:   0.0.11-py36_0                 conda-forge --> 0.2.0-py36_1000                      conda-forge
    jupyter_latex_envs:                1.3.8.2-py36_1                conda-forge --> 1.4.4-py36_1000                      conda-forge
    jupyter_nbextensions_configurator: 0.2.7-py36_0                  conda-forge --> 0.4.0-py36_1000                      conda-forge
    jupyterlab:                        0.27.0-py36_1                 conda-forge --> 0.35.1-py36_0                        conda-forge
    jupyterlab_launcher:               0.5.1-py36_0                  conda-forge --> 0.13.1-py_2                          conda-forge
    libffi:                            3.2.1-3                       conda-forge --> 3.2.1-hfc679d8_5                     conda-forge
    libgcc:                            5.2.0-0                       defaults    --> 7.2.0-h69d50b8_2                     conda-forge
    libgfortran:                       3.0.0-1                       defaults    --> 3.0.0-1                              conda-forge
    libiconv:                          1.14-4                        conda-forge --> 1.15-h470a237_3                      conda-forge
    libpng:                            1.6.28-1                      conda-forge --> 1.6.35-ha92aebf_2                    conda-forge
    libxcb:                            1.12-1                        conda-forge --> 1.13-h470a237_2                      conda-forge
    libxml2:                           2.9.5-0                       conda-forge --> 2.9.8-h422b904_5                     conda-forge
    mako:                              1.0.7-py36_0                  conda-forge --> 1.0.7-py_1                           conda-forge
    markupsafe:                        1.0-py36_0                    conda-forge --> 1.0-py36h470a237_1                   conda-forge
    matplotlib:                        2.1.0-py36_0                  conda-forge --> 3.0.0-2                              conda-forge
    metis:                             5.1.0-3                       conda-forge --> 5.1.0-h470a237_3                     conda-forge
    mistune:                           0.7.4-py36_0                  conda-forge --> 0.8.4-py36h470a237_0                 conda-forge
    mpc:                               1.0.3-4                       conda-forge --> 1.1.0-hb705a9b_6                     conda-forge
    mpfr:                              3.1.5-0                       conda-forge --> 4.0.1-h16a7912_0                     conda-forge
    mpich:                             3.2-5                         conda-forge --> 3.2.1-h26a2512_5                     conda-forge
    mpmath:                            1.0.0-py_0                    conda-forge --> 1.0.0-py_1                           conda-forge
    mumps:                             5.0.2-blas_openblas_205       conda-forge [blas_openblas] --> 5.0.2-blas_openblash613969f_212      conda-forge [blas_openblas]
    nbformat:                          4.4.0-py36_0                  conda-forge --> 4.4.0-py_1                           conda-forge
    nbgrader:                          0.5.3-py36_0                  conda-forge --> 0.5.4-py36_0                         conda-forge
    ncurses:                           5.9-10                        conda-forge --> 6.1-hfc679d8_1                       conda-forge
    nodejs:                            6.11.0-0                      conda-forge --> 8.10.0-0                             conda-forge
    notebook:                          5.1.0-py36_0                  conda-forge --> 5.7.0-py36_1000                      conda-forge
    numexpr:                           2.6.4-py36_0                  conda-forge --> 2.6.6-py36h470a237_1                 conda-forge
    numpy:                             1.13.3-py36_blas_openblas_200 conda-forge [blas_openblas] --> 1.15.2-py36_blas_openblashd3ea46f_1  conda-forge [blas_openblas]
    openblas:                          0.2.19-2                      conda-forge --> 0.2.20-8                             conda-forge
    opty:                              0.2.0-py36_0                  conda-forge --> 1.0.0-py36_2                         conda-forge
    packaging:                         16.8-py36_0                   conda-forge --> 18.0-py_0                            conda-forge
    pamela:                            0.3.0-py36_0                  conda-forge --> 0.3.0-py_1001                        conda-forge
    pandas:                            0.20.3-py36_1                 conda-forge --> 0.23.4-py36hf8a1672_0                conda-forge
    pandoc:                            1.19.2-0                      conda-forge --> 2.3.1-0                              conda-forge
    pandocfilters:                     1.4.1-py36_0                  conda-forge --> 1.4.2-py_1                           conda-forge
    pcre:                              8.39-0                        conda-forge --> 8.41-hfc679d8_3                      conda-forge
    pexpect:                           4.2.1-py36_0                  conda-forge --> 4.6.0-py36_1000                      conda-forge
    pickleshare:                       0.7.4-py36_0                  conda-forge --> 0.7.5-py36_1000                      conda-forge
    pip:                               9.0.1-py36_0                  conda-forge --> 18.1-py36_1000                       conda-forge
    prompt_toolkit:                    1.0.15-py36_0                 conda-forge --> 2.0.6-py_0                           conda-forge
    ptscotch:                          6.0.4-4                       conda-forge --> 6.0.6-h889977d_0                     conda-forge
    ptyprocess:                        0.5.2-py36_0                  conda-forge --> 0.6.0-py36_1000                      conda-forge
    pycparser:                         2.18-py36_0                   conda-forge --> 2.19-py_0                            conda-forge
    pycurl:                            7.43.0-py36_2                 defaults    --> 7.43.0.2-py36hb7f436b_0              defaults   
    pydy:                              0.4.0-py36_0                  pydy        --> 0.4.0-py36_0                         conda-forge
    pygments:                          2.2.0-py36_0                  conda-forge --> 2.2.0-py_1                           conda-forge
    pyopenssl:                         17.2.0-py36_0                 conda-forge --> 18.0.0-py36_1000                     conda-forge
    pyparsing:                         2.2.0-py36_0                  conda-forge --> 2.2.2-py_0                           conda-forge
    pyqt:                              5.6.0-py36_4                  conda-forge --> 5.6.0-py36h8210e8a_7                 conda-forge
    pysocks:                           1.6.7-py36_0                  conda-forge --> 1.6.8-py36_1002                      conda-forge
    python:                            3.6.2-0                       conda-forge --> 3.6.6-h5001a0f_3                     conda-forge
    python-dateutil:                   2.6.1-py36_0                  conda-forge --> 2.7.3-py_0                           conda-forge
    python-editor:                     1.0.3-py36_0                  conda-forge --> 1.0.3-py_0                           conda-forge
    pytz:                              2017.2-py36_0                 conda-forge --> 2018.5-py_0                          conda-forge
    pyyaml:                            3.12-py36_1                   conda-forge --> 3.13-py36h470a237_1                  conda-forge
    qt:                                5.6.2-3                       conda-forge --> 5.6.2-hf70d934_9                     conda-forge
    qtconsole:                         4.3.1-py36_0                  conda-forge --> 4.4.2-py36_0                         conda-forge
    readline:                          6.2-0                         conda-forge --> 7.0-haf1bffa_1                       conda-forge
    requests:                          2.18.4-py36_1                 conda-forge --> 2.19.1-py36_1001                     conda-forge
    ruamel_yaml:                       0.11.14-py36_0                conda-forge --> 0.15.71-py36h470a237_0               conda-forge
    scalapack:                         2.0.2-0                       conda-forge --> 2.0.2-h186ce0b_3                     conda-forge
    scipy:                             0.19.1-py36_blas_openblas_202 conda-forge [blas_openblas] --> 1.1.0-py36_blas_openblash7943236_201 conda-forge [blas_openblas]
    scotch:                            6.0.4-3                       conda-forge --> 6.0.6-hc058ab9_0                     conda-forge
    setuptools:                        36.3.0-py36_0                 conda-forge --> 40.4.3-py36_0                        conda-forge
    simplegeneric:                     0.8.1-py36_0                  conda-forge --> 0.8.1-py_1                           conda-forge
    sip:                               4.18-py36_1                   conda-forge --> 4.18.1-py36hfc679d8_0                conda-forge
    six:                               1.11.0-py36_1                 conda-forge --> 1.11.0-py36_1001                     conda-forge
    sqlalchemy:                        1.1.13-py36_0                 conda-forge --> 1.2.12-py36h470a237_0                conda-forge
    sqlite:                            3.13.0-1                      conda-forge --> 3.25.2-hb1c47c0_0                    conda-forge
    sympy:                             1.1.1-py36_0                  conda-forge --> 1.3-py36_1000                        conda-forge
    terminado:                         0.6-py36_0                    conda-forge --> 0.8.1-py36_1001                      conda-forge
    testpath:                          0.3.1-py36_0                  conda-forge --> 0.4.2-py36_1000                      conda-forge
    tk:                                8.5.19-2                      conda-forge --> 8.6.8-ha92aebf_0                     conda-forge
    traitlets:                         4.3.2-py36_0                  conda-forge --> 4.3.2-py36_1000                      conda-forge
    uncertainties:                     3.0.2-py_0                    conda-forge --> 3.0.2-py36_1001                      conda-forge
    urllib3:                           1.22-py36_0                   conda-forge --> 1.23-py36_1001                       conda-forge
    wcwidth:                           0.1.7-py36_0                  conda-forge --> 0.1.7-py_1                           conda-forge
    webencodings:                      0.5-py36_0                    conda-forge --> 0.5.1-py_1                           conda-forge
    wheel:                             0.30.0-py_1                   conda-forge --> 0.32.1-py36_0                        conda-forge
    widgetsnbextension:                3.0.3-py36_2                  conda-forge --> 3.4.2-py36_1000                      conda-forge
    xlrd:                              1.0.0-py36_1                  conda-forge --> 1.1.0-py_2                           conda-forge
    xlwt:                              1.3.0-py36_0                  conda-forge --> 1.3.0-py_1                           conda-forge
    xorg-libxau:                       1.0.8-3                       conda-forge --> 1.0.8-h470a237_6                     conda-forge
    xorg-libxdmcp:                     1.1.2-3                       conda-forge --> 1.1.2-h470a237_7                     conda-forge
    xz:                                5.2.3-0                       conda-forge --> 5.2.4-h470a237_1                     conda-forge
    yaml:                              0.1.6-0                       conda-forge --> 0.1.7-h470a237_1                     conda-forge
    zlib:                              1.2.8-3                       conda-forge --> 1.2.11-h470a237_3                    conda-forge

Unlikely I'll get it back into that combo of packages. Not sure of best course.

moorepants commented 5 years ago

rebooted and now seems to work

moorepants commented 5 years ago

one thing isn't clear: is there a mixture of installs via conda and pip? if so, seems prone to error and not clear how jupyter is installed, conda or pip?

ixjlyons commented 5 years ago

Sorry it wasn't...straightforward to get things going. Reviewing what happened, it seems like the big mistake was not just locking down the version of all of the conda packages. Definitely have to be careful when updating the Jupyter stuff. I think it's an issue with each package's dependency version requirements getting tricky -- some of these projects have quite a few dependencies they all seem to move quickly.

Regarding the conda/pip stuff, the setup is pretty much the same as when I forked the repo. I suspect there may be cleaner or more robust ways to do it, but here's how it's put together as far as I can tell. There is system python (I think python2 only) by default in Ubuntu, but when conda is installed, the path to the conda bin directory is inserted first in PATH (in /etc/bash.bashrc and in a shell script in /etc/profile.d). As far as I know, this will cover all cases of users on the JupyterHub. You can see how this is done by Ansible here. Now, the conda task is run before the python3 task, so that means the pip_packages list here probably shouldn't be used unless there are dependencies you need that aren't on conda-forge or whatever. It's installing in the conda environment.

One thing you may want to look into a little is how to deal with multiple kernels and whatnot. I remember this being a little difficult, especially if it affects things in the jupyterhub config.

Is the whole thing working now?

moorepants commented 5 years ago

Sorry it wasn't...straightforward to get things going.

It was straightforward. There were a million things that just worked and a handful that I'll update the docs for.

Is the whole thing working now?

Yep.