choldgraf commented 6 years ago

Just had a meeting helping Aleksandr Blekh set up a JupyterHub for the South Big Data Hub. Here are some notes that we should follow up on:

Performance is not standard across Firefox vs. Chrome - we had issues getting the UI to work in firefox.
- [ ] TODO: recommend using google chrome
[x] TODO: We need to update the z2jh guide to use the online shell not the SDK
The issue he'd had before was something about read-only filesystems when running helm init. Worth following up on?
[ ] TODO: We need to remove the clusterrolebinding bit on helm
[x] TODO: Add 'nbgitpuller' to the "prepopulating" guide
[x] TODO: Make sure <your-release-name> is called this everywhere (that's what it's called in the helm upgrade section, and it may already be fine, just want to double check)
[x] TODO: Double check that the docker image that's in there works properly. I remember this didn't work at a recent workshop
QUESTION: quick search doesn't show up on the default page, is this the case for anybody else?
[x] TODO: make clear that the expanding and contracting size of cluster section is only for gcloud

Errors

We ran into the following error when trying to use a different docker image for the JHub. We tried both the jupyter scipy image and the jupyter minimal image, and both resulted in the same error:

`Timed out waiting for the condition`

QUESTION: Is nbgitpuller installed w/ repo2docker?

Other notes

Aleks may be interested in getting z2jh working with open stack
this is their project: https://matin.gatech.edu/about/hubzero
- Something like 200 users
- https://matin.gatech.edu:9012/hub/login

cc @yuvipanda

choldgraf commented 6 years ago

I'll make a PR to fix many of these in z2jh once we figure out the docker image problem

yuvipanda commented 6 years ago

https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/238 sorts out the clusterrole stuff in helm. Note that as of Kubernetes 1.8 (which will probably be default in a month or so) all the clusterrole / clusterrolebinding stuff will be completely required. The current default is basically 'everyone is root!' which is great UX wise but bad security wise. With 1.8, this is no longer the default, so we gotta adapt.

ablekh commented 6 years ago

@choldgraf Hi Chris, I just discovered this (will subscribe). Thank you for creating this issue and capturing detailed notes from our meeting. I have tried moving forward with this effort and I have both good and bad news.

The good news is that I was able to successfully update the cluster with a different JupyterHub image (your Errors section above). Specifically, I installed the jupyter/datascience-notebook image (tag: da2c5a4d00fa, which corresponds to latest as of today). I think that the issue we have experienced was due to a specific tag, that is, that specific image of minimal-notebook was, if not broken, not playing well with this GKE-based Kubernetes cluster. Another good news is that I was able to run our planned example notebooks without any problems.

The bad news is that, at some point, JupyterHub became unusable, due to refreshing the user home page forever (see attached screenshot). This is very frustrating, to a point that I started thinking again of using MATIN JH instance for the Oct 31 workshop. Nevertheless, I would appreciate help with that refresh issue as well as - if that is fixed - some direction on how to install nbgitpuller on my cluster: which node to attach to and how (kubectl attach?) as well as how to handle the lack of (or lack of access to) git in the JH image (which is very strange, considering its focus).

jh_refresh_error

ablekh commented 6 years ago

Some additional output (Cloud Shell) from my current attempts:

aleksandr_blekh@sbdh-jupyterhub-184020:~$ kubectl --namespace=sbdh-jhub get pods
NAME                                READY     STATUS              RESTARTS   AGE
hub-deployment-1179676071-c9x4q     1/1       Running             0          1h
jupyter-alex                        0/1       ContainerCreating   0          9s
proxy-deployment-1227971824-ffdhs   1/1       Running             0          1h
aleksandr_blekh@sbdh-jupyterhub-184020:~$ kubectl --namespace=sbdh-jhub get pods
NAME                                READY     STATUS              RESTARTS   AGE
hub-deployment-1179676071-c9x4q     1/1       Running             0          1h
jupyter-alex                        0/1       ContainerCreating   0          15s
proxy-deployment-1227971824-ffdhs   1/1       Running             0          1h
aleksandr_blekh@sbdh-jupyterhub-184020:~$ kubectl --namespace=sbdh-jhub get pods
NAME                                READY     STATUS    RESTARTS   AGE
hub-deployment-1179676071-c9x4q     1/1       Running   0          1h
jupyter-alex                        1/1       Running   1          21s
proxy-deployment-1227971824-ffdhs   1/1       Running   0          1h
aleksandr_blekh@sbdh-jupyterhub-184020:~$ kubectl --namespace=sbdh-jhub get pods
NAME                                READY     STATUS    RESTARTS   AGE
hub-deployment-1179676071-c9x4q     1/1       Running   0          1h
jupyter-alex                        0/1       Error     1          31s
proxy-deployment-1227971824-ffdhs   1/1       Running   0          1h

Note Error listed for my JH container, which manifested in the following error message on the Web:

500 : Internal Server Error
Spawner failed to start [status=1]

I tried to attach to the container(s) and/or get logs to investigate the issue, but so far failed. Any thoughts?

choldgraf commented 6 years ago

hmmm - so it works the first time, but then stops working after that. Does that happen after you log out / log back in? Or turn off / turn back on the server?

Could you paste the output of

kubectl --namespace=sbdh-jhub logs jupyter-alex?

(see this page for more helpful debugging info https://zero-to-jupyterhub.readthedocs.io/en/latest/debug.html)

ablekh commented 6 years ago

It has been happening without me logging out / back in. Here is the output of the kubectl logs command:

[W 2017-10-28 17:23:04.072 alex configurable:168] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
/opt/conda/lib/python3.6/site-packages/traitlets/config/configurable.py:84: UserWarning: Config option `cookie_name` not recognized by `HubOAuth`.
  self.config = config
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 528, in get
    value = obj._trait_values[self.name]
KeyError: 'oauth_client_id'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 6, in <module>
    main()
  File "/opt/conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 455, in main
    return SingleUserNotebookApp.launch_instance(argv)
  File "/opt/conda/lib/python3.6/site-packages/jupyter_core/application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 657, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-7>", line 2, in initialize
  File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1368, in initialize
    self.init_webapp()
  File "/opt/conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 393, in init_webapp
    self.init_hub_auth()
  File "/opt/conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 388, in init_hub_auth
    if not self.hub_auth.oauth_client_id:
  File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 556, in __get__
    return self.get(obj, cls)
  File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 535, in get
    value = self._validate(obj, dynamic_default())
  File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 593, in _validate
    value = self._cross_validate(obj, value)
  File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 599, in _cross_validate
    value = obj._trait_validators[self.name](obj, proposal)
  File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 907, in __call__
    return self.func(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/jupyterhub/services/auth.py", line 439, in _ensure_not_empty
    raise ValueError("%s cannot be empty." % proposal.trait.name)
ValueError: oauth_client_id cannot be empty.

choldgraf commented 6 years ago

hmmm - are you trying to use user authentication? And can you paste the content of your config.yaml file?

ablekh commented 6 years ago

Re: user authentication - No, currently, I'm using whatever is set up as default in the JH image, that is, no user authentication at all. The following is the contents of my config.yaml file:

hub:
  # output of first execution of 'openssl rand -hex 32'
  cookieSecret: "..."
proxy:
  # output of second execution of 'openssl rand -hex 32'
  secretToken: "..."

singleuser:
  image:
    name: jupyter/datascience-notebook
    tag: da2c5a4d00fa

choldgraf commented 6 years ago

hmm - can you try with a different docker image? I'm a bit worried that the jupyter-datascience image isn't configured to work with JupyterHub. Let's try building one with repo2docker (which will build an image assuming you're using JupyterHub to serve it). Do you want to give that a shot or alternatively I can try and then give you the URL of the image.

ablekh commented 6 years ago

If you could build a new suitable image, I would appreciate it. Ideally, it should contain most popular Python packages, including numpy and matplotlib, as well as git. Let me know, if you want to meet on WebEx - I am available pretty much all afternoon today ...

ablekh commented 6 years ago

@choldgraf I think I might know what is going on. Check out the following snippet from jupyter/docker-stacks repo's README (https://github.com/jupyter/docker-stacks#other-tips-and-known-issues):

Every notebook stack is compatible with JupyterHub 0.5 or higher. When running with JupyterHub, you must override the Docker run command to point to the start-singleuser.sh script, which starts a single-user instance of the Notebook server. See each stack's README for instructions on running with JupyterHub.

choldgraf commented 6 years ago

ah yeah, I bet that's it. OK lemme build an image...will do this when I get back home in a bit

choldgraf commented 6 years ago

hmmm, although it does look like that script exists in the datascience docker image: https://github.com/jupyter/docker-stacks/tree/master/datascience-notebook

we should try with a r2d-built image anyway

ablekh commented 6 years ago

Yes, the script, indeed, does exist in each of the jupyter/docker-stacks repos. I am currently trying to figure out how / whether it is possible to specify it instead of the default one. If not, then, as you're suggesting, building a new image is the way to go (though I am trying to avoid it).

ablekh commented 6 years ago

I refreshed on Docker documentation and think we can override the default command in the image like this:

hub:
  # output of first execution of 'openssl rand -hex 32'
  cookieSecret: "..."
proxy:
  # output of second execution of 'openssl rand -hex 32'
  secretToken: "..."

singleuser:
  image:
    name: jupyter/datascience-notebook
    tag: da2c5a4d00fa
    command: start-singleuser.sh

choldgraf commented 6 years ago

Can you put the course materials in a public github repo and add me as an owner?

ablekh commented 6 years ago

I've just tried to verify my suggestion by adding relevant command to config.yaml and updating the cluster with new settings. Unfortunately, the original issue still persists. I will think what else could be done, but it looks like your suggestion (rebuilding the image) is the way to go. I look forward to hearing from you.

ablekh commented 6 years ago

OK - give me a couple of minutes ...

ablekh commented 6 years ago

I'm having some git environment issues on my PC ... Could you create the repo? The content is attached. python_courses.zip

ablekh commented 6 years ago

Or you could use one I've already created. Upon your acceptance of invite, I will convert you into an owner.

choldgraf commented 6 years ago

cool, just accepted

ablekh commented 6 years ago

Initiated repo transfer to you ...

choldgraf commented 6 years ago

ok can you give this a shot:

https://hub.docker.com/r/choldgraf/sbdh/tags/

there's only one tag in there (v1)...see if that results in the same reboot errors.

(sorry it took so long, actually had to install ubuntu on my new laptop first)

choldgraf commented 6 years ago

(also you shouldn't need the command: line in config.yaml, just FYI)

ablekh commented 6 years ago

OK, after a second attempt (starting a new / non-existent user session), it seems to work (partially) - thank you! Having said that, so far only the home page works - when I tried to create a new notebook, JH could not connect to relevant kernel (see log below). Any thoughts?

[I 2017-10-28 23:20:39.527 aleks handlers:170] Creating new notebook in
[I 2017-10-28 23:20:44.653 aleks kernelmanager:98] Kernel started: 6595826c-e843-47d1-a5db-e8212ab29399
[I 2017-10-28 23:20:45.490 aleks handlers:193] Adapting to protocol v5.1 for kernel 6595826c-e843-47d1-a5db-e8212ab29399
[W 2017-10-28 23:20:45.491 aleks log:47] 400 GET /user/aleks/api/kernels/6595826c-e843-47d1-a5db-e8212ab29399/channels?session_id=7C28011107AB4CEF8D88444D9E637DE4 (10.16.0.1) 159.99ms referer=None
[W 2017-10-28 23:20:52.871 aleks handlers:257] Replacing stale connection: 6595826c-e843-47d1-a5db-e8212ab29399:7C28011107AB4CEF8D88444D9E637DE4
[W 2017-10-28 23:21:23.867 aleks handlers:257] Replacing stale connection: 6595826c-e843-47d1-a5db-e8212ab29399:7C28011107AB4CEF8D88444D9E637DE4
[W 2017-10-28 23:22:00.388 aleks handlers:257] Replacing stale connection: 6595826c-e843-47d1-a5db-e8212ab29399:7C28011107AB4CEF8D88444D9E637DE4
[I 2017-10-28 23:22:44.691 aleks handlers:181] Saving file at /Untitled1.ipynb
[W 2017-10-28 23:22:44.806 aleks handlers:257] Replacing stale connection: 6595826c-e843-47d1-a5db-e8212ab29399:7C28011107AB4CEF8D88444D9E637DE4
[W 2017-10-28 23:22:52.916 aleks handlers:257] Replacing stale connection: 6595826c-e843-47d1-a5db-e8212ab29399:7C28011107AB4CEF8D88444D9E637DE4
[W 2017-10-28 23:23:23.972 aleks handlers:257] Replacing stale connection: 6595826c-e843-47d1-a5db-e8212ab29399:7C28011107AB4CEF8D88444D9E637DE4
[W 2017-10-28 23:23:41.636 aleks handlers:257] Replacing stale connection: 6595826c-e843-47d1-a5db-e8212ab29399:7C28011107AB4CEF8D88444D9E637DE4
[W 2017-10-28 23:24:00.453 aleks handlers:257] Replacing stale connection: 6595826c-e843-47d1-a5db-e8212ab29399:7C28011107AB4CEF8D88444D9E637DE4
[W 2017-10-28 23:24:44.917 aleks handlers:257] Replacing stale connection: 6595826c-e843-47d1-a5db-e8212ab29399:7C28011107AB4CEF8D88444D9E637DE4
[W 2017-10-28 23:24:56.872 aleks handlers:257] Replacing stale connection: 6595826c-e843-47d1-a5db-e8212ab29399:7C28011107AB4CEF8D88444D9E637DE4
[W 2017-10-28 23:25:41.679 aleks handlers:257] Replacing stale connection: 6595826c-e843-47d1-a5db-e8212ab29399:7C28011107AB4CEF8D88444D9E637DE4

ablekh commented 6 years ago

Also, when creating a terminal session, it gets created, but terminal window is invisible in the new tab.

choldgraf commented 6 years ago

hmmm, but it seems to be working OK from a jupyterhub standpoint? Does the kernel connect properly when you run code in one notebook?

Also, what kind of kernel do these notebook expect? Perhaps they're expecting python 2 but it doesn't exist?

ablekh commented 6 years ago

No, that's my point - I cannot run any code in a notebook, since it could not connect to the kernel. A panel on the notebook page says "Connecting to kernel", then it times out with the following message:

A connection to the notebook server could not be established. The notebook will continue trying to reconnect. Check your network connection or notebook server configuration.

Here is the most recent log (for the latest session I just described):

[I 2017-10-28 23:37:23.604 aleks handlers:170] Creating new notebook in
[I 2017-10-28 23:37:25.660 aleks kernelmanager:98] Kernel started: 9dd1033f-1be1-4ee7-8e08-d2b3d8801a59
[I 2017-10-28 23:37:26.445 aleks handlers:193] Adapting to protocol v5.1 for kernel 9dd1033f-1be1-4ee7-8e08-d2b3d8801a59
[W 2017-10-28 23:37:26.447 aleks log:47] 400 GET /user/aleks/api/kernels/9dd1033f-1be1-4ee7-8e08-d2b3d8801a59/channels?session_id=4D8A6784A9C443D7935B661CC4B0B9CB (10.16.0.1) 319.68ms referer=None
[W 2017-10-28 23:37:27.579 aleks handlers:257] Replacing stale connection: 9dd1033f-1be1-4ee7-8e08-d2b3d8801a59:4D8A6784A9C443D7935B661CC4B0B9CB
[W 2017-10-28 23:37:49.649 aleks handlers:257] Replacing stale connection: 9dd1033f-1be1-4ee7-8e08-d2b3d8801a59:4D8A6784A9C443D7935B661CC4B0B9CB
[W 2017-10-28 23:38:13.674 aleks handlers:257] Replacing stale connection: 9dd1033f-1be1-4ee7-8e08-d2b3d8801a59:4D8A6784A9C443D7935B661CC4B0B9CB
[W 2017-10-28 23:38:41.714 aleks handlers:257] Replacing stale connection: 9dd1033f-1be1-4ee7-8e08-d2b3d8801a59:4D8A6784A9C443D7935B661CC4B0B9CB
[W 2017-10-28 23:39:17.730 aleks handlers:257] Replacing stale connection: 9dd1033f-1be1-4ee7-8e08-d2b3d8801a59:4D8A6784A9C443D7935B661CC4B0B9CB
[I 2017-10-28 23:39:25.616 aleks handlers:181] Saving file at /Untitled.ipynb
[W 2017-10-28 23:39:27.619 aleks handlers:257] Replacing stale connection: 9dd1033f-1be1-4ee7-8e08-d2b3d8801a59:4D8A6784A9C443D7935B661CC4B0B9CB

choldgraf commented 6 years ago

ah - I just checked the metadata of those ipynb files, and they're python 2 not 3. The environment I'm installing w/ environment.yml is for python 3. I'll build another w/ python 2, gimme one sec.

ablekh commented 6 years ago

I see. No problem - take your time ... UPDATE: Could you build one with support for both Python 2 and 3 (if not, I guess, Python 3 would be preferred, since most code in our sample content is P3-compatible)?

choldgraf commented 6 years ago

hmmm - I think it'll be easier to keep it w/ one language. It is possible to use multiple kernels, but might be overkill for this one. I uploaded another image (tag == 'v2') with two of the notebooks set to python 3 kernel (notebooks 1 and 2). Could you give that a shot?

https://hub.docker.com/r/choldgraf/sbdh/tags/

ablekh commented 6 years ago

I updated the cluster with your new image, but the issue remains. I suspect that I doing something wrong (or you haven't propagated the notebooks 1/2), because I also don't see the notebooks 1/2 you mentioned.

choldgraf commented 6 years ago

Can you give me the IP of your deployment?

ablekh commented 6 years ago

http://35.202.3.141

choldgraf commented 6 years ago

Hmm, I was able to create a notebook, though I didn't have any notebooks in my home dir... Hmm

ablekh commented 6 years ago

Hmm ... Does the notebook show that it is connected to a kernel? Can you run some code in it?

I just confirmed via GKE console that a single-user server, indeed, is running your latest image. I also found the following interesting info, which might shed some light on what is going on. Finally, based on my brief search, the issue might be related with how WebSockets interact with GKE's firewalls / load balancing ...

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: 2017-10-29T01:41:51Z
  labels:
    app: jupyterhub
    component: singleuser-server
    heritage: jupyterhub
    hub.jupyter.org/username: "11"
  name: jupyter-11
  namespace: sbdh-jhub
  resourceVersion: "292171"
  selfLink: /api/v1/namespaces/sbdh-jhub/pods/jupyter-11
  uid: 5626fb9e-bc4a-11e7-8242-42010a800034
spec:
  containers:
  - args:
    - jupyterhub-singleuser
    - --user="11"
    - --cookie-name="jupyter-hub-token-11"
    - --base-url="/user/11"
    - --hub-host=""
    - --hub-prefix="/hub/"
    - --hub-api-url="http://10.19.240.77:8081/hub/api"
    - --ip="0.0.0.0"
    - --port=8888
    env:
    - name: EMAIL
      value: 11@local
    - name: JPY_API_TOKEN
      value: cee14c4f0a4a4822bd5ee875e49236d7
    - name: JPY_COOKIE_NAME
      value: jupyter-hub-token-11
    - name: JPY_BASE_URL
      value: /user/11
    - name: GIT_AUTHOR_NAME
      value: "11"
    - name: JPY_USER
      value: "11"
    - name: JPY_HUB_PREFIX
      value: /hub/
    - name: JUPYTERHUB_API_TOKEN
      value: cee14c4f0a4a4822bd5ee875e49236d7
    - name: MEM_GUARANTEE
      value: "1073741824"
    - name: GIT_COMMITTER_NAME
      value: "11"
    - name: JPY_HUB_API_URL
      value: http://10.19.240.77:8081/hub/api
    image: choldgraf/sbdh:v2
    imagePullPolicy: IfNotPresent
    lifecycle: {}
    name: notebook
    ports:
    - containerPort: 8888
      name: notebook-port
      protocol: TCP
    resources:
      requests:
        memory: "1073741824"
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /home/jovyan
      name: volume-11
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: no-api-access-please
      readOnly: true
  dnsPolicy: ClusterFirst
  nodeName: gke-sbdh-jhub-default-pool-84ddf97a-h3tf
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1000
    runAsUser: 1000
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.alpha.kubernetes.io/notReady
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.alpha.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: volume-11
    persistentVolumeClaim:
      claimName: claim-11
  - emptyDir: {}
    name: no-api-access-please
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2017-10-29T01:41:58Z
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2017-10-29T01:42:08Z
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2017-10-29T01:41:58Z
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://cf5d6802671f1c7e79f090651cc88b40fcc79d9997d17573c474a35d77eb3f5d
    image: choldgraf/sbdh:v2
    imageID: docker-pullable://choldgraf/sbdh@sha256:325396468149c3d0ff9e23723ffb0dddd4c44dad24bd9518612dd6afcb26ffa4
    lastState: {}
    name: notebook
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2017-10-29T01:42:08Z
  hostIP: 10.128.0.2
  phase: Running
  podIP: 10.16.0.35
  qosClass: Burstable
  startTime: 2017-10-29T01:41:58Z

choldgraf commented 6 years ago

Yep, I was able to import numpy. What browser are you on?

On Sat, Oct 28, 2017, 7:26 PM Aleksandr Blekh notifications@github.com wrote:

Hmm ... Does the notebook show that it is connected to a kernel? Can you run some code in it?

I just confirmed via GKE console that a single-user server, indeed, is running your latest image. I also found the following interesting info, which might shed some light on what is going on. Finally, based on my brief search, the issue might be related with how WebSockets interact with GKE's firewalls / load balancing ...

apiVersion: v1 kind: Pod metadata: creationTimestamp: 2017-10-29T01:41:51Z labels: app: jupyterhub component: singleuser-server heritage: jupyterhub hub.jupyter.org/username: "11" name: jupyter-11 namespace: sbdh-jhub resourceVersion: "292171" selfLink: /api/v1/namespaces/sbdh-jhub/pods/jupyter-11 uid: 5626fb9e-bc4a-11e7-8242-42010a800034 spec: containers:

args:

jupyterhub-singleuser

--user="11"

--cookie-name="jupyter-hub-token-11"

--base-url="/user/11"

--hub-host=""

--hub-prefix="/hub/"

--hub-api-url="http://10.19.240.77:8081/hub/api"

--ip="0.0.0.0"

--port=8888 env:

name: EMAIL value: 11@local

name: JPY_API_TOKEN value: cee14c4f0a4a4822bd5ee875e49236d7

name: JPY_COOKIE_NAME value: jupyter-hub-token-11

name: JPY_BASE_URL value: /user/11

name: GIT_AUTHOR_NAME value: "11"

name: JPY_USER value: "11"

name: JPY_HUB_PREFIX value: /hub/

name: JUPYTERHUB_API_TOKEN value: cee14c4f0a4a4822bd5ee875e49236d7

name: MEM_GUARANTEE value: "1073741824"

name: GIT_COMMITTER_NAME value: "11"

name: JPY_HUB_API_URL value: http://10.19.240.77:8081/hub/api image: choldgraf/sbdh:v2 imagePullPolicy: IfNotPresent lifecycle: {} name: notebook ports:

containerPort: 8888 name: notebook-port protocol: TCP resources: requests: memory: "1073741824" terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:

mountPath: /home/jovyan name: volume-11

mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: no-api-access-please readOnly: true dnsPolicy: ClusterFirst nodeName: gke-sbdh-jhub-default-pool-84ddf97a-h3tf restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 1000 runAsUser: 1000 serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations:

effect: NoExecute key: node.alpha.kubernetes.io/notReady operator: Exists tolerationSeconds: 300

effect: NoExecute key: node.alpha.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes:

name: volume-11 persistentVolumeClaim: claimName: claim-11

emptyDir: {} name: no-api-access-please status: conditions:

lastProbeTime: null lastTransitionTime: 2017-10-29T01:41:58Z status: "True" type: Initialized

lastProbeTime: null lastTransitionTime: 2017-10-29T01:42:08Z status: "True" type: Ready

lastProbeTime: null lastTransitionTime: 2017-10-29T01:41:58Z status: "True" type: PodScheduled containerStatuses:

containerID: docker://cf5d6802671f1c7e79f090651cc88b40fcc79d9997d17573c474a35d77eb3f5d image: choldgraf/sbdh:v2 imageID: docker-pullable://choldgraf/sbdh@sha256:325396468149c3d0ff9e23723ffb0dddd4c44dad24bd9518612dd6afcb26ffa4 lastState: {} name: notebook ready: true restartCount: 0 state: running: startedAt: 2017-10-29T01:42:08Z hostIP: 10.128.0.2 phase: Running podIP: 10.16.0.35 qosClass: Burstable startTime: 2017-10-29T01:41:58Z

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/237#issuecomment-340232905, or mute the thread https://github.com/notifications/unsubscribe-auth/ABwSHejmj560sQKy0MONfnJhsBx8mU8Vks5sw-JAgaJpZM4QGs_l .

ablekh commented 6 years ago

Firefox. Trying to test this on Chrome now ...

choldgraf commented 6 years ago

FWIW, I'm testing it out on my phone haha. Just tried logging our and in, and importing matplotlib, and it worked

ablekh commented 6 years ago

Ha! Cool. I just tested in Chrome and it still cannot connect to the kernel ... :-(

ablekh commented 6 years ago

One thing that I have figured out, though, is the reason for not seeing the JH user homepage (was stuck on refresh page). Basically, the cluster did not have enough resources and some pods were unschedulable. I've increased the cluster size to two nodes and now it works OK.

choldgraf commented 6 years ago

Man that is confusing. I'll try from another computer when I get home. Do you have any non standard settings with cookies? I'm pretty confused as to why the notebooks aren't copied to my home folder.

Can you add me as an owner to the cloud project? Maybe I can debug more quickly there.

choldgraf commented 6 years ago

Ahh I see. Makes sense. I'll try again

choldgraf commented 6 years ago

Just tried starting a pod but it seems to be erroring. Any errors with kubectl? You can also try deleting the hub and proxy pods. Sometimes thats needed to clear stuff out.

ablekh commented 6 years ago

Re: cookie settings - not sure, but will take a look. BTW, out of curiosity, I tried to run a notebook via my phone (Windows Mobile) and, to my surprise, the connection to kernel seems to be successful. Sure, I will add you to the cloud project ...

ablekh commented 6 years ago

Re: starting your pod - Yeah, this is the same situation: too many pods for modest resources. Let me delete some unused pods ... Try again in a couple of minutes.

choldgraf commented 6 years ago

Specifically try deleting the hub and proxy. They'll be restarted automatically

ablekh commented 6 years ago

Done. I left your session pods intact (let me know, if you want me to delete them as well). The restarted hub and proxy pods are ready.

ablekh commented 6 years ago

Can I use your Berkeley e-mail address, when adding to the cloud project?

choldgraf commented 6 years ago

OK, pod spun up fine. Still no files in there though. I bet we can get that working with nbgitpuller in a pinch though

jupyterhub / zero-to-jupyterhub-k8s

Notes from aleks (South Big Data Hub) meeting #237

Errors

Other notes