Add documentation that the nodes need to have permission to write to the s3 bucket. I forgot to do this and my server wouldn't start, giving FailedMount errors:
[ec2-user@ip-172-31-34-163 ~]$ kubectl describe pod jupyter-rsignell-2dusgs
Name: jupyter-rsignell-2dusgs
Namespace: esip-pangeo
Priority: 0
PriorityClassName: <none>
Node: ip-172-20-129-141.ec2.internal/172.20.129.141
Start Time: Thu, 14 Mar 2019 13:04:05 +0000
Labels: app=jupyterhub
component=singleuser-server
heritage=jupyterhub
hub.jupyter.org/network-access-hub=true
Annotations: hub.jupyter.org/username: rsignell-usgs
Status: Pending
IP:
Init Containers:
block-cloud-metadata:
Container ID:
Image: jupyterhub/k8s-network-tools:81c2613
Image ID:
Port: <none>
Host Port: <none>
Command:
iptables
-A
OUTPUT
-d
169.254.169.254
-j
DROP
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from daskkubernetes-token-nqzk8 (ro)
Containers:
notebook:
Container ID:
Image: esip/pangeo-notebook:2019-03-14
Image ID:
Port: 8888/TCP
Host Port: 0/TCP
Args:
jupyterhub-singleuser
--ip="0.0.0.0"
--port=8888
--NotebookApp.default_url="/tree"
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 8
memory: 16106127360
Requests:
cpu: 1
memory: 4294967296
Environment:
EMAIL: rsignell-usgs@local
GIT_AUTHOR_NAME: rsignell-usgs
GIT_COMMITTER_NAME: rsignell-usgs
JUPYTERHUB_API_TOKEN: e6aff820af014be1b252508102fee5e7
JPY_API_TOKEN: e6aff820af014be1b252508102fee5e7
JUPYTERHUB_ADMIN_ACCESS: 1
JUPYTERHUB_CLIENT_ID: user-rsignell-usgs
JUPYTERHUB_HOST:
JUPYTERHUB_OAUTH_CALLBACK_URL: /user/rsignell-usgs/oauth_callback
JUPYTERHUB_USER: rsignell-usgs
JUPYTERHUB_API_URL: http://100.66.125.232:8081/hub/api
JUPYTERHUB_BASE_URL: /
JUPYTERHUB_SERVICE_PREFIX: /user/rsignell-usgs/
MEM_LIMIT: 16106127360
MEM_GUARANTEE: 4294967296
CPU_LIMIT: 8.0
CPU_GUARANTEE: 1.0
Mounts:
/home/jovyan from volume-rsignell-2dusgs (rw)
/s3 from s3 (rw)
/scratch from scratch (rw)
/var/run/secrets/kubernetes.io/serviceaccount from daskkubernetes-token-nqzk8 (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
volume-rsignell-2dusgs:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: claim-rsignell-2dusgs
ReadOnly: false
s3:
Type: FlexVolume (a generic volume resource that is provisioned/attached using an exec based plugin)
Driver: informaticslab/pysssix-flex-volume
FSType:
SecretRef: nil
ReadOnly: false
Options: map[readonly:true]
scratch:
Type: FlexVolume (a generic volume resource that is provisioned/attached using an exec based plugin)
Driver: informaticslab/goofys-flex-volume
FSType:
SecretRef: nil
ReadOnly: false
Options: map[bucket:esip-pangeo-scratch dirMode:0777 fileMode:0777]
daskkubernetes-token-nqzk8:
Type: Secret (a volume populated by a Secret)
SecretName: daskkubernetes-token-nqzk8
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 18s default-scheduler Successfully assigned esip-pangeo/jupyter-rsignell-2dusgs to ip-172-20-129-141.ec2.internal
Warning FailedMount 17s kubelet, ip-172-20-129-141.ec2.internal MountVolume.SetUp failed for volume "scratch" : mount command failed, status: Failure, reason: exit status 1: 2019/03/14 13:04:06.664104 main.FATAL Unable to mount file system, see syslog for details
Add documentation that the nodes need to have permission to write to the s3 bucket. I forgot to do this and my server wouldn't start, giving
FailedMounterrors: