Installing Jupyterhub on a cluster created by Kops on AWS gets stuck and does not install anything

[ocloarec]

I have been trying to install JupyterHub on a cluster installed by kopson on aws. When I run

helm install jupyterhub/jupyterhub --version=v0.4 --name=origin --namespace=origin -f config.yaml

it get stuck and when I call kubectl --namespace=origin get pod after more tha half an hour I obtain

NAME READY STATUS RESTARTS AGE pull-all-nodes-1509098240-origin-1-gt6n0 1/1 Running 0 35m

I make it work with the Heptio version but I struggle using HTTPS with this one. I wanted to add ingress with HTTPS access on the cluster created using kops.

Thanks is advance.

[yuvipanda]

Heya! I suspect it's because it has RBAC enabled. We're working on making that work better, but in the meantime you can use https://kubernetes.io/docs/admin/authorization/rbac/#permissive-rbac-permissions.

Also, we're working on making https much easier, with #229, and hopefully have a v0.5 out this week.

Thank you for filing this issue!

[ocloarec]

Thank you for your answer!

This command was part of my installation flow since it is already in your documentation.

I am glad to know that your are making https much easier. I would be happy to help testing.

[yuvipanda]

@ocloarec cool! can you check out https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/229#issuecomment-340620215?

[rwangr]

@yuvipanda Same issue I have met even enabled RBAC. The log of pod pull-all-nodes-1510502451-jupyter-xxx is as below:

2017-11-12T16:00:52.984259669Z Pulling jupyterhub/k8s-singleuser-sample:3bf055a on 0 nodes
2017-11-12T16:00:53.146211584Z job "pull--3bf055a-1510502452" created
2017-11-12T16:00:55.247064525Z Pulled  of 0 nodes
2017-11-12T16:00:55.247093982Z sh: 0: unknown operand
2017-11-12T16:00:57.35643594Z Pulled  of 0 nodes
2017-11-12T16:00:57.356464196Z sh: 0: unknown operand
2017-11-12T16:00:59.461607034Z Pulled  of 0 nodes
2017-11-12T16:00:59.461635059Z sh: 0: unknown operand
......

After disable prePull in config.yaml, the helm release creation is successful. So the error happened while pull image to nodes probably. But no more idea then.

[yuvipanda]

This should be fixed once #275 is merged!

[tianhuil]

Hi, @yuvipanda: I'm encountered a similar issue as but on DO instead of AWS. I have a repo with a Makefile documenting how to bring up DO + kubernetes. Granting permissive rbac and using v0.5.0-aee2160 version as per https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/229#issuecomment-340620215 allowed the installation of Jupyterhub!

[houghb]

@yuvipanda, I am experiencing the same problem as ocloarec, and see the same thing in the logs as rwangr. I've tried using permissive RBAC role bindings, disabling prePull in config.yaml, disabling rbac in config.yaml, and using both v0.5.0-aee2160 and v0.5 but still experience this issue and can't install Jupyterhub. Do you have any other suggestions?

[cam72cam]

I am seeing the same issue with 0.4 on AWS kops. How do I disable prePull? I am having trouble finding docs on that.

Pulling jupyterhub/k8s-singleuser-sample:v0.4 on 0 nodes
job "pull--v0-4-1513097698" created
Pulled  of 0 nodes
sh: 0: unknown operand
Pulled  of 0 nodes
sh: 0: unknown operand
Pulled  of 0 nodes
sh: 0: unknown operand
sh: 0: unknown operand
Pulled  of 0 nodes
Pulled  of 0 nodes
sh: 0: unknown operand
Pulled  of 0 nodes
sh: 0: unknown operand
sh: 0: unknown operand
Pulled  of 0 nodes
sh: 0: unknown operand
Pulled  of 0 nodes
sh: 0: unknown operand

[tianhuil]

Christian: you can disable prepull in the Yaml config. Look at the yaml files in this repo. On Tue, Dec 12, 2017 at 11:56 AM Christian Mesh notifications@github.com wrote:

I am seeing the same issue with 0.4 on AWS kops. How do I disable prePull? I am having trouble finding docs on that.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/239#issuecomment-351114203, or mute the thread https://github.com/notifications/unsubscribe-auth/AB0WkX5v2sPHjx_gBNZ1rHnuGQbwolmKks5s_rAvgaJpZM4QJFgB .

-- Michael Li Training and placing PhDs to be data scientists: http://www.thedataincubator.com (Venture Beat) New bootcamp more competitive than Harvard http://venturebeat.com/2014/04/15/ny-gets-new-bootcamp-for-data-scientists-its-free-but-harder-to-get-into-than-harvard/ (HBR) The question to ask before hiring a data scientist http://blogs.hbr.org/2014/08/the-question-to-ask-before-hiring-a-data-scientist/

[cam72cam]

I found my issue, the github search was useless. After I cloned and grepped I found

prePuller:
  enabled: false

which fixed the issue. Thanks

[willingc]

@yuvipanda I'm thinking that this issue can be closed.

[cam72cam]

I am still having issues with the pre-puller when trying to use RBAC on AWS:

1045  kops create cluster $NAME   --zones $ZONES  --master-size t2.micro   --master-volume-size 10   --node-size t2.medium   --node-volume-size 10 --yes
 1046  kops validate cluster
 1047  kubectl apply -f storage.yml 
 1048  kubectl --namespace kube-system create serviceaccount tiller
 1049  kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
 1050  helm init --service-account tiller
 1051  helm version
 1052  kubectl --namespace=kube-system patch deployment tiller-deploy --type=json --patch='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["/tiller", "--listen=localhost:44134"]}]'
 1053  helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
 1054  helm repo update
 1055  helm install jupyterhub/jupyterhub     --version=v0.6     --name=rbac-test     --namespace=rbac-test     -f config.yaml

Error: clusterroles.rbac.authorization.k8s.io "pre-puller-1521469466-rbac-test-1" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["nodes"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["nodes"], APIGroups:[""], Verbs:["get"]}] user=&{system:serviceaccount:kube-system:tiller 19eb1fd4-2b81-11e8-87bd-12a057a9bbbe [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[] ruleResolutionErrors=[clusterroles.rbac.authorization.k8s.io "cluster-admin" not found]

config.yaml

proxy:
  secretToken: "<snip>"

[cam72cam]

Scratch that, I forgot to add the --authorization RBAC to the create cluster command

[consideRatio]

RBAC is now enabled by default on AWS, we have also updated the docs to repflect this. The issue causing image-pulling to halt is also fixed properly with the 0.7.0 release of the chart.