RBAC errors - Githubissues

tomerleib commented 6 years ago

Hello, I'm trying to deploy a chart of 0.7.0 on RBAC enabled K8s and encountering the following error:

Error: roles.rbac.authorization.k8s.io "hook-image-awaiter" is forbidden: 
attempt to grant extra privileges: [{[get] [apps] [daemonsets] [] []}] 
user=&{system:serviceaccount:kube-system:tiller a771a5f1-a11f-11e8-b230-0a34a3b41e3c [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} 
ownerrules=[] ruleResolutionErrors=[clusterroles.rbac.authorization.k8s.io "cluster-admin" not found]

This also repeats itself for the hub service account. If I manually create the service accounts, the deployment is completed but Jupyterhub is not working and provides the following error:

Spawn failed: (422) Reason: error HTTP response headers: HTTPHeaderDict({'Audit-Id': 'af07a73f-48ea-4b49-914d-ecbe9904661b', 'Content-Type': 'application/json', 'Date': 'Thu, 13 Sep 2018 12:17:27 GMT', 'Content-Length': '438'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod \"jupyter-test\" is invalid: spec.initContainers[0].securityContext.privileged: Forbidden: disallowed by cluster policy","reason":"Invalid","details":{"name":"jupyter-test","kind":"Pod","causes":[{"reason":"FieldValueForbidden","message":"Forbidden: disallowed by cluster policy","field":"spec.initContainers[0].securityContext.privileged"}]},"code":422}

I'm using K8s 1.11 running on ec2 instances (not EKS). Thanks for the help.

consideRatio commented 6 years ago

Oh cool K8s 1.11! Are you using a release candidate of helm 2.11? I don't think helm supports K8s 1.11 in version 2.10, but I would have expected another error earlier if that was the case. Can you run helm version for me?

Thoughts:

oh so there is no cluster-admin role being predefined in the cluster as it should be as far as I understand. How was the kubernetes cluster setup?
Forbidden: disallowed by cluster policy makes me ask the same question as point 1

tomerleib commented 6 years ago

Hi, well, I'll surprise you here with my Helm versions:

Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

Running multiple services and all of them were deployed using the stable helm charts.

Now, for cluster-admin, I get your point, however, the role does exists in my cluster. For example, my tiller configuration is:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: tiller-clusterrolebinding
subjects:
- kind: ServiceAccount
  name: tiller
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: ""

I guess that if I had any issues with the cluster-admin, any other chart that does requires RBAC will fail as well. An example of such chart that I'm using and also use RBAC is jenkins. https://github.com/helm/charts/tree/master/stable/jenkins

LAST DEPLOYED: Thu Sep 13 17:59:30 2018
NAMESPACE: tests
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRoleBinding
NAME                        AGE
jenkins-tests-role-binding  8m

==> v1/Service
NAME                        TYPE          CLUSTER-IP      EXTERNAL-IP       PORT(S)                       AGE
jenkins-tests-agent         ClusterIP     10.152.183.105  <none>            50000/TCP                     8m
jenkins-tests-internal-svc  LoadBalancer  10.152.183.146  internal-a9e1...  80:31351/TCP,50000:31634/TCP  8m
jenkins-tests               LoadBalancer  10.152.183.70   <pending>         443:30842/TCP                 8m

==> v1beta1/Deployment
NAME           DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
jenkins-tests  1        1        1           1          8m

==> v1/Pod(related)
NAME                            READY  STATUS   RESTARTS  AGE
jenkins-tests-6dcd767ff4-6hqbp  1/1    Running  0         8m

==> v1/Secret
NAME           TYPE    DATA  AGE
jenkins-tests  Opaque  2     8m

==> v1/ConfigMap
NAME                 DATA  AGE
jenkins-tests        3     8m
jenkins-tests-tests  1     8m

==> v1/PersistentVolumeClaim
NAME           STATUS  VOLUME                                    CAPACITY  ACCESS MODES  STORAGECLASS  AGE
jenkins-tests  Bound   pvc-9e0f7ec0-b765-11e8-8ab7-0a34a3b41e3c  1Mi       RWO           efs           8m

==> v1/ServiceAccount
NAME           SECRETS  AGE
jenkins-tests  1        8m

consideRatio commented 6 years ago

I'm out of my comfort zone trying to help here, so I'm going to ask questions that may help or may be pointless to know the answer to. But with my limited knowledge it may guide me.

As I understand it, when you run things on amazon EC2, the nodes wont have a concept of "kubernetes" until you have configured them to get that. Have you used kubeadm, kops to setup kubernetes? Did you set it up by following the z2jh.jupyter.org guide?

Also, in what namespace did you install the chart?

tomerleib commented 6 years ago

I have used kubeadm for bootstrapping and creating the cluster. It was installed to namespace data-dev

בתאריך יום ה׳, 13 בספט׳ 2018, 18:53, מאת Erik Sundell ‏< notifications@github.com>:

I'm out of my comfort zone trying to help here, so I'm going to ask questions that may help or may be pointless to know the answer to. But with my limited knowledge it may guide me.

As I understand it, when you run things on amazon EC2, the nodes wont have a concept of "kubernetes" until you have configured them to get that. Have you used kubeadm, kops? Did you set it up by following the z2jh.jupyter.org guide?

Also, in what namespace did you install the chart?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/937#issuecomment-421057206, or mute the thread https://github.com/notifications/unsubscribe-auth/AWNlPxZDMa6vukA4Cg65RWJ0WqjVPfLZks5uan9lgaJpZM4WnK7a .

consideRatio commented 6 years ago

@tomerleib okay, regarding the second error Spawn failed: (422): I have never worked with "cluster policy" and such, it seems like one needs to learn more about that. We are having a initContainer that requests to run with higher privileges, and that seems to crash things.

To disable this security fix that makes us need higher privileges:

singleuser:
  cloudMetadata:
    enabled: true # this is false by default

consideRatio commented 6 years ago

I dont know the parts that needed fixing in Helm to support k8s 1.11, perhaps you should try their release candidates?They are supporting k8s 1.11 i know, and see if the issues goes away.

andrewcheny commented 5 years ago

Same issue here running on microk8s with RBAC enabled. k8s 1.16

ghost commented 4 years ago

I can say I might hit the same issue here with the below microk8s helm enabled environment when I am trying to install jupyterhub from the helm repository as follow :

sudo microk8s helm version
Client: &version.Version{SemVer:"v2.16.7", GitCommit:"5f2584fd3d35552c4af26036f0c464191287986b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.7", GitCommit:"5f2584fd3d35552c4af26036f0c464191287986b", GitTreeState:"clean"}

Kubernetes info :


sudo snap list
microk8s  v1.18.4    1503  latest/stable  canonical✓  classic

sudo microk8s status --wait-ready
microk8s is running
addons:
dashboard: enabled
dns: enabled
helm: enabled
helm3: enabled
ingress: enabled
metrics-server: enabled
storage: enabled

sudo microk8s kubectl get pods --namespace jhub -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP           NODE      NOMINATED NODE   READINESS GATES
continuous-image-puller-wkst8     1/1     Running   0          57m   10.1.23.37   shuswap   <none>           <none>
hub-845885fb99-rk5rv              1/1     Running   4          57m   10.1.23.40   shuswap   <none>           <none>
proxy-8b7cff7dc-mkfzk             1/1     Running   0          57m   10.1.23.38   shuswap   <none>           <none>
user-scheduler-65f4cbb9b7-7s8hq   1/1     Running   0          57m   10.1.23.41   shuswap   <none>           <none>
user-scheduler-65f4cbb9b7-frjfl   1/1     Running   0          57m   10.1.23.39   shuswap   <none>           <none>

This is what I see when I navigate to the jupyter's proxy-public ip address and I am unable to start the jupyterhub server

Spawn failed

The latest attempt to start your server has failed. Would you like to retry starting it?

and the logs of the jhub pod at that time :

 sudo microk8s kubectl --namespace jhub get services -o wide
NAME           TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE   SELECTOR
hub            ClusterIP      10.152.183.241   <none>        8081/TCP                     65m   app=jupyterhub,component=hub,release=jhub
proxy-api      ClusterIP      10.152.183.114   <none>        8001/TCP                     65m   app=jupyterhub,component=proxy,release=jhub
proxy-public   LoadBalancer   10.152.183.96    <pending>     443:32442/TCP,80:30851/TCP   65m   component=proxy,release=jhub

sudo microk8s kubectl --namespace jhub logs hub-845885fb99-rk5rv -f

[W 2020-06-27 22:29:40.912 JupyterHub base:950] User admin is slow to start (timeout=0)
[I 2020-06-27 22:29:40.913 JupyterHub log:174] 302 GET /hub/spawn/admin -> /hub/spawn-pending/admin (admin@10.1.23.1) 32.39ms
[I 2020-06-27 22:29:40.919 JupyterHub spawner:1769] PVC claim-admin already exists, so did not create new pvc.
[E 2020-06-27 22:29:40.927 JupyterHub spawner:1807] Failed for {'api_version': 'v1',
     'kind': 'Pod',
     'metadata': {'annotations': {'hub.jupyter.org/username': 'admin'},
                  'cluster_name': None,
                  'creation_timestamp': None,
                  'deletion_grace_period_seconds': None,
                  'deletion_timestamp': None,
                  'finalizers': None,
                  'generate_name': None,
                  'generation': None,
                  'initializers': None,
                  'labels': {'app': 'jupyterhub',
                             'chart': 'jupyterhub-0.9.0',
                             'component': 'singleuser-server',
                             'heritage': 'jupyterhub',
                             'hub.jupyter.org/network-access-hub': 'true',
                             'release': 'jhub'},
                  'managed_fields': None,
                  'name': 'jupyter-admin',
                  'namespace': None,
                  'owner_references': None,
                  'resource_version': None,
                  'self_link': None,
                  'uid': None},
     'spec': {'active_deadline_seconds': None,
              'affinity': {'node_affinity': {'preferred_during_scheduling_ignored_during_execution': [{'preference': {'matchExpressions': [{'key': 'hub.jupyter.org/node-purpose',
                                                                                                                                            'operator': 'In',
                                                                                                                                            'values': ['user']}]},
                                                                                                       'weight': 100}],
                                             'required_during_scheduling_ignored_during_execution': None},
                           'pod_affinity': None,
                           'pod_anti_affinity': None},
              'automount_service_account_token': False,
              'containers': [{'args': ['jupyterhub-singleuser',
                                       '--ip=0.0.0.0',
                                       '--port=8888'],
                              'command': None,
                              'env': [{'name': 'JUPYTERHUB_API_TOKEN',
                                       'value': '6ab063b8ab0d4379acdbda5a0480d4af',
                                       'value_from': None},
                                      {'name': 'JPY_API_TOKEN',
                                       'value': '6ab063b8ab0d4379acdbda5a0480d4af',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_ADMIN_ACCESS',
                                       'value': '1',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_CLIENT_ID',
                                       'value': 'jupyterhub-user-admin',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_HOST',
                                       'value': '',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_OAUTH_CALLBACK_URL',
                                       'value': '/user/admin/oauth_callback',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_USER',
                                       'value': 'admin',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_SERVER_NAME',
                                       'value': '',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_API_URL',
                                       'value': 'http://10.152.183.241:8081/hub/api',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_ACTIVITY_URL',
                                       'value': 'http://10.152.183.241:8081/hub/api/users/admin/activity',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_BASE_URL',
                                       'value': '/',
                                       'value_from': None},
                                      {'name': 'JUPYTERHUB_SERVICE_PREFIX',
                                       'value': '/user/admin/',
                                       'value_from': None},
                                      {'name': 'MEM_GUARANTEE',
                                       'value': '1073741824',
                                       'value_from': None},
                                      {'name': 'JUPYTER_IMAGE_SPEC',
                                       'value': 'jupyterhub/k8s-singleuser-sample:0.9.0',
                                       'value_from': None},
                                      {'name': 'JUPYTER_IMAGE',
                                       'value': 'jupyterhub/k8s-singleuser-sample:0.9.0',
                                       'value_from': None}],
                              'env_from': None,
                              'image': 'jupyterhub/k8s-singleuser-sample:0.9.0',
                              'image_pull_policy': 'IfNotPresent',
                              'lifecycle': {},
                              'liveness_probe': None,
                              'name': 'notebook',
                              'ports': [{'container_port': 8888,
                                         'host_ip': None,
                                         'host_port': None,
                                         'name': 'notebook-port',
                                         'protocol': None}],
                              'readiness_probe': None,
                              'resources': {'limits': {},
                                            'requests': {'memory': 1073741824}},
                              'security_context': {'allow_privilege_escalation': None,
                                                   'capabilities': None,
                                                   'privileged': None,
                                                   'proc_mount': None,
                                                   'read_only_root_filesystem': None,
                                                   'run_as_group': 0,
                                                   'run_as_non_root': None,
                                                   'run_as_user': 1000,
                                                   'se_linux_options': None},
                              'stdin': None,
                              'stdin_once': None,
                              'termination_message_path': None,
                              'termination_message_policy': None,
                              'tty': None,
                              'volume_devices': None,
                              'volume_mounts': [{'mount_path': '/home/jovyan',
                                                 'mount_propagation': None,
                                                 'name': 'volume-admin',
                                                 'read_only': None,
                                                 'sub_path': None,
                                                 'sub_path_expr': None}],
                              'working_dir': None}],
              'dns_config': None,
              'dns_policy': None,
              'enable_service_links': None,
              'host_aliases': None,
              'host_ipc': None,
              'host_network': None,
              'host_pid': None,
              'hostname': None,
              'image_pull_secrets': None,
              'init_containers': [{'args': None,
                                   'command': ['iptables',
                                               '-A',
                                               'OUTPUT',
                                               '-d',
                                               '169.254.169.254',
                                               '-j',
                                               'DROP'],
                                   'env': None,
                                   'env_from': None,
                                   'image': 'jupyterhub/k8s-network-tools:0.9.0',
                                   'image_pull_policy': None,
                                   'lifecycle': None,
                                   'liveness_probe': None,
                                   'name': 'block-cloud-metadata',
                                   'ports': None,
                                   'readiness_probe': None,
                                   'resources': None,
                                   'security_context': {'allow_privilege_escalation': None,
                                                        'capabilities': {'add': ['NET_ADMIN'],
                                                                         'drop': None},
                                                        'privileged': True,
                                                        'proc_mount': None,
                                                        'read_only_root_filesystem': None,
                                                        'run_as_group': None,
                                                        'run_as_non_root': None,
                                                        'run_as_user': 0,
                                                        'se_linux_options': None},
                                   'stdin': None,
                                   'stdin_once': None,
                                   'termination_message_path': None,
                                   'termination_message_policy': None,
                                   'tty': None,
                                   'volume_devices': None,
                                   'volume_mounts': None,
                                   'working_dir': None}],
              'node_name': None,
              'node_selector': None,
              'priority': None,
              'priority_class_name': None,
              'readiness_gates': None,
              'restart_policy': 'OnFailure',
              'runtime_class_name': None,
              'scheduler_name': 'jhub-user-scheduler',
              'security_context': {'fs_group': 100,
                                   'run_as_group': None,
                                   'run_as_non_root': None,
                                   'run_as_user': None,
                                   'se_linux_options': None,
                                   'supplemental_groups': None,
                                   'sysctls': None},
              'service_account': None,
              'service_account_name': None,
              'share_process_namespace': None,
              'subdomain': None,
              'termination_grace_period_seconds': None,
              'tolerations': [{'effect': 'NoSchedule',
                               'key': 'hub.jupyter.org/dedicated',
                               'operator': 'Equal',
                               'toleration_seconds': None,
                               'value': 'user'},
                              {'effect': 'NoSchedule',
                               'key': 'hub.jupyter.org_dedicated',
                               'operator': 'Equal',
                               'toleration_seconds': None,
                               'value': 'user'}],
              'volumes': [{'aws_elastic_block_store': None,
                           'azure_disk': None,
                           'azure_file': None,
                           'cephfs': None,
                           'cinder': None,
                           'config_map': None,
                           'csi': None,
                           'downward_api': None,
                           'empty_dir': None,
                           'fc': None,
                           'flex_volume': None,
                           'flocker': None,
                           'gce_persistent_disk': None,
                           'git_repo': None,
                           'glusterfs': None,
                           'host_path': None,
                           'iscsi': None,
                           'name': 'volume-admin',
                           'nfs': None,
                           'persistent_volume_claim': {'claimName': 'claim-admin'},
                           'photon_persistent_disk': None,
                           'portworx_volume': None,
                           'projected': None,
                           'quobyte': None,
                           'rbd': None,
                           'scale_io': None,
                           'secret': None,
                           'storageos': None,
                           'vsphere_volume': None}]},
     'status': None}
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 1801, in _start
        pod,
      File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 1560, in asynchronize
        return method(*args, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6115, in create_namespaced_pod
        (data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6206, in create_namespaced_pod_with_http_info
        collection_formats=collection_formats)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 334, in call_api
        _return_http_data_only, collection_formats, _preload_content, _request_timeout)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 168, in __call_api
        _request_timeout=_request_timeout)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 377, in request
        body=body)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/rest.py", line 266, in POST
        body=body)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/rest.py", line 222, in request
        raise ApiException(http_resp=r)
    kubernetes.client.rest.ApiException: (422)
    Reason: Unprocessable Entity
    HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Sat, 27 Jun 2020 22:29:40 GMT', 'Content-Length': '440'})
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod \"jupyter-admin\" is invalid: spec.initContainers[0].securityContext.privileged: Forbidden: disallowed by cluster policy","reason":"Invalid","details":{"name":"jupyter-admin","kind":"Pod","causes":[{"reason":"FieldValueForbidden","message":"Forbidden: disallowed by cluster policy","field":"spec.initContainers[0].securityContext.privileged"}]},"code":422}

[E 2020-06-27 22:29:40.928 JupyterHub user:640] Unhandled error starting admin's server: (422)
    Reason: Unprocessable Entity
    HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Sat, 27 Jun 2020 22:29:40 GMT', 'Content-Length': '440'})
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod \"jupyter-admin\" is invalid: spec.initContainers[0].securityContext.privileged: Forbidden: disallowed by cluster policy","reason":"Invalid","details":{"name":"jupyter-admin","kind":"Pod","causes":[{"reason":"FieldValueForbidden","message":"Forbidden: disallowed by cluster policy","field":"spec.initContainers[0].securityContext.privileged"}]},"code":422}

[W 2020-06-27 22:29:40.943 JupyterHub base:922] 2 consecutive spawns failed.  Hub will exit if failure count reaches 5 before succeeding
[E 2020-06-27 22:29:40.944 JupyterHub gen:599] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py:845> exception=ApiException()> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 593, in error_callback
        future.result()
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 852, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/user.py", line 656, in spawn
        raise e
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/user.py", line 560, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 1801, in _start
        pod,
      File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 1560, in asynchronize
        return method(*args, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6115, in create_namespaced_pod
        (data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6206, in create_namespaced_pod_with_http_info
        collection_formats=collection_formats)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 334, in call_api
        _return_http_data_only, collection_formats, _preload_content, _request_timeout)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 168, in __call_api
        _request_timeout=_request_timeout)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 377, in request
        body=body)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/rest.py", line 266, in POST
        body=body)
      File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/rest.py", line 222, in request
        raise ApiException(http_resp=r)
    kubernetes.client.rest.ApiException: (422)
    Reason: error
    HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Sat, 27 Jun 2020 22:29:40 GMT', 'Content-Length': '440'})
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod \"jupyter-admin\" is invalid: spec.initContainers[0].securityContext.privileged: Forbidden: disallowed by cluster policy","reason":"Invalid","details":{"name":"jupyter-admin","kind":"Pod","causes":[{"reason":"FieldValueForbidden","message":"Forbidden: disallowed by cluster policy","field":"spec.initContainers[0].securityContext.privileged"}]},"code":422}

[E 2020-06-27 22:29:41.057 JupyterHub pages:325] Previous spawn for admin failed: (422)
    Reason: error
    HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Sat, 27 Jun 2020 22:29:40 GMT', 'Content-Length': '440'})
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod \"jupyter-admin\" is invalid: spec.initContainers[0].securityContext.privileged: Forbidden: disallowed by cluster policy","reason":"Invalid","details":{"name":"jupyter-admin","kind":"Pod","causes":[{"reason":"FieldValueForbidden","message":"Forbidden: disallowed by cluster policy","field":"spec.initContainers[0].securityContext.privileged"}]},"code":422}

[E 2020-06-27 22:29:41.058 JupyterHub log:166] {
      "X-Forwarded-Host": "10.152.183.96",
      "X-Forwarded-Proto": "http",
      "X-Forwarded-Port": "80",
      "X-Forwarded-For": "10.1.23.1",
      "Upgrade-Insecure-Requests": "1",
      "Cookie": "jupyterhub-hub-login=[secret]; jupyterhub-session-id=[secret]",
      "Connection": "close",
      "Referer": "http://10.152.183.96/hub/spawn-pending/admin",
      "Accept-Encoding": "gzip, deflate",
      "Accept-Language": "en-CA,en-US;q=0.7,en;q=0.3",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
      "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0",
      "Host": "10.152.183.96"
    }
[E 2020-06-27 22:29:41.058 JupyterHub log:174] 500 GET /hub/spawn-pending/admin (admin@10.1.23.1) 4.01ms
[I 2020-06-27 22:29:43.545 JupyterHub log:174] 200 GET /hub/health (@10.1.23.1) 0.77ms
^C

I installed jupyterhub after adding repo and update the helm repositories and issuing the install command as following :

RELEASE=jhub
NAMESPACE=jhub
config.yaml contains the hex value

sudo microk8s helm install jupyterhub/jupyterhub --name $RELEASE  --namespace $NAMESPACE    --version=0.9.0   --values config.yaml

This is all the environment I have :

sudo microk8s kubectl get all --all-namespaces
NAMESPACE     NAME                                            READY   STATUS    RESTARTS   AGE
default       pod/microbot-6d97548556-gtppx                   1/1     Running   0          163m
default       pod/microbot-6d97548556-xlq2z                   1/1     Running   0          163m
ingress       pod/nginx-ingress-microk8s-controller-m6jjk     1/1     Running   0          70m
jhub          pod/continuous-image-puller-wkst8               1/1     Running   0          69m
jhub          pod/hub-845885fb99-rk5rv                        1/1     Running   4          69m
jhub          pod/proxy-8b7cff7dc-mkfzk                       1/1     Running   0          69m
jhub          pod/user-scheduler-65f4cbb9b7-7s8hq             1/1     Running   0          69m
jhub          pod/user-scheduler-65f4cbb9b7-frjfl             1/1     Running   0          69m
kube-system   pod/coredns-588fd544bf-pr9bd                    1/1     Running   0          170m
kube-system   pod/dashboard-metrics-scraper-59f5574d4-brwcl   1/1     Running   0          170m
kube-system   pod/hostpath-provisioner-75fdc8fccd-s4x49       1/1     Running   0          135m
kube-system   pod/kubernetes-dashboard-6d97855997-g7rhj       1/1     Running   0          170m
kube-system   pod/metrics-server-c65c9d66-6v2v6               1/1     Running   0          170m
kube-system   pod/tiller-deploy-6cc8b596c8-fb4lf              1/1     Running   0          155m

NAMESPACE     NAME                                TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
default       service/kubernetes                  ClusterIP      10.152.183.1     <none>        443/TCP                      171m
default       service/microbot-service            NodePort       10.152.183.33    <none>        80:31280/TCP                 163m
jhub          service/hub                         ClusterIP      10.152.183.241   <none>        8081/TCP                     69m
jhub          service/proxy-api                   ClusterIP      10.152.183.114   <none>        8001/TCP                     69m
jhub          service/proxy-public                LoadBalancer   10.152.183.96    <pending>     443:32442/TCP,80:30851/TCP   69m
kube-system   service/dashboard-metrics-scraper   ClusterIP      10.152.183.112   <none>        8000/TCP                     170m
kube-system   service/kube-dns                    ClusterIP      10.152.183.10    <none>        53/UDP,53/TCP,9153/TCP       170m
kube-system   service/kubernetes-dashboard        ClusterIP      10.152.183.121   <none>        443/TCP                      170m
kube-system   service/metrics-server              ClusterIP      10.152.183.234   <none>        443/TCP                      170m
kube-system   service/tiller-deploy               ClusterIP      10.152.183.21    <none>        44134/TCP                    159m

NAMESPACE   NAME                                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
ingress     daemonset.apps/nginx-ingress-microk8s-controller   1         1         1       1            1           <none>          70m
jhub        daemonset.apps/continuous-image-puller             1         1         1       1            1           <none>          69m

NAMESPACE     NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
default       deployment.apps/microbot                    2/2     2            2           163m
jhub          deployment.apps/hub                         1/1     1            1           69m
jhub          deployment.apps/proxy                       1/1     1            1           69m
jhub          deployment.apps/user-scheduler              2/2     2            2           69m
kube-system   deployment.apps/coredns                     1/1     1            1           170m
kube-system   deployment.apps/dashboard-metrics-scraper   1/1     1            1           170m
kube-system   deployment.apps/hostpath-provisioner        1/1     1            1           135m
kube-system   deployment.apps/kubernetes-dashboard        1/1     1            1           170m
kube-system   deployment.apps/metrics-server              1/1     1            1           170m
kube-system   deployment.apps/tiller-deploy               1/1     1            1           159m

NAMESPACE     NAME                                                  DESIRED   CURRENT   READY   AGE
default       replicaset.apps/microbot-6d97548556                   2         2         2       163m
jhub          replicaset.apps/hub-845885fb99                        1         1         1       69m
jhub          replicaset.apps/proxy-8b7cff7dc                       1         1         1       69m
jhub          replicaset.apps/user-scheduler-65f4cbb9b7             2         2         2       69m
kube-system   replicaset.apps/coredns-588fd544bf                    1         1         1       170m
kube-system   replicaset.apps/dashboard-metrics-scraper-59f5574d4   1         1         1       170m
kube-system   replicaset.apps/hostpath-provisioner-75fdc8fccd       1         1         1       135m
kube-system   replicaset.apps/kubernetes-dashboard-6d97855997       1         1         1       170m
kube-system   replicaset.apps/metrics-server-c65c9d66               1         1         1       170m
kube-system   replicaset.apps/tiller-deploy-596578979c              0         0         0       159m
kube-system   replicaset.apps/tiller-deploy-6cc8b596c8              1         1         1       155m

NAMESPACE   NAME                                READY   AGE
jhub        statefulset.apps/user-placeholder   0/0     69m

I would appreciate any wise advise to resolve that issue of ,"message":"Forbidden: disallowed by cluster policy" ?

Thank you!

consideRatio commented 4 years ago

@aydintd see #1798, set singleuser.cloudMetada.blockWithIptables: false, while using the absolute latest version where #1798 is merged, as found through https://jupyterhub.github.io/helm-chart/#development-releases-jupyterhub

consideRatio commented 4 years ago

Closing this as it is a collection of different issues hard to track, but most are resolved by the PR mentioned above, and the others relate to use of helm2, and now helm3 is out.

jupyterhub / zero-to-jupyterhub-k8s

RBAC errors #937