search

jupyterhub / kubespawner

Kubernetes spawner for JupyterHub
https://jupyterhub-kubespawner.readthedocs.io
BSD 3-Clause "New" or "Revised" License
541 stars 303 forks source link

Setting ResourceQuota for hub namespace not letting spawn any pod #669

Closed lucianolacurcia closed 1 year ago

lucianolacurcia commented 1 year ago

Bug description

Unable to set ResourceQuota for limiting memory on jupyterhub namespace. When pods are spawned they appear to not have the resource.limits.memory value required by the ResourceQuota. When the singleuser pods is spawned by the hub by sending the request to kubernetes api, it gets 403 error.

Expected behaviour

The pod is spawned if it fits on the quota or is not spawned if doesnt fit on the quota.

Actual behaviour

This is what the hub sends to the kubernetes api:

{
   "api_version":"v1",
   "kind":"Pod",
   "metadata":{
      "annotations":{
         "hub.jupyter.org/username":"appopenshift"
      },
      "cluster_name":"None",
      "creation_timestamp":"None",
      "deletion_grace_period_seconds":"None",
      "deletion_timestamp":"None",
      "finalizers":"None",
      "generate_name":"None",
      "generation":"None",
      "labels":{
         "app":"jupyterhub",
         "chart":"jupyterhub-2.0.0",
         "component":"singleuser-server",
         "heritage":"jupyterhub",
         "hub.jupyter.org/network-access-hub":"true",
         "hub.jupyter.org/servername":"",
         "hub.jupyter.org/username":"appopenshift",
         "release":"jupyterhub-stable"
      },
      "managed_fields":"None",
      "name":"jupyter-appopenshift",
      "namespace":"None",
      "owner_references":"None",
      "resource_version":"None",
      "self_link":"None",
      "uid":"None"
   },
   "spec":{
      "active_deadline_seconds":"None",
      "affinity":{
         "node_affinity":{
            "preferred_during_scheduling_ignored_during_execution":[
               {
                  "preference":{
                     "matchExpressions":[
                        {
                           "key":"hub.jupyter.org/node-purpose",
                           "operator":"In",
                           "values":[
                              "user"
                           ]
                        }
                     ]
                  },
                  "weight":100
               }
            ],
            "required_during_scheduling_ignored_during_execution":"None"
         },
         "pod_affinity":"None",
         "pod_anti_affinity":"None"
      },
      "automount_service_account_token":false,
      "containers":[
         {
            "args":[
               "jupyterhub-singleuser"
            ],
            "command":"None",
            "env":[
               {
                  "name":"JPY_API_TOKEN",
                  "value":"5d95a0fa25f3419f9de70c50065b4f8a",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTERHUB_ACTIVITY_URL",
                  "value":"http://hub:8081/hub/api/users/appopenshift/activity",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTERHUB_ADMIN_ACCESS",
                  "value":"1",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTERHUB_API_TOKEN",
                  "value":"5d95a0fa25f3419f9de70c50065b4f8a",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTERHUB_API_URL",
                  "value":"http://hub:8081/hub/api",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTERHUB_BASE_URL",
                  "value":"/",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTERHUB_CLIENT_ID",
                  "value":"jupyterhub-user-appopenshift",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTERHUB_DEFAULT_URL",
                  "value":"/lab",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTERHUB_HOST",
                  "value":"",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTERHUB_OAUTH_ACCESS_SCOPES",
                  "value":"[\"access:servers!server=appopenshift/\", ""\"access:servers!user=appopenshift\"]",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTERHUB_OAUTH_CALLBACK_URL",
                  "value":"/user/appopenshift/oauth_callback",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES",
                  "value":"[]",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTERHUB_OAUTH_SCOPES",
                  "value":"[\"access:servers!server=appopenshift/\", ""\"access:servers!user=appopenshift\"]",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTERHUB_SERVER_NAME",
                  "value":"",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTERHUB_SERVICE_PREFIX",
                  "value":"/user/appopenshift/",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTERHUB_SERVICE_URL",
                  "value":"http://0.0.0.0:8888/user/appopenshift/",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTERHUB_USER",
                  "value":"appopenshift",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTER_IMAGE",
                  "value":"lucianolacurcia/jupyterhub-rstudio:94d4b1611e25",
                  "value_from":"None"
               },
               {
                  "name":"JUPYTER_IMAGE_SPEC",
                  "value":"lucianolacurcia/jupyterhub-rstudio:94d4b1611e25",
                  "value_from":"None"
               },
               {
                  "name":"MEM_GUARANTEE",
                  "value":"1073741824",
                  "value_from":"None"
               },
               {
                  "name":"MEM_LIMIT",
                  "value":"1073741824",
                  "value_from":"None"
               }
            ],
            "env_from":"None",
            "image":"lucianolacurcia/jupyterhub-rstudio:94d4b1611e25",
            "image_pull_policy":"IfNotPresent",
            "lifecycle":{

            },
            "liveness_probe":"None",
            "name":"notebook",
            "ports":[
               {
                  "container_port":8888,
                  "host_ip":"None",
                  "host_port":"None",
                  "name":"notebook-port",
                  "protocol":"None"
               }
            ],
            "readiness_probe":"None",
            "resources":{
               "limits":{
                  "memory":1073741824
               },
               "requests":{
                  "memory":1073741824
               }
            },
            "security_context":{
               "allowPrivilegeEscalation":false,
               "runAsUser":1000
            },
            "startup_probe":"None",
            "stdin":"None",
            "stdin_once":"None",
            "termination_message_path":"None",
            "termination_message_policy":"None",
            "tty":"None",
            "volume_devices":"None",
            "volume_mounts":[
               {
                  "mount_path":"/home/jovyan",
                  "mount_propagation":"None",
                  "name":"volume-appopenshift",
                  "read_only":"None",
                  "sub_path":"None",
                  "sub_path_expr":"None"
               }
            ],
            "working_dir":"None"
         }
      ],
      "dns_config":"None",
      "dns_policy":"None",
      "enable_service_links":"None",
      "ephemeral_containers":"None",
      "host_aliases":"None",
      "host_ipc":"None",
      "host_network":"None",
      "host_pid":"None",
      "hostname":"None",
      "image_pull_secrets":[

      ],
      "init_containers":[
         {
            "args":"None",
            "command":[
               "iptables",
               "-A",
               "OUTPUT",
               "-d",
               "169.254.169.254",
               "-j",
               "DROP"
            ],
            "env":"None",
            "env_from":"None",
            "image":"jupyterhub/k8s-network-tools:2.0.0",
            "image_pull_policy":"None",
            "lifecycle":"None",
            "liveness_probe":"None",
            "name":"block-cloud-metadata",
            "ports":"None",
            "readiness_probe":"None",
            "resources":{

            },
            "security_context":{
               "allow_privilege_escalation":"None",
               "capabilities":{
                  "add":[
                     "NET_ADMIN"
                  ],
                  "drop":"None"
               },
               "privileged":true,
               "proc_mount":"None",
               "read_only_root_filesystem":"None",
               "run_as_group":"None",
               "run_as_non_root":"None",
               "run_as_user":0,
               "se_linux_options":"None",
               "seccomp_profile":"None",
               "windows_options":"None"
            },
            "startup_probe":"None",
            "stdin":"None",
            "stdin_once":"None",
            "termination_message_path":"None",
            "termination_message_policy":"None",
            "tty":"None",
            "volume_devices":"None",
            "volume_mounts":"None",
            "working_dir":"None"
         }
      ],
      "node_name":"None",
      "node_selector":"None",
      "os":"None",
      "overhead":"None",
      "preemption_policy":"None",
      "priority":"None",
      "priority_class_name":"None",
      "readiness_gates":"None",
      "restart_policy":"OnFailure",
      "runtime_class_name":"None",
      "scheduler_name":"jupyterhub-stable-user-scheduler",
      "security_context":{
         "fsGroup":100
      },
      "service_account":"None",
      "service_account_name":"None",
      "set_hostname_as_fqdn":"None",
      "share_process_namespace":"None",
      "subdomain":"None",
      "termination_grace_period_seconds":"None",
      "tolerations":[
         {
            "effect":"NoSchedule",
            "key":"hub.jupyter.org/dedicated",
            "operator":"Equal",
            "toleration_seconds":"None",
            "value":"user"
         },
         {
            "effect":"NoSchedule",
            "key":"hub.jupyter.org_dedicated",
            "operator":"Equal",
            "toleration_seconds":"None",
            "value":"user"
         }
      ],
      "topology_spread_constraints":"None",
      "volumes":[
         {
            "aws_elastic_block_store":"None",
            "azure_disk":"None",
            "azure_file":"None",
            "cephfs":"None",
            "cinder":"None",
            "config_map":"None",
            "csi":"None",
            "downward_api":"None",
            "empty_dir":"None",
            "ephemeral":"None",
            "fc":"None",
            "flex_volume":"None",
            "flocker":"None",
            "gce_persistent_disk":"None",
            "git_repo":"None",
            "glusterfs":"None",
            "host_path":"None",
            "iscsi":"None",
            "name":"volume-appopenshift",
            "nfs":"None",
            "persistent_volume_claim":{
               "claimName":"claim-appopenshift"
            },
            "photon_persistent_disk":"None",
            "portworx_volume":"None",
            "projected":"None",
            "quobyte":"None",
            "rbd":"None",
            "scale_io":"None",
            "secret":"None",
            "storageos":"None",
            "vsphere_volume":"None"
         }
      ]
   },
   "status":"None"
}

This is the payload of the kubernetes response:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods \"jupyter-appopenshift\" is forbidden: failed quota: compute-resources: must specify limits.memory",
  "reason": "Forbidden",
  "details": {
    "name": "jupyter-appopenshift",
    "kind": "pods"
  },
  "code": 403
}

How to reproduce

  1. Create a ResourceQuota fit memory limit and apply it to the jupyterhub namespace.
  2. Log in into the hub
  3. Start the server
  4. See error

Your personal set up

Jupyterhub chart 2.0.0

welcome[bot] commented 1 year ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

consideRatio commented 1 year ago

You are in control of what memory limits etc you declare on the singleuser pods, those spawned by KubeSpawner. In the JupyterHub Helm chart, it will be configured under singleuser.memory, see https://z2jh.jupyter.org/en/stable/resources/reference.html#singleuser-memory.

If you have a ResourceQuota that enforces values to be set, or to be in a certain range etc, its a matter of configuration of KubeSpawner to work against your provided ResourceQuota I assume. Thinking that, I see no action point or something to fix in this repo and will go for a close.

lucianolacurcia commented 1 year ago

Thanks, the problem was my mistake, I was already setting the singleuser.memory.limit as the ResourceQuota enforces, but I was not setting on the network tools container, so that why the error was occurring.

https://stackoverflow.com/questions/62569161/error-creating-pods-is-forbiddenpod-failed-quota-namespace-must-specify

Thanks!