Resourcequota pod subdomain bug

[janlauber]

What happened?

I recently tried to deploy the bitnami postgresql-ha helm chart into a vcluster. The helm values are as follows:

global:
  storageClass: notdefault-storageClass
  postgresql:
    existingSecret: postgresql-login
persistence:
  size: 10Gi
postgresql:
  resources:
    requests:
      cpu: "125m"
      memory: "256Mi"
pgpool:
  resources:
    requests:
      cpu: "125m"
      memory: "256Mi"

The hostnamespace has a resourcequota set with the following values:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: hostnamespace-resourcequota
spec:
  hard:
    cpu: "2"
    memory: 5Gi

But the postgresql statefulset pods stayed on pending. When I described them I saw the following error:

Events:
  Type     Reason     Age               From        Message
  ----     ------     ----              ----        -------
  Warning  SyncError  2s (x10 over 5s)  pod-syncer  Error syncing to physical cluster: pods "postgresql-ha-postgresql-0-x-default-x-vcluster" is forbidden: failed quota: hostnamespace-resourcequota: must specify cpu,memory

What did you expect to happen?

So I debugged the pod specs and found out that when you unset the following spec value, the pod will get deployed:

subdomain: <some-value>

It should be possible to deploy pods with the subdomain set.

How can we reproduce it (as minimally and precisely as possible)?

deploy the following resource quota in the host namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: test-resourcequota
spec:
  hard:
    cpu: "2"
    memory: 5Gi

deploy the following test pod in a vCluster:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test-pod
    image: ubuntu
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo hello; sleep 10;done"]
    resources:
      requests:
        memory: "1Gi"
        cpu: "500m"
  subdomain: test

It will be stuck in pending

Anything else we need to know?

No response

Host cluster Kubernetes version

```console $ kubectl version Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.2", GitCommit:"9d142434e3af351a628bffee3939e64c681afa4d", GitTreeState:"clean", BuildDate:"2022-01-19T17:27:51Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/arm64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5+rke2r1", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T17:22:22Z", GoVersion:"go1.16.10b7", Compiler:"gc", Platform:"linux/amd64"} ```

Host cluster Kubernetes distribution

``` v1.22.5+rke2r1 ```

vlcuster version

```console $ vcluster --version vcluster version 0.5.3 ```

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

``` k3s ```

OS and Arch

``` OS: Ubuntu 20.04.3 Arch: ? ```

[FabianKramm]

@janlauber thanks for creating this issue! Can you post the failing pod yaml here? I guess there will be a container inside this pod that does not specify resources, for example an init container. You can circumvent this problem by creating a limit range in the host namespace like this:

apiVersion: v1
kind: LimitRange
metadata:
  name: vcluster-limit-range
spec:
  limits:
  - default:
      cpu: 100m
      memory: 256Mi
    defaultRequest:
      cpu: 20m
      memory: 64Mi
    min:
      cpu: 10m
      memory: 32Mi
    type: Container

EDIT: looking at the bitnami chart there are definitely some init containers that do not have resources defined.

[janlauber]

Hey @FabianKramm I tested this again on a minikube cluster with the following manifest files Host minikube Cluster:

# hostcluster namespace
apiVersion: v1
kind: Namespace
metadata:
  name: test

# hostcluster namespace resourcequota
apiVersion: v1
kind: ResourceQuota
metadata:
  name: test-resourcequota
  namespace: test
spec:
  hard:
    cpu: "2"
    memory: 5Gi

# hostcluster namespace limit range
apiVersion: v1
kind: LimitRange
metadata:
  name: vcluster-limit-range
  namespace: test
spec:
  limits:
  - default:
      cpu: 100m
      memory: 256Mi
    defaultRequest:
      cpu: 20m
      memory: 64Mi
    min:
      cpu: 10m
      memory: 32Mi
    type: Container

Inside the vCluster:

# pod with set subdomain and resource requests for cpu and memory
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test-pod
    image: ubuntu
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo hello; sleep 10;done"]
    resources:
      requests:
        memory: "1Gi"
        cpu: "500m"
  subdomain: test

The following event occurs when you describe the pending test-pod:

Events:
  Type     Reason     Age                From        Message
  ----     ------     ----               ----        -------
  Warning  SyncError  2s (x13 over 23s)  pod-syncer  Error syncing to physical cluster: pods "test-pod-x-default-x-vcluster" is forbidden: failed quota: test-resourcequota: must specify cpu,memory

I think this is a general problem which occurs in vclusters.

[FabianKramm]

@janlauber it works for me with:

# pod with set subdomain and resource requests for cpu and memory
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
    - name: test-pod
      image: ubuntu
      command: ["/bin/sh"]
      args: ["-c", "while true; do echo hello; sleep 10;done"]
      resources:
        limits:
          memory: "1Gi"
          cpu: "500m"
        requests:
          memory: "1Gi"
          cpu: "500m"
  subdomain: test

[FabianKramm]

problem is that the default limit from the LimitRange is smaller than the requested resources from the pod which results in the error:

Error syncing to physical cluster: Pod "test-pod-x-default-x-vcluster" is invalid: [spec.containers[0].resources.requests: Invalid value: "500m": must be less than or equal to cpu limit, spec.containers[0].resources.requests: Invalid value: "1Gi": must be less than or equal to memory limit]

[janlauber]

@FabianKramm Thank you very much for this. Implemented this with postgresql-ha cluster and it's working! Thanks for your help, closing this one!