Resource quota of Clusterworkspaces prevents the creation of new workspaces due to exceeded quota

[wangke19]

version

$ kcp --version
kcp version v1.24.3+kcp-v0.9.1

$ k version -ojson | jq .serverVersion
{
  "major": "1",
  "minor": "24",
  "gitVersion": "v1.24.3+kcp-v0.9.1-12-gb8354c9b868524",
  "gitCommit": "b8354c9b",
  "gitTreeState": "clean",
  "buildDate": "2022-10-27T16:34:20Z",
  "goVersion": "go1.18",
  "compiler": "gc",
  "platform": "linux/amd64"
}

Describe the bug Resource quota of Clusterworkspaces doesn’t work as expected, preventing the creation of new workspaces, just can count workspaces we created before the quota was created.

To Reproduce

$ kubectl kcp workspace use '~'
Current workspace is "root:users:tw:sz:rh-sso-kewangkcp".

$ k get ws
No resources found

$ k ws create ke-ws01 --enter
Workspace "ke-ws01" (type root:universal) created. Waiting for it to be ready...
Workspace "ke-ws01" (type root:universal) is ready to use.
Current workspace is "root:users:tw:sz:rh-sso-kewangkcp:ke-ws01".

# Using Shared Compute provided by ACM,
...
apibinding.apis.kcp.dev/acm-kubernetes created

# Create one quota in namespace admin with the following yaml file,
$ cat adminquota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  annotations:
    experimental.quota.kcp.dev/cluster-scoped: "true"
  name: quota
spec:
  hard:
    count/configmaps: "3"
    count/namespaces: "2"
    count/secrets: "3"
    count/clusterworkspaces.tenancy.kcp.dev: "4"
    count/workspaces.tenancy.kcp.dev: "2"

$ k create ns admin
namespace/admin created

$ k create -f adminquota.yaml -n admin
resourcequota/quota created

$ k get quota -n admin
NAME    AGE REQUEST                                                                                                                                                 LIMIT
quota   171m   count/clusterworkspaces.tenancy.kcp.dev: 0/4, count/configmaps: 2/3, count/namespaces: 2/2, count/secrets: 2/3, count/workspaces.tenancy.kcp.dev: 0/2

# We can see the count of clusterworkspaces is zero, but when I tried to create new one workspace, that is fobidden by exceeded quota.

$ k ws .
Current workspace is "root:users:tw:sz:rh-sso-kewangkcp:ke-ws01".

$ k ws create ke-ws01-ws001
Error: clusterworkspaces.tenancy.kcp.dev "ke-ws01-ws001" is forbidden: exceeded quota: quota, requested: count/clusterworkspaces.tenancy.kcp.dev=1, used: count/clusterworkspaces.tenancy.kcp.dev=4, limited: count/clusterworkspaces.tenancy.kcp.dev=4

# We remove the count of clusterworkspaces.tenancy.kcp.dev from adminquota.yaml and apply again,

$ k apply -f adminquota.yaml -n admin;k get quota -n admin
resourcequota/quota configured
NAME    AGE REQUEST                                                                                                 LIMIT
quota   173m   count/configmaps: 2/3, count/namespaces: 2/2, count/secrets: 2/3, count/workspaces.tenancy.kcp.dev: 0/2

# Let's create new one workspace again,

$ k ws create ke-ws01-ws001
Workspace "ke-ws01-ws001" (type root:universal) created. Waiting for it to be ready...
Workspace "ke-ws01-ws001" (type root:universal) is ready to use.

# Add the count of clusterworkspaces.tenancy.kcp.dev of adminquota.yaml and apply again,
$ k apply -f adminquota.yaml -n admin;k get quota -n admin
resourcequota/quota configured
NAME    AGE REQUEST                                                                                                                                                 LIMIT
quota   175m   count/clusterworkspaces.tenancy.kcp.dev: 1/4, count/configmaps: 2/3, count/namespaces: 2/2, count/secrets: 2/3, count/workspaces.tenancy.kcp.dev: 0/2

# Something strange has happened，the count of clusterworkspaces.tenancy.kcp.dev increases by one.

# Let’s create another workspace to see what happens,
$ k ws create ke-ws01-ws002
Error: clusterworkspaces.tenancy.kcp.dev "ke-ws01-ws002" is forbidden: exceeded quota: quota, requested: count/clusterworkspaces.tenancy.kcp.dev=1, used: count/clusterworkspaces.tenancy.kcp.dev=4, limited: count/clusterworkspaces.tenancy.kcp.dev=4

# We can see the workspace before the resource quota creation can be counted,  after the resource quota creation will be  forbidden: exceeded quota. 

Expected Results Resource quota of Clusterworkspaces should count correctly, whatever before the resource quota creation or after.

[wangke19]

/assign @stevekuznetsov CC: @ncdc

[ncdc]

Might be a dupe of #2220?

[ncdc]

@wangke19 could you please retest on main, modifying your quota configuration to count workspaces.tenancy.kcp.io, and let us know if you still run into any issues? FYI, we no longer have clusterworkspaces - it's just workspaces now.

[wangke19]

@ncdc I refereed to README of https://github.com/kcp-dev/kcp/ and re-tested, below are steps, I cloned the https://github.com/kcp-dev/kcp.git to the local and enter the repo to exeucte make install with main branch, after that, $ kcp start &

$ export KUBECONFIG=~/.kcp/admin.kubeconfig

$ k version --short Flag --short has been deprecated, and will be removed in the future. The --short output will become the default. Client Version: v1.24.3 Kustomize Version: v4.5.4 Server Version: v1.24.3+kcp-v0.10.0-390-g4b36a8393e63f1

$ kcp --version kcp version v1.24.3+kcp-v0.10.0-390-g4b36a8393e63f1

$ k kcp workload sync kind --syncer-image ghcr.io/kcp-dev/kcp/syncer:main -o syncer-kind-main.yaml Creating synctarget "kind" Creating service account "kcp-syncer-kind-o04o95le"

$ export KUBECONFIG=~/.kube/config

$ kubectl apply -f "syncer-kind-main.yaml" namespace/kcp-syncer-kind-o04o95le created serviceaccount/kcp-syncer-kind-o04o95le created secret/kcp-syncer-kind-o04o95le-token created clusterrole.rbac.authorization.k8s.io/kcp-syncer-kind-o04o95le created clusterrolebinding.rbac.authorization.k8s.io/kcp-syncer-kind-o04o95le created role.rbac.authorization.k8s.io/kcp-dns-kind-o04o95le created rolebinding.rbac.authorization.k8s.io/kcp-dns-kind-o04o95le created secret/kcp-syncer-kind-o04o95le created deployment.apps/kcp-syncer-kind-o04o95le created

$ export KUBECONFIG=~/.kcp/admin.kubeconfig

$ k get ws NAME TYPE PHASE URL AGE compute universal https://192.168.9.66:6443/clusters/1ebmddue4a3szxjd 69s

$ k kcp workspace use '~' Current workspace is "kvdk2spgmbix".

$ k get ws No resources found

$ k ws create ke-ws01 --enter Workspace "ke-ws01" (type root:universal) created. Waiting for it to be ready... Workspace "ke-ws01" (type root:universal) is ready to use. Current workspace is "kvdk2spgmbix:ke-ws01" (type root:universal).

Created one quota config file, $ cat adminquota.yaml apiVersion: v1 kind: ResourceQuota metadata: annotations: experimental.quota.kcp.dev/cluster-scoped: "true" name: myquota spec: hard: count/configmaps: "3" count/namespaces: "2" count/secrets: "3" count/workspaces.tenancy.kcp.io: "4"

$ k create ns admin namespace/admin created

$ k create -f adminquota.yaml -n admin resourcequota/myquota created

$ k get quota -n admin NAME AGE REQUEST LIMIT myquota 10s count/configmaps: 1/3, count/namespaces: 0/2, count/secrets: 1/3, count/workspaces.tenancy.kcp.io: 0/4

Let's create some workspaces, $ k ws create ke-ws01-ws001 Workspace "ke-ws01-ws001" (type root:universal) created. Waiting for it to be ready... Workspace "ke-ws01-ws001" (type root:universal) is ready to use.

$ k ws create ke-ws01-ws002 Workspace "ke-ws01-ws002" (type root:universal) created. Waiting for it to be ready... Workspace "ke-ws01-ws002" (type root:universal) is ready to use.

$ k get quota -n admin NAME AGE REQUEST LIMIT myquota 50s count/configmaps: 1/3, count/namespaces: 0/2, count/secrets: 1/3, count/workspaces.tenancy.kcp.io: 2/4

We can see the workspace number is starting to count.

Let's delete one to see if count is correct. $ k delete ws ke-ws01-ws002 workspace.tenancy.kcp.io "ke-ws01-ws002" deleted

$ k get quota -n admin NAME AGE REQUEST LIMIT myquota 99s count/configmaps: 1/3, count/namespaces: 0/2, count/secrets: 1/3, count/workspaces.tenancy.kcp.io: 0/4

$ k get ws NAME TYPE PHASE URL AGE ke-ws01-ws001 universal https://192.168.9.66:6443/clusters/2nj1i2g6sbtze7gs 71s

Something wrong with workspaces counting when delete the workspace.

Let's recreate one we just deleted workspace, $ k ws create ke-ws01-ws002 Workspace "ke-ws01-ws002" (type root:universal) created. Waiting for it to be ready... Workspace "ke-ws01-ws002" (type root:universal) is ready to use.

$ k get quota -n admin NAME AGE REQUEST LIMIT myquota 6m20s count/configmaps: 1/3, count/namespaces: 0/2, count/secrets: 1/3, count/workspaces.tenancy.kcp.io: 1/4

$ k get ws NAME TYPE PHASE URL AGE ke-ws01-ws001 universal https://192.168.9.66:6443/clusters/2nj1i2g6sbtze7gs 6m28s ke-ws01-ws002 universal https://192.168.9.66:6443/clusters/dh0oe3zk3xx0buod 52s

We can see new one is counted, workspaces counting is still a problem. CC: @kasturinarra

[kasturinarra]

@wangke19 thanks, hope we have a test case for this and automated, if not can we try to add it and automate it so that when ever the bug is said to be fixed we could simply run our automation and also may be can add it to our basic sanity suite. WDYT ?

[wangke19]

@kasturinarra @zxiao-redhat has automated one quota case, bug not included workspace quota, zimo you can refer above steps add workspace quota in the case.

[kcp-ci-bot]

Issues go stale after 90d of inactivity. After a furter 30 days, they will turn rotten. Mark the issue as fresh with /remove-lifecycle stale.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[kcp-ci-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[kcp-ci-bot]

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

/close

[kcp-ci-bot]

@kcp-ci-bot: Closing this issue.

In response to [this](https://github.com/kcp-dev/kcp/issues/2314#issuecomment-2167531930): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.