kubectl apply's get /namespaces/{} output is disregarded

mfranzil commented 5 days ago

What happened?

Every time kubectl apply is used, instead of the direct API call, kubectl automatically evaluates the correct command to be used (e.g., a patch for an existent object, a create for a new one). To do so, usually kubectl runs the following commands (assume I am manipulating a ServiceAccount for simplicity)

GET to /openapi/v3/ and /openapi/v3/api/{etc..}
GET /api/v1/namespaces/{namespace}/serviceaccounts/{my_sa} - here the decision is taken whether to call additional APIs; if so,
GET /api/v1/namespaces/{namespace}/
POST/PATCH/etc...

The problem is, the GET to the namespace object is completely irrelevant to the calls made after: even if the API call reports a 403, or 404 (extreme corner case in which the namespace was deleted in the middle), kubectl mindlessly goes forward with the following call. This leads me to think that this GET to the namespace is irrelevant, and could be skipped.

What did you expect to happen?

Either the GET call not to be present or to have some sort of role in the apply process.

How can we reproduce it (as minimally and precisely as possible)?

Create a ClusterRole that can get and create serviceaccounts, but has no access to namespaces.
Bind the ClusterRole to any kind of authenticated User/SA of your choice.
Try to create a ServiceAccount with kubectl apply, put a non-existent name and an existent namespace:

$ kubectl --v=6 apply -f - <<EOF                                                                                                                                                                                                         
apiVersion: v1
kind: ServiceAccount
metadata:
  name: non-existent-sa
  namespace: default
EOF

Inspect the API calls, you will find a GET https://[redacted]:6443/api/v1/namespaces/default 403 Forbidden. Yet, the POST goes forward.
Even without a clusterrole, try creating a ServiceAccount on a non-existent namespace:

$ kubectl --v=6 apply -f - <<EOF                                                                                                                                                                                                         
apiVersion: v1
kind: ServiceAccount
metadata:
  name: non-existent-sa
  namespace: your-sa
EOF

Here are the API calls, in which even after the 404 on the your-sa namespace apply goes forward:

I0704 13:17:56.795130   49554 round_trippers.go:553] GET https://[redacted]:6443/openapi/v3?timeout=32s 200 OK in 20 milliseconds
I0704 13:17:56.800759   49554 round_trippers.go:553] GET https://[redacted]:6443/openapi/v3/api/v1?hash=[redacted]&timeout=32s 200 OK in 3 milliseconds
I0704 13:17:56.834894   49554 round_trippers.go:553] GET https://[redacted]:6443/api/v1/namespaces/your-sa/serviceaccounts/non-existent-sa 404 Not Found in 5 milliseconds
I0704 13:17:56.840224   49554 round_trippers.go:553] GET https://[redacted]:6443/api/v1/namespaces/your-sa 404 Not Found in 5 milliseconds
I0704 13:17:56.900061   49554 round_trippers.go:553] POST https://[redacted]:6443/api/v1/namespaces/your-sa/serviceaccounts?fieldManager=kubectl-client-side-apply&fieldValidation=Strict 404 Not Found in 59 milliseconds

Anything else we need to know?

No response

Kubernetes version

```console $ k version Client Version: v1.28.6 Server Version: v1.28.7 ```

Cloud provider

None (self-managed kubeadm instance)

OS version

```console $ cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.4 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.4 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy $ uname -a Linux kubeadm-master 5.15.0-107-generic kubernetes/kubernetes#117-Ubuntu SMP Fri Apr 26 12:26:49 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux ```

Install tools

None

Container runtime (CRI) and version (if applicable)

```console $ sudo ctr version [...] Server: Version: 1.7.18 Revision: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e UUID: bbc7ef22-56e4-4989-8119-897b2d6efe00 ```

Related plugins (CNI, CSI, ...) and versions (if applicable)

Flannel is used, but is irrelevant.

k8s-ci-robot commented 5 days ago

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

mfranzil commented 5 days ago

/sig api-machinery

sftim commented 4 days ago

/sig cli /remove-sig api-machinery

sftim commented 4 days ago

This might be a client-go thing but I think kubectl should triage it first. /transfer kubectl

ardaguclu commented 11 hours ago

Did you test the same steps with different resource (e.g. pods, etc.) other than serviceaccounts?. Because there might be an explicitly defined behavior specifically for serviceaccounts. If this happens other resources as well, I think it requires investigation.

kubernetes / kubectl