`sendInitialEvents` error

[swhoro]

code:

    WrappedEndpointsList endpointsList(CoreV1API_listNamespacedEndpoints(watchEndpointApiClient.get(),
                                                                         queryNamespace.data(),
                                                                         nullptr,
                                                                         0,
                                                                         nullptr,
                                                                         nullptr,
                                                                         nullptr,
                                                                         0,
                                                                         nullptr,
                                                                         nullptr,
                                                                         0,
                                                                         0,
                                                                         1),
                                       v1_endpoints_list_free);

return:

{
    "kind": "Status",
    "apiVersion": "v1",
    "metadata": {},
    "status": "Failure",
    "message": "ListOptions.meta.k8s.io \"\" is invalid: [resourceVersionMatch: Forbidden: sendInitialEvents requires setting resourceVersionMatch to NotOlderThan, sendInitialEvents: Forbidden: sendInitialEvents is forbidden for watch unless the WatchList feature gate is enabled]",
    "reason": "Invalid",
    "details": {
        "group": "meta.k8s.io",
        "kind": "ListOptions",
        "causes": [
            {
                "reason": "FieldValueForbidden",
                "message": "Forbidden: sendInitialEvents requires setting resourceVersionMatch to NotOlderThan",
                "field": "resourceVersionMatch"
            },
            {
                "reason": "FieldValueForbidden",
                "message": "Forbidden: sendInitialEvents is forbidden for watch unless the WatchList feature gate is enabled",
                "field": "sendInitialEvents"
            }
        ]
    },
    "code": 422
}

process_one_watch_event(): Cannot get type in watch event.
The return code of HTTP request=422

kubectl version:
Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.1

kubelete version:
Kubernetes v1.28.1

I have set sendInitialEvents to 0, why there is error related with it. I can run the code on minikube, but cannot run it on a real k8s cluster.

[ityuhui]

I guess the parameter sendInitialEvents is forbidden to be set to 0 when watch is on.

If you're sure you don't need sendInitialEvents, you can try preventing sending by removing the following code:

https://github.com/kubernetes-client/c/blob/5ac5ff25e9809a92a48111b1f77574b6d040b711/kubernetes/api/CoreV1API.c#L20985-L20995

[swhoro]

I guess the parameter sendInitialEvents is forbidden to be set to 0 when watch is on.

If you're sure you don't need sendInitialEvents, you can try preventing sending by removing the following code:

https://github.com/kubernetes-client/c/blob/5ac5ff25e9809a92a48111b1f77574b6d040b711/kubernetes/api/CoreV1API.c#L20985-L20995

Thank you. And I still have some questions:

1. why in my k8s cluster the code cannot run, but in minikube and another k8s cluster the code run nomally. Which k8s setting will affect this?
2. If I want to get all past endpoint as soon as the watch begin, what should I do? If I set sendInitialEvents to 1 and resourceVersionMatch to NotOlderThan (which is the error recommend), only the endpoint created after the watch will be notified.

[brendandburns]

It's very odd that it would work in minikube but not another cluster. Are all of the clusters the same version? It's possible that this error is tied to specific Kubernetes versions.

[swhoro]

It's very odd that it would work in minikube but not another cluster. Are all of the clusters the same version? It's possible that this error is tied to specific Kubernetes versions.

my k8s which cannot run:

kubectl version:
Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.1

kubelete version:
Kubernetes v1.28.1

minikube:

minikube version: v1.30.1
commit: 08896fd1dc362c097c925146c4a0d0dac715ace0

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:20:54Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:33:12Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}

the another k8s cluster that can run the code: in fact it is a k3s server, for some reason I've not been able to get to this cluster lately and I don't know it's version.

[ityuhui]

Minikube version is 1.26.3 for your code. The k8s version is 1.28.1 and cannot use your code.

Maybe we can release a new version of the C client for Kubernetes 1.28.x and try again.

I'm not sure the new client can work with your code, so if you need a quick solution, please consider removing sending sendInitialEvents or using generic client, see example.

[swhoro]

Minikube version is 1.26.3 for your code. The k8s version is 1.28.1 and cannot use your code.

Maybe we can release a new version of the C client for Kubernetes 1.28.x and try again.

I'm not sure the new client can work with your code, so if you need a quick solution, please consider removing sending sendInitialEvents or using generic client, see example.

How to watch resource with generic client? I only seee list, patch, delete, replace and create.

[swhoro]

Minikube version is 1.26.3 for your code. The k8s version is 1.28.1 and cannot use your code.

Maybe we can release a new version of the C client for Kubernetes 1.28.x and try again.

I'm not sure the new client can work with your code, so if you need a quick solution, please consider removing sending sendInitialEvents or using generic client, see example.

Removing sendInitialEvents works for me. So does it mean sendInitialEvent= Cannot appear in the param of request? But why some other clusters can run nomally?

[ityuhui]

Removing sendInitialEvents works for me. So does it mean sendInitialEvent= Cannot appear in the param of request? But why some other clusters can run nomally?

As @brendandburns said, it should be due to the different versions of Kubernetes API server. Removing sendInitialEvent= is a work-around for the Kubernetes 1.28 for the current C client. It's not an official fix.

Please try again after v0.8.0 released. https://github.com/kubernetes-client/c/issues/204

[ityuhui]

v0.8.0 released. Can you try with your code now ?

[swhoro]

v0.8.0 released. Can you try with your code now ?

Ok, I will try it this evening.

[swhoro]

v0.8.0 released. Can you try with your code now ?

It reports the same error:

Forbidden: sendInitialEvents requires setting resourceVersionMatch to NotOlderThan

I am not sure if it is related with featureGates. I install the cluster with kubeadm. By default, it does not enable the WatchList featureGate. So I edit the ClusterConfiguration file to enable WatchList:

apiServer:
  extraArgs:
    authorization-mode: Node,RBAC
    feature-gates: WatchList=true
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: cluster-endpoint:6443
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: v1.28.1
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}

And use kubeadm init phase control-plane apiserver --config t.yaml to generate new apiserver manifest and restart it.

Now I can use the WatchList featureGate but it still report the error above.

[swhoro]

 WrappedEndpointsList endpointsList(CoreV1API_listNamespacedEndpoints(watchEndpointApiClient.get(),
                                                                         queryNamespace.data(),
                                                                         nullptr,
                                                                         0,
                                                                         nullptr,
                                                                         nullptr,
                                                                         nullptr,
                                                                         0,
                                                                         "0",
                                                                         "NotOlderThan",
                                                                         1,
                                                                         0,
                                                                         1),
                                       v1_endpoints_list_free);

This code can achive what I want: no error, and report the endpoints that already exist.

[KagumiRA]

Hello, I have the same error when I try to run c/examples/list_pod/main.c , the program return The return code of HTTP request=422 Cannot get any pod. My k8s client version is "v1.27.3" and server version is "v1.27.0".

I guess the parameter sendInitialEvents is forbidden to be set to 0 when watch is on.

If you're sure you don't need sendInitialEvents, you can try preventing sending by removing the following code:

https://github.com/kubernetes-client/c/blob/5ac5ff25e9809a92a48111b1f77574b6d040b711/kubernetes/api/CoreV1API.c#L20985-L20995

I have tried it but still have the same error. How can i fixed this problem?

[ityuhui]

@KagumiRA

I think @swhoro has already resolved this issue via passing the following parameters:

resourceVersion: "0"
resourceVersionMatch: "NotOlderThan"

Can you log a new issue describing your problem in detail to avoid confusion with this issue ?