After upgrading Springboot to 3.2.2 version and fabric8io<openshift-client> to 6.9.2 version getting -> Received 403 on websocket on Pod watcher

[sachinkaushik]

Describe the bug

We have upgraded our service to springboot 3.2.2 version and also upgraded fabric8 openshiftclient version to 6.9.2. After upgrading same we are getting issue with Pod Watcher.

Received 403 on websocket

exception in pod watching: Received 403 on websocket. Failure executing: GET at: https:///api/v1/pods?allowWatchBookmarks=true&labelSelector=platform%3Ddev&watch=true. Message: Forbidden.

Fabric8 Kubernetes Client version

6.9.2

Steps to reproduce

Using below dep in pom.xaml

io.fabric8 openshift-client 6.9.2

Created below bean to connect to opesnhiftclient.

@Bean public OpenShiftClient getOpenShiftClient() {

    OpenShiftClient openShiftClient = new KubernetesClientBuilder()
            .withConfig(new OpenShiftConfigBuilder()
                    .withMasterUrl(Url)
                    .withUsername(UserName)
                    .withPassword(Password).build()).build().adapt(OpenShiftClient.class);

    return openShiftClient;
}   

Code to watch pod events:

Watch watch = client.pods().inAnyNamespace().withLabel(labelKey, envName).watch(----)   

Expected behavior

it should not throw below exception, watcher should be able to get all pod events

exception in pod watching: Received 403 on websocket. Failure executing: GET at: https:///api/v1/pods?allowWatchBookmarks=true&labelSelector=platform%3Ddev&watch=true. Message: Forbidden.

Runtime

OpenShift

Kubernetes API Server version

1.25.3@latest

Environment

other (please specify in additional context)

Fabric8 Kubernetes Client Logs

No response

Additional context

No response

[manusa]

It seems that your authentication configuration is not working with the cluster.

However, with the current information is hard to know what has changed. Have you tried updating the components (Spring and Fabric8) separately?

Also, have you checked with more verbose logging to see if the authentication headers are properly sent? (e.g. -Dorg.slf4j.simpleLogger.log.io.fabric8=trace or manually adding the HttpLoggingInterceptor)

[sachinkaushik]

@manusa yes, we have upgraded version of springboot from 3.1.6 to 3.2.2 and openshift-client version 6.2.0 to 6.9.2 . and we have started seeing mentioned error.

it was working fine with springboot 3.1.6 and openshift-client 6.2.0 version.

[manusa]

As I said, this is not enough information.

• Does it work with Spring Boot 3.2.2 and Fabric8 6.2.0?
• Does it work with Spring Boot 3.1.6 and Fabric8 6.9.2?
• What does the trace look like? (e.g. -Dorg.slf4j.simpleLogger.log.io.fabric8=trace or manually adding the HttpLoggingInterceptor)

[aidanleuck]

I am seeing something very similar where I am getting unexpected 403 errors when hitting the events API. (Not using Springboot) I have checked with kubectl and the service account does have permissions to list the event API. I will turn on trace logging tomorrow to see if that gives us any more useful information.

Block of code var result = client.events().v1().events().inNamespace(namespace).list();

Failure executing: GET at: 
https://172.17.0.1:6443/apis/events.k8s.io/v1/namespaces/default/events. Message: 
events.events.k8s.io is forbidden: User "system:serviceaccount:jenkins:jenkins" cannot 
list resource "events" in API group "events.k8s.io" in the namespace "default". 
Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], 
group=events.k8s.io, kind=events, name=null, retryAfterSeconds=null, uid=null, 
additionalProperties={}), kind=Status, message=events.events.k8s.io is forbidden: 
User "system:serviceaccount:jenkins:jenkins" cannot list resource "events" in API 
group "events.k8s.io" in the namespace "default", 
metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, 
selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure,
additionalProperties={})

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!

[noelvillador]

Hello,

i have the same issue. I upgraded to springboot 3.3.3. I’m getting 403 when using openshift client 6.9.2 but it is working in openshift client 6.2.0 and springboot 3.3.3

using username and password in configbuilder

[manusa]

Hello,

i have the same issue. I upgraded to springboot 3.3.3. I’m getting 403 when using openshift client 6.9.2 but it is working in openshift client 6.2.0 and springboot 3.3.3

using username and password in configbuilder

It's difficult for us to know what might be happening in your case, you need to provide more details:

See https://github.com/fabric8io/kubernetes-client/issues/5768#issuecomment-1968295211

Have you tried updating components independently? is the cluster version constant? and so on.

[noelvillador]

Hello,

I have tried updating the component independently and it's the same result. The cluster version is constant. It seems that from the exception logs, it is throwing a 403 error using a service account. But we are connecting to the openshift with the specific account, using ConfigBuilder

return new ConfigBuilder() .withMasterUrl(url) .withUsername(username) .withPassword(password) .build();

Failure executing: GET at: https://cluster/api/v1/namespaces/my-namespace/pods?labelSelector=pod-name. Message: pods is forbidden: User "system:serviceaccount:user:default" cannot list resource "pods" in API group "" in the namespace "my-namespace". Received status: Status(apiVersion=v1, code=403, kind=Status, message=pods is forbidden: User "system:serviceaccount:user:default" cannot list resource "pods" in API group "" in the namespace "my-namespace".

[manusa]

OK, this brings some light, so the configured user is not the one used by the client. Is this happening with 6.9.1? it would make it much easier if we new which version introduced the bug.

@shawkins does this ring a bell? I remember we did change some stuff with interceptors and overall OpenShift auth procedures.

[noelvillador]

I tried it with 2 version, latest 6.13.4 and 6.9.2 both of them are failing. but on 6.2.0 it is working fine

[noelvillador]

after checking the versions, on 6.4.1 the error is not there. on 6.5.0 i'm seeing a forbidden error

Exec Failure: HTTP 403, Status: 403 - Forbidden log.logger:o.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager

then 6.5.1 to latest same as the error on my previous post