Closed MissakaI closed 6 months ago
I don't think that this is a bug in the client. If you look at the place where the exception was thrown:
It is setting response.message()
to be the message for the exception. For some reason the server that you are talking to is returning an empty status line in it's HTTP response. I think you will need to debug your server for more information.
You can try printing APIException.getCode()
or APIException.getResponseBody()
to get more details from the exception that may help with this debugging.
You can try printing
APIException.getCode()
orAPIException.getResponseBody()
to get more details from the exception that may help with this debugging.
Thanks for the tip on ability to print the response body using APIException.getResponseBody()
. So this is one of output I got when i printed the response body.
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"StatefulSet.apps \"selenium-standalone-firefox04699455\" is invalid: [spec.selector: Required value, spec.template.metadata.labels: Invalid value: map[string]string(nil): `selector` does not match template `labels`]","reason":"Invalid","details":{"name":"selenium-standalone-firefox04699455","group":"apps","kind":"StatefulSet","causes":[{"reason":"FieldValueRequired","message":"Required value","field":"spec.selector"},{"reason":"FieldValueInvalid","message":"Invalid value: map[string]string(nil): `selector` does not match template `labels`","field":"spec.template.metadata.labels"}]},"code":422}
In the response the message field is present. But still the ApiException
stacktrace doesn't print the message.
fwiw, the message
field in the Status response is unrelated to the message that is passed to the exception.
The message passed to the ApiException
is an okhttp.Response.message
so I still suspect that this is because the HTTP status message coming back from the server is empty.
If you look at the ApiException
code there's no way that the response body can get set w/o also passing in a message to the exception.
Can you try printing ApiException.getMessage()
explicitly and see what it prints?
As you suggested ApiException.getMessage()
function worked. It printed the returned message field in the previously attached response body.
We are struggling with similar scenario but we cannot call methods since errors occur during startup via Spring configuration for example.
We end up with simple io.kubernetes.client.openapi.ApiException:
with no info, for example could be for lack of permissions for the service account or API not available in some cloud provider/k8s version, but it's not possible to know. Would it make sense (even if under some config flag) to add the response body to the message so it's shown by default.
We are running into this problem as well - ApiException
is thrown with a null
message, and the inherited Exception.toString
method does not know about the responseBody
, so it does not get printed and all you see is ApiException: null
which is extremely unhelpful.
I run into the same thing today, and resorted to making a custom build of the client adding this to ApiException
(btw, kudos for how easy it was, mvn package and done!).
@Override
public String getMessage() {
String code = String.valueOf(getCode());
return code + getResponseBody();
}
With that I went from
2022-03-23 17:32:02.999 ERROR 1 --- [-lease-worker-1] i.k.c.e.leaderelection.LeaderElector : Error retrieving resource lock spring-cloud-gateway/configmap-reconciler
io.kubernetes.client.openapi.ApiException:
at io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:974) ~[na:na]
at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:886) ~[na:na]
at io.kubernetes.client.openapi.apis.CoordinationV1Api.readNamespacedLeaseWithHttpInfo(CoordinationV1Api.java:2743) ~[na:na]
to
2022-03-23 17:37:39.518 ERROR 1 --- [-lease-worker-1] i.k.c.e.leaderelection.LeaderElector : Error retrieving resource lock spring-cloud-gateway/configmap-reconciler
io.kubernetes.client.openapi.ApiException: 403{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"leases.coordination.k8s.io \"configmap-reconciler\" is forbidden: User \"system:serviceaccount:test:default\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"spring-cloud-gateway\"","reason":"Forbidden","details":{"name":"configmap-reconciler","group":"coordination.k8s.io","kind":"leases"},"code":403}
at io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:974) ~[client-java-api-13.0.3.jar:na]
at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:886) ~[client-java-api-13.0.3.jar:na]
I am not argueing this is trivial and should be done based on code lines count, we should probably sanitize the output and account for many more things. But I hope this exemplifies the struggles of not having any information when an error happens and you are not the one calling the API directly. Now I know I forgot to correctly setup the service account 🤦
Maybe we can start a conversation on a approach?
I run into the same issue too.
I can't get some important information about the problem, because the ApiException#message
is null when an exception occurs during leader election.
I create a CustomLeaderElector
class which copy from io.kubernetes.client.extended.leaderelection.LeaderElector
and overwrite tryAcquireOrRenew()
method to print more message of ApiException
, finally, I find my problem from the ResponseBody
.
public class CustomLeaderElector {
// ...
private boolean tryAcquireOrRenew() {
// ...
try {
oldLeaderElectionRecord = lock.get();
} catch (ApiException var6) {
// print more message of ApiException
log.error("tryAcquireOrRenew ApiException, code: {}, message: {}, header: {}, response: {}", var6.getCode(), var6.getMessage(), var6.getResponseHeaders(), var6.getResponseBody());
if (var6.getCode() != 404) {
log.error("Error retrieving resource lock {}", lock.describe(), var6);
return false;
}
}
}
Log:
[2022-06-16 17:56:05.464] [ERROR] [leader-elector-lease-worker-1] com.aliyun.actiontrail.client.CustomLeaderElector tryAcquireOrRenew:233 - tryAcquireOrRenew ApiException, code: 403, message: , header: {audit-id=[3316d6ae-1619-4676-8132-ecbbbacb0579], cache-control=[no-cache, private], content-length=[427], content-type=[application/json], date=[Thu, 16 Jun 2022 09:56:05 GMT], x-content-type-options=[nosniff], x-kubernetes-pf-flowschema-uid=[6373319d-163d-483f-98c5-e8b80c0cd018], x-kubernetes-pf-prioritylevel-uid=[8aadd4ee-cb09-40c3-9b4b-814d2b7d4b38]}, response: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"leases.coordination.k8s.io \"xxx\" is forbidden: User \"system:serviceaccount:xx-service:default\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"xx-service\"","reason":"Forbidden","details":{"name":"xxx,"group":"coordination.k8s.io","kind":"leases"},"code":403}
The ApiException#message
is null and the ApiException#responseBody#message
is not null, but we can't get responseBody
in stack trace. So I think it would be better to add correct error message
in ApiException#message
.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
fwiw, the
message
field in the Status response is unrelated to the message that is passed to the exception.
For some reason the Response#message()
being passed is proably null? It's supposed to be an string interpretation of the HTTP Status code based on https://square.github.io/okhttp/4.x/okhttp/okhttp3/-response/. Maybe we can inspect the response object prior to the ApiException
being code inside a debugger and isolate it into an OkHttp bug?
in practice, i liked setting ApiClient#setDebugging(true)
to workaround the lack of verbosity in ApiException
when trying to reproduce issues.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
Turns out, if there is no change, the issue doesn't just magically solve itself, no matter what the bot wants.
/remove-lifecycle stale
oops...
/remove-lifecycle rotten
fwiw, I did actually notice that there is a constructor where the response body can be set without the message getting set:
We may need an upstream fix in the client generator to fix this.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
We are still affected by this. What will it take to get some attention here? Serviceability is very important for having Kubernetes be a usable platform. I would be happy to file a PR, but it looked like the above fix was too surface level to be the right fix, and it is not likely that a community member will be able to step up and push through a fix to a code generator...
/remove-lifecycle stale
when create k8s apiclient, please try to set http protocol to http 1.1
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
/remove-lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/reopen
/remove-lifecycle rotten
Please send feedback to sig-contributor-experience at kubernetes/community.
/close not-planned
@k8s-triage-robot: Closing this issue, marking it as "Not Planned".
Describe the bug I am trying to create a statefulset using AppsV1Api and it throws a ApiException with no cause to identify what the issue is.
Compete Stacktrace as follows:
Client Version
14.0.0
Kubernetes Version
1.22.4
on Docker Desktop 4.3.1Java Version Java 11
To Reproduce
Expected behavior Either create the statefulset or throw an exception with proper message
KubeConfig If applicable, add a KubeConfig file with secrets redacted.
Server (please complete the following information):
Additional context Tried it with a incorrect attempt to create a pod using CoreV1Api which resulted in an exception with no cause as well. And with a correct config it creates the pod with no issues at all. So that proves the kubernetes api is reachable and has proper access priviledges.