Closed deejonz closed 7 months ago
adietish
commented
8 months ago copied over from https://github.com/redhat-developer/intellij-kubernetes/issues/717#issuecomment-2004261151
I tried three times and I can reproduce everytime. It seems that error is not coming with a specific element, but only after opening a certain amount of items.
adietish
commented
8 months ago copied over from https://github.com/redhat-developer/intellij-kubernetes/issues/717#issuecomment-2004347691
this last time 47 clicks, including opening items (like expanding pods one by one), I tried another time and it was 48.. very similar.
adietish
commented
8 months ago @deejonz: would it be possible for you to provide a redacted version of your configs in kube config so that we can try to reproduce this? It looks as if there's a problem with the OIDC authentication. The bug occurrs when the client library tries to refresh the token. We'd love to try to replicate this 😃
deejonz
commented
8 months ago @adietish sure, this is my config file, I hope this is what you were looking for:
---
apiVersion: "v1"
kind: "Config"
clusters:
- cluster:
certificate-authority: "certs/xxxxx-id/k8s-ca.crt"
server: "https://api.a-central-1.aws.xxxx.com"
name: "xxxxx-id"
contexts:
- context:
cluster: "xxxxx-id"
namespace: "id-stag"
user: "stefano-xxxxx-id"
name: "xxxxx-id"
current-context: "xxxxx-id"
preferences: {}
users:
- name: "stefano-xxxxx-id"
user:
auth-provider:
config:
client-id: "de-k8s-authenticator"
client-secret: "pUBnBOY8[...]ijwadxreNGQok"
id-token: "eyJhbGciOiJSUzI1NiIsImtpZCI6IjRjZmRiYjcx[...]eiuhedeaZmQifQ.eyJpc3MiOiJodHRwczovL2RleC5pZC5hd3MuY3[...]6IkNpUXhZV1l5TlRnd05TMWtZ[...]aGVudGljYXRvciIsImV4cCI6MTcxMDgxMzc3NSwiaWF0IjoxNzEwNzcwNTc1LCJhdF9oYXNoIjoiVHAwenM1RFBNRjI2WnNpOEI1cmdyUSIsImVtYWlsIjoic3RlZmFuby5icnVzYUB2b"
idp-issuer-url: "https://de.id.aws.xxxxx.com"
refresh-token: "Chl4eG0zZmhzd2aeiuhE[...]kaHRhN3h6bXlqZmtoYXdrY2Zt"
name: "oidc"@deejonz thanks for that snippet. According to @rohanKanojia this is related to your local certificates. Here's his question to you:
KubernetesClient seems to expect cluster certificate info either in idp-certificate-authority-data field in auth-provider config or cluster caCertData in OpenIDConnectionUtils. Probably in user's case both of them are null. How are certificates configured for the cluster?
deejonz
commented
8 months ago sorry but I don't know how this is configured in the organization.
adietish
commented
8 months ago @deejonz I think that @rohanKanojia is talking about the certificates that you have locally. I think that he's guessing from the stacktrace that the certificates is null and he is thus wondering if you can confirm/refute that the local certificates are all ok. If those weren't you should fail to talk to the cluster using kubectl once the token is out of validity and should be refreshed.
deejonz
commented
8 months ago if I do kubectl get po -n id-stag it returns the details as expected. Even when I'm getting the error on the intellij plugin, kubectl commandline works fine.
rohanKanojia
commented
8 months ago @deejonz : Could you please open an issue on Fabric8 Kubernetes Client (with the ~/.kube/config and location of certificate file)
adietish
commented
8 months ago I'm doing the issue
rohanKanojia
commented
8 months ago @deejonz : I see that your certificate file is a relative path certs/xxxxx-id/k8s-ca.crt . Am I right? What happens if you change it to absolute path?
adietish
commented
8 months ago @deejonz, @rohanKanojia: I created https://github.com/fabric8io/kubernetes-client/issues/5817
adietish
commented
8 months ago If confirmed that https://github.com/fabric8io/kubernetes-client/issues/4960 was the same problem then the fix would be as simple as upgrading our client-library 6.4.0 to >= 6.5.1, crossing fingers 😃
adietish
commented
8 months ago @deejonz I could try to make a binary build for you that you can test if you'd agree? Would take a bit longer though because of API breakages this bump may imply.
deejonz
commented
8 months ago sure @adietish, I can test that np.
rohanKanojia
commented
8 months ago @deejonz : Is it possible for you to run this reproducer project on your machine?
In https://github.com/fabric8io/kubernetes-client/issues/4960 we default to currentConfig.getCaCertData() if idp-certificate-authority-data is not provided. If
Could you please run mvn clean install after extracting the reproducer project to see if the certificate gets loaded into KubernetesClient config? If not, which attributes are loaded (I've added a print statement for Config in the test)
fabric8-oidc-config-certdata-reproducer.zip
deejonz
commented
8 months ago [ERROR] io.fabric8.reproducer.ConfigReadsCertTest.configLoadsCertData Time elapsed: 0.4 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: not <null>
at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:152)
at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertNotNull.failNull(AssertNotNull.java:49)
at org.junit.jupiter.api.AssertNotNull.assertNotNull(AssertNotNull.java:35)
at org.junit.jupiter.api.AssertNotNull.assertNotNull(AssertNotNull.java:30)
at org.junit.jupiter.api.Assertions.assertNotNull(Assertions.java:304)
at io.fabric8.reproducer.ConfigReadsCertTest.configLoadsCertData(ConfigReadsCertTest.java:20)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at java.util.ArrayList.forEach(ArrayList.java:1259)probably you needed this:
{"authProvider":{"config":{"client-id":"de-k8s-authenticator","client-secret":"pUBnBOY8[...]Y2xreNGQok","id-token":"eyJhbGciOiJSUzI1Ni[...]4cCI6MTcxMDg5OTg0MiwiaWF0IjoxNzEwODU2NjQyLCJhdF9oYXNoIjoiczZZVUxCazhGV0VxdmE4WVpTWlg2dyIsImVtYWlsI[...]Aj3NUExIYKwTsGcEZGiPLnNdyb5WUNbNgotnw","idp-issuer-url":"https://de.id.aws.xxxxx.com","refresh-token":"Chl4eG0zZ[...]pmcm5hZG8yeWZqYTN5"},"name":"oidc"},"maxConcurrentRequests":64,"maxConcurrentRequestsPerHost":5,"requestConfig":{"impersonateUsername":null,"impersonateGroups":[""],"impersonateExtras":{},"watchReconnectInterval":1000,"watchReconnectLimit":-1,"uploadRequestTimeout":120000,"requestRetryBackoffLimit":10,"requestRetryBackoffInterval":100,"requestTimeout":10000,"scaleTimeout":600000,"loggingInterval":20000},"contexts":[{"context":{"cluster":"xxxxx-id","namespace":"id-stag","user":"stefano-xxxxx-id"},"name":"xxxxx-id"},{"context":{"cluster":"docker-desktop","user":"docker-desktop"},"name":"docker-desktop"}],"currentContext":{"context":{"cluster":"xxxxx-id","namespace":"id-stag","user":"stefano.xxxxx-id"},"name":"xxxxx-id"},"onlyHttpWatches":false,"autoConfigure":true,"file":"/Users/deej/.kube/config","trustCerts":false,"disableHostnameVerification":false,"masterUrl":"https://api.k8s.eu-central-1.aws.xxxxx.com/","apiVersion":"v1","namespace":"id-stag","defaultNamespace":true,"caCertFile":"/Users/deej/.kube/certs/xxxxx-id/k8s-ca.crt","clientKeyPassphrase":"meeeee","websocketPingInterval":30000,"connectionTimeout":10000,"watchReconnectInterval":1000,"watchReconnectLimit":-1,"uploadRequestTimeout":120000,"requestRetryBackoffLimit":10,"requestRetryBackoffInterval":100,"requestTimeout":10000,"scaleTimeout":600000,"loggingInterval":20000,"impersonateGroups":[""],"impersonateExtras":{},"http2Disable":false,"noProxy":[],"userAgent":"fabric8-kubernetes-client/6.10.0","tlsVersions":["TLS_1_3","TLS_1_2"],"errorMessages":{}}@deejonz : oh, I see in your case caCertFile is set instead of caCertData.
I think we should update OpenIDConnectionUtils to consider both caCertData and caCertFile.
rohanKanojia
commented
8 months ago @deejonz : I have created a PR that might fix this issue. Is it possible for you to try it out and confirm if you still get NPE?
git clone https://github.com/rohanKanojia/kubernetes-client.git -b pr/openid-cert-data-or-filemvn clean install -DskipTestsfabric8.version property to point to SNAPSHOT in reproducer project
<fabric8.version>6.11-SNAPSHOT</fabric8.version>mvn clean install in reproducer project, the test just invokes OIDC refresh method. Do you still get the NPE? Or is it a different error?deejonz
commented
8 months ago @rohanKanojia
[ERROR] Errors:
[ERROR] OIDCTokenRefreshTest.resolveOIDCTokenFromAuthConfig:20 » IllegalArgument Illegal base64 character 2d@deejonz : Okay, so we're getting past NPE. I wasn't encoding the read cert file contents, this seems to throw exception when pemString is decoded later.
I've pushed an update to my branch. Is it possible for you to give it a try again :pray: ?
deejonz
commented
8 months ago sure, there you go:
[ERROR] Failures:
[ERROR] OIDCTokenRefreshTest.resolveOIDCTokenFromAuthConfig:23 expected: <true> but was: <false>@deejonz : Hmm, now we don't seem to get any exception but maybe token isn't getting refreshed :thinking: . Not sure whether it's due to some misconfiguration from our side or we still need to update something in KubernetesClient.
Load the reproducer project in IntelliJ as a maven project. Is it possible for you to set a breakpoint in OpenIDConnectionUtils#resolveOIDCTokenFromAuthConfig and observe what's happening?
deejonz
commented
8 months ago @rohanKanojia I tried to import in intellij but the project is not compiling for some reason.. so I cannot spent too much time on this. If you want to give me another version with more logging I'll run it from terminal
rohanKanojia
commented
8 months ago @deejonz : I have enabled trace logging in the reproducer, maybe we can get more insight from this.
deejonz
commented
8 months ago @rohanKanojia
[main] DEBUG io.fabric8.kubernetes.client.internal.CertUtils - The trailing entry generated a certificate exception. More than likely the contents end with comments.
java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input
at java.base/sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:115)
at java.base/java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:355)
at io.fabric8.kubernetes.client.internal.CertUtils.mergePemCertsIntoTrustStore(CertUtils.java:108)
at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:84)
at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:169)
at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.getDefaultHttpClientWithPemCert(OpenIDConnectionUtils.java:282)
at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.getOIDCProviderTokenEndpointAndRefreshToken(OpenIDConnectionUtils.java:317)
at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.resolveOIDCTokenFromAuthConfig(OpenIDConnectionUtils.java:96)
at io.fabric8.reproducer.OIDCTokenRefreshTest.resolveOIDCTokenFromAuthConfig(OIDCTokenRefreshTest.java:20)Load the reproducer project in IntelliJ as a maven project. Is it possible for you to set a breakpoint in OpenIDConnectionUtils#resolveOIDCTokenFromAuthConfig and observe what's happening?)
I tried again and I get this error from intellij:
java: java.lang.NoClassDefFoundError: io/fabric8/kubernetes/api/model/KubernetesResource
io.fabric8.kubernetes.api.model.KubernetesResource@deejonz : It looks like contents of the certificate file we provided are empty "certs/xxxxx-id/k8s-ca.crt". Could you please check if it is true?
rohanKanojia
commented
8 months ago @deejonz : Sorry, I didn't mean to open KubernetesClient project in IntelliJ. I meant opening the reproducer project, you should be able to access OpenIDConnectionUtils.class from there.
deejonz
commented
8 months ago I can see a certificate inside /Users/deej/.kube/certs/xxxxx-id/k8s-ca.crt
rohanKanojia
commented
8 months ago @deejonz : We need to debug why this is happening. Are you able to open reproducer project in IntelliJ? It shouldn't be that heavy.
deejonz
commented
8 months ago @rohanKanojia
return client.sendAsync(request, String.class).thenApply((response) -> { <---- Note1 below
try {
if (response.isSuccessful() && response.body() != null) { <---- it never enters here
return convertJsonStringToMap((String)response.body());
}
String responseBody = (String)response.body();
LOGGER.warn("oidc: failed to query metadata endpoint: {} {}", response.code(), responseBody);
} catch (Exception var2) {
LOGGER.warn("Could not refresh OIDC token, failure in getting refresh URL", var2);
}
return Collections.emptyMap();
});Note1: here it is calling the endpoint https://xxxxxx/.well-known/openid-configuration which gives:
if I call the endpoint manually from a browser I see the correct json returned.
rohanKanojia
commented
8 months ago That's strange. Not 100% sure but could this be due to default value of KubernetesClient timeout? kubernetes.request.timeout . Does it work after increasing timeout value?
deejonz
commented
8 months ago like this?
if so, then it's still failing in the same way.
rohanKanojia
commented
8 months ago Yes, I was referring to adding a maven <property> but this should work too.
deejonz
commented
8 months ago same with maven property:
...
<junit5.version>5.10.1</junit5.version>
<kubernetes.request.timeout>10000</kubernetes.request.timeout>
</properties>@deejonz : Would it be okay for you to share what you're doing over a short video call (preferably Google Meet) ? Otherwise, I need to contact my team lead to get an EKS cluster for testing (this can take time though)
deejonz
commented
8 months ago sure, we can do in around 4 hours from now.
rohanKanojia
commented
8 months ago @deejonz : Could you please share your email? I can send you a meeting invite.
adietish
commented
8 months ago @deejonz: The other possibility is that the thread pool in the plugin, that is watching resource kinds, is overhelmed. Whenever you expand a category (Pods, Jobs, Nodes, etc.) I create a new watch (-ing thread). I see that I currently use 20 threads:
Executors.newWorkStealingPool(20)Can you see the plugin starting to fail when there are 20 categories in the tree expanded? Btw. Collapsing a category terminates the watch and would free up the executor pool.
deejonz
commented
8 months ago I restarted intellij and opened 1 category only. Then after 1 hour I saw this error:
2024-03-22 16:25:00,218 [3483605] SEVERE - io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager - Exception in reconnect
java.lang.NullPointerException: Cannot invoke "String.getBytes(java.nio.charset.Charset)" because "src" is null
at java.base/java.util.Base64$Decoder.decode(Base64.java:589)
at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.getDefaultHttpClientWithPemCert(OpenIDConnectionUtils.java:292)
at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.getOIDCProviderTokenEndpointAndRefreshToken(OpenIDConnectionUtils.java:330)
at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.resolveOIDCTokenFromAuthConfig(OpenIDConnectionUtils.java:86)
at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.extractNewAccessTokenFrom(TokenRefreshInterceptor.java:83)
at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.refreshToken(TokenRefreshInterceptor.java:76)
at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.before(TokenRefreshInterceptor.java:58)
at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$buildWebSocket$4(StandardHttpClient.java:124)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at io.fabric8.kubernetes.client.http.StandardHttpClient.buildWebSocket(StandardHttpClient.java:124)
at io.fabric8.kubernetes.client.http.StandardWebSocketBuilder.buildAsync(StandardWebSocketBuilder.java:43)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.start(WatchConnectionManager.java:113)
at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.startWatch(AbstractWatchManager.java:221)
at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.reconnect(AbstractWatchManager.java:150)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)then after another hour, when I tried to use the plugin I've got this:
2024-03-22 16:49:37,916 [4961303] WARN - #com.redhat.devtools.intellij.kubernetes.model.ResourceWatch - Could not watch resource(s) ResourceKind(version=v1, clazz=class io.fabric8.kubernetes.api.model.Pod, kind=Pod).
java.lang.NullPointerException: Cannot invoke "String.getBytes(java.nio.charset.Charset)" because "src" is null
at java.base/java.util.Base64$Decoder.decode(Base64.java:589)
at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.getDefaultHttpClientWithPemCert(OpenIDConnectionUtils.java:292)
at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.getOIDCProviderTokenEndpointAndRefreshToken(OpenIDConnectionUtils.java:330)
at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.resolveOIDCTokenFromAuthConfig(OpenIDConnectionUtils.java:86)
at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.extractNewAccessTokenFrom(TokenRefreshInterceptor.java:83)
at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.refreshToken(TokenRefreshInterceptor.java:76)
at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.before(TokenRefreshInterceptor.java:58)
at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$buildWebSocket$4(StandardHttpClient.java:124)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at io.fabric8.kubernetes.client.http.StandardHttpClient.buildWebSocket(StandardHttpClient.java:124)
at io.fabric8.kubernetes.client.http.StandardWebSocketBuilder.buildAsync(StandardWebSocketBuilder.java:43)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.start(WatchConnectionManager.java:113)
at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.startWatch(AbstractWatchManager.java:221)
at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.<init>(AbstractWatchManager.java:87)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.<init>(WatchConnectionManager.java:74)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.<init>(WatchConnectionManager.java:83)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.submitWatch(BaseOperation.java:635)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.watch(BaseOperation.java:617)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.watch(BaseOperation.java:605)
at com.redhat.devtools.intellij.kubernetes.model.resource.NamespacedResourceOperator.watchAll(NamespacedResourceOperator.kt:82)
at com.redhat.devtools.intellij.kubernetes.model.context.ActiveContext$watch$1.invoke(ActiveContext.kt:296)
at com.redhat.devtools.intellij.kubernetes.model.context.ActiveContext$watch$1.invoke(ActiveContext.kt:296)
at com.redhat.devtools.intellij.kubernetes.model.ResourceWatch$WatchOperation.run(ResourceWatch.kt:141)
at com.redhat.devtools.intellij.kubernetes.model.ResourceWatch$WatchOperationsRunner.run(ResourceWatch.kt:125)
at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1375)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
2024-03-22 16:49:37,916 [4961303] WARN - #com.redhat.devtools.intellij.kubernetes.tree.TreeStructure - Cannot invoke "String.getBytes(java.nio.charset.Charset)" because "src" is null
java.lang.NullPointerException: Cannot invoke "String.getBytes(java.nio.charset.Charset)" because "src" is null
at java.base/java.util.Base64$Decoder.decode(Base64.java:589)
at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.getDefaultHttpClientWithPemCert(OpenIDConnectionUtils.java:292)
at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.getOIDCProviderTokenEndpointAndRefreshToken(OpenIDConnectionUtils.java:330)
at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.resolveOIDCTokenFromAuthConfig(OpenIDConnectionUtils.java:86)
at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.extractNewAccessTokenFrom(TokenRefreshInterceptor.java:83)
at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.refreshToken(TokenRefreshInterceptor.java:76)
at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.before(TokenRefreshInterceptor.java:58)
at io.fabric8.kubernetes.client.http.StandardHttpClient.consumeBytes(StandardHttpClient.java:65)
at io.fabric8.kubernetes.client.http.SendAsyncUtils.bytes(SendAsyncUtils.java:51)
at io.fabric8.kubernetes.client.http.HttpResponse$SupportedResponses.sendAsync(HttpResponse.java:105)
at io.fabric8.kubernetes.client.http.StandardHttpClient.sendAsync(StandardHttpClient.java:52)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.retryWithExponentialBackoff(OperationSupport.java:604)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:581)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.submitList(BaseOperation.java:414)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:427)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:392)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:93)
at com.redhat.devtools.intellij.kubernetes.model.resource.NamespacedResourceOperator.loadAllResources(NamespacedResourceOperator.kt:68)
at com.redhat.devtools.intellij.kubernetes.model.resource.NamespacedResourceOperator.getAllResources(NamespacedResourceOperator.kt:54)
at com.redhat.devtools.intellij.kubernetes.model.resource.NamespacedResourceOperator.getAllResources(NamespacedResourceOperator.kt:36)
at com.redhat.devtools.intellij.kubernetes.model.context.ActiveContext.getAllResources(ActiveContext.kt:148)
at com.redhat.devtools.intellij.kubernetes.model.ResourceModel.getAllResources(ResourceModel.kt:122)
at com.redhat.devtools.intellij.kubernetes.model.ListableResources.list(ResourceModelQuery.kt:39)
at com.redhat.devtools.intellij.kubernetes.tree.KubernetesStructure$createWorkloadElements$9$3.invoke(KubernetesStructure.kt:256)
at com.redhat.devtools.intellij.kubernetes.tree.KubernetesStructure$createWorkloadElements$9$3.invoke(KubernetesStructure.kt:252)
at com.redhat.devtools.intellij.kubernetes.tree.AbstractTreeStructureContribution$ElementNode.getChildElements(AbstractTreeStructureContribution.kt:97)
at com.redhat.devtools.intellij.kubernetes.tree.AbstractTreeStructureContribution.getChildElements(AbstractTreeStructureContribution.kt:28)
at com.redhat.devtools.intellij.kubernetes.tree.TreeStructure.getChildElements(TreeStructure.kt:71)
at com.redhat.devtools.intellij.kubernetes.tree.TreeStructure.getChildElements(TreeStructure.kt:64)
at com.intellij.ui.tree.StructureTreeModel.getValidChildren(StructureTreeModel.java:411)
at com.intellij.ui.tree.StructureTreeModel.validateChildren(StructureTreeModel.java:329)
at com.intellij.ui.tree.StructureTreeModel.getNode(StructureTreeModel.java:323)
at com.intellij.ui.tree.StructureTreeModel.getChildren(StructureTreeModel.java:343)
at com.intellij.ui.tree.AsyncTreeModel$CmdGetChildren.computeNode(AsyncTreeModel.java:613)
at com.intellij.ui.tree.AsyncTreeModel$Command.computeNode(AsyncTreeModel.java:489)
at com.intellij.util.concurrency.Invoker$Task.run(Invoker.java:381)
at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$1(CoreProgressManager.java:192)
at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$executeProcessUnderProgress$12(CoreProgressManager.java:610)
at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:685)
at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:641)
at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:609)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:78)
at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:179)
at com.intellij.util.concurrency.Invoker.startTask(Invoker.java:236)
at com.intellij.util.concurrency.Invoker.invokeSafely(Invoker.java:194)
at com.intellij.util.concurrency.Invoker.lambda$offerSafely$0(Invoker.java:177)
at com.intellij.util.concurrency.Invoker$Background.lambda$offer$0(Invoker.java:508)
at com.intellij.util.concurrency.BoundedTaskExecutor.doRun(BoundedTaskExecutor.java:244)
at com.intellij.util.concurrency.BoundedTaskExecutor.access$200(BoundedTaskExecutor.java:30)
at com.intellij.util.concurrency.BoundedTaskExecutor$1.executeFirstTaskAndHelpQueue(BoundedTaskExecutor.java:222)
at com.intellij.util.ConcurrencyUtil.runUnderThreadName(ConcurrencyUtil.java:218)
at com.intellij.util.concurrency.BoundedTaskExecutor$1.run(BoundedTaskExecutor.java:210)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:702)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:699)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:699)
at java.base/java.lang.Thread.run(Thread.java:840)regarding 20 categories, no, I don't see any relation
adietish
commented
8 months ago ok, we need to have an OIDC enabled EKS cluster and try to reproduce this then. @rohanKanojia and team is working on it. I need to configure OIDC on mine.
adietish
commented
8 months ago @deejonz: I "kinda" can replicate it. I have an EKS cluster with keycloak OIDC (@sabre1041 set it up for me, kudos!). I created some deployment, fiddled around an had all the tree items erroring after a few minuntes:
java.lang.NullPointerException
at java.base/java.util.Base64$Decoder.decode(Base64.java:561)
at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.getDefaultHttpClientWithPemCert(OpenIDConnectionUtils.java:292)
at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.getOIDCProviderTokenEndpointAndRefreshToken(OpenIDConnectionUtils.java:330)
at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.resolveOIDCTokenFromAuthConfig(OpenIDConnectionUtils.java:86)
at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.extractNewAccessTokenFrom(TokenRefreshInterceptor.java:83)
at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.refreshToken(TokenRefreshInterceptor.java:76)
at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.before(TokenRefreshInterceptor.java:58)
at io.fabric8.kubernetes.client.http.StandardHttpClient.consumeBytes(StandardHttpClient.java:65)
at io.fabric8.kubernetes.client.http.SendAsyncUtils.bytes(SendAsyncUtils.java:51)
at io.fabric8.kubernetes.client.http.HttpResponse$SupportedResponses.sendAsync(HttpResponse.java:105)
at io.fabric8.kubernetes.client.http.StandardHttpClient.sendAsync(StandardHttpClient.java:52)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.retryWithExponentialBackoff(OperationSupport.java:604)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:581)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.submitList(BaseOperation.java:414)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:427)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:392)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:93)
at com.redhat.devtools.intellij.kubernetes.model.resource.NamespacedResourceOperator.loadAllResources(NamespacedResourceOperator.kt:68)
at com.redhat.devtools.intellij.kubernetes.model.resource.NamespacedResourceOperator.getAllResources(NamespacedResourceOperator.kt:54)
at com.redhat.devtools.intellij.kubernetes.model.resource.NamespacedResourceOperator.getAllResources(NamespacedResourceOperator.kt:36)
at com.redhat.devtools.intellij.kubernetes.model.context.ActiveContext.getAllResources(ActiveContext.kt:148)
at com.redhat.devtools.intellij.kubernetes.model.ResourceModel.getAllResources(ResourceModel.kt:122)
at com.redhat.devtools.intellij.kubernetes.model.ResourceModel.getAllResources$default(ResourceModel.kt:121)
at com.redhat.devtools.intellij.kubernetes.model.FilterableResources.list(ResourceModelQuery.kt:63)
at com.redhat.devtools.intellij.kubernetes.tree.KubernetesStructure$createWorkloadElements$14$3.invoke(KubernetesStructure.kt:307)
at com.redhat.devtools.intellij.kubernetes.tree.KubernetesStructure$createWorkloadElements$14$3.invoke(KubernetesStructure.kt:304)
at com.redhat.devtools.intellij.kubernetes.tree.AbstractTreeStructureContribution$ElementNode.getChildElements(AbstractTreeStructureContribution.kt:97)
at com.redhat.devtools.intellij.kubernetes.tree.AbstractTreeStructureContribution.getChildElements(AbstractTreeStructureContribution.kt:28)
at com.redhat.devtools.intellij.kubernetes.tree.TreeStructure.getChildElements(TreeStructure.kt:71)
at com.redhat.devtools.intellij.kubernetes.tree.TreeStructure.getChildElements(TreeStructure.kt:64)
at com.intellij.ui.tree.StructureTreeModel.getValidChildren(StructureTreeModel.java:383)
at com.intellij.ui.tree.StructureTreeModel.validateChildren(StructureTreeModel.java:299)
at com.intellij.ui.tree.StructureTreeModel.getNode(StructureTreeModel.java:293)
at com.intellij.ui.tree.StructureTreeModel.getChildren(StructureTreeModel.java:313)
at com.intellij.ui.tree.AsyncTreeModel$CmdGetChildren.getNode(AsyncTreeModel.java:545)
at com.intellij.ui.tree.AsyncTreeModel$Command.get(AsyncTreeModel.java:440)
at com.intellij.ui.tree.AsyncTreeModel$Command.get(AsyncTreeModel.java:406)
at com.intellij.util.concurrency.Invoker$Task.run(Invoker.java:314)
at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:189)
at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$executeProcessUnderProgress$12(CoreProgressManager.java:608)
at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:683)
at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:639)
at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:607)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:60)
at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:176)
at com.intellij.util.concurrency.Invoker.invokeSafely(Invoker.java:201)
at com.intellij.util.concurrency.Invoker.lambda$offerSafely$0(Invoker.java:181)
at com.intellij.util.concurrency.Invoker$Background.lambda$offer$0(Invoker.java:481)
at com.intellij.util.concurrency.BoundedTaskExecutor.doRun(BoundedTaskExecutor.java:241)
at com.intellij.util.concurrency.BoundedTaskExecutor.access$200(BoundedTaskExecutor.java:31)
at com.intellij.util.concurrency.BoundedTaskExecutor$1.execute(BoundedTaskExecutor.java:214)
at com.intellij.util.ConcurrencyUtil.runUnderThreadName(ConcurrencyUtil.java:212)
at com.intellij.util.concurrency.BoundedTaskExecutor$1.run(BoundedTaskExecutor.java:203)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665)
at java.base/java.lang.Thread.run(Thread.java:829)I'll now try with the updated client that i should build manually.
adietish
commented
8 months ago @deejonz: Using the fixed client mentioned in https://github.com/redhat-developer/intellij-kubernetes/issues/726#issuecomment-2007880915 I dont face the issue any more. What about me making you a binary build of the plugin and kindly asking you to test it?
ps. I found other non-related issues though. Refresh causes the plugin to break. Changing the current namespace also breaks it. Will file those.
deejonz
commented
8 months ago sure np I can do it, but after tuesday. Thanks for your effort.
adietish
commented
8 months ago @deejonz awesomeness. Thanks for reporting and testing, highly appreciated.
adietish
commented
8 months ago for my own documentation, here's how to set up the whole thing:
ASSERT: Have a keycloak service running.
EXEC: add it as OIDC identity provider in your EKS cluster
EXEC: in bash, query keycloak for refresh-token and id-token:
curl \
-d "grant_type=password" \
-d "scope=openid" \
-d "client_id=kubernetes" \
-d "client_secret=<shared secret> \
-d "username=<myuser>" \
-d "password=<mypassword>" \
https://<keycloak-host>/auth/realms/eks/protocol/openid-connect/token | jq .you get the following output:
{
"access_token": "eyJhbGciOiJSUzI1...0zojav4wbx3gg",
"expires_in": 300,
"refresh_expires_in": 1800,
"refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCIgOiAi...P2W0Gy6VTiQHD1fLRJSDDmHm0",
"token_type": "Bearer",
"id_token": "eyJhbGciOiJSUzI1NiIsInR5...DxDnqo31mVmmBMhn11w",
"not-before-policy": 0,
"session_state": "44d1f4...bb3ffaa7",
"scope": "openid email profile"
}kubectl issue the following command using refresh_token and id_token given in the former output to update the kube config (creates/updates an 'eks' context):
kubectl config set-credentials eks \
"--auth-provider=oidc" \
"--auth-provider-arg=idp-issuer-url=https://<keycloak-host>/auth/realms/eks" \
"--auth-provider-arg=client-id=kubernetes" \
"--auth-provider-arg=client-secret=<same shared-secret>" \
"--auth-provider-arg=refresh-token=<refresh-token>" \
"--auth-provider-arg=id-token=<id-token>"Good morning @adietish can I have the binary build to try?
I get the following stacktrace when browsing the kubernates tree in many places, I can see the pods item well BTW: