Unable to interact with cluster through Teleport

nicolas-goudry commented 8 months ago

Describe the bug

Using a cluster with authn via Teleport, it is not possible to interact with the cluster.

This is working fine with kubectl and after testing with the Python client it works too. This seems to only happen with the Java client (I didn’t test other clients though).

Client Version

20

Kubernetes Version

1.28.6

Java Version

17

To Reproduce

configure your cluster with Teleport
generate a kubeconfig through tsh

Edit the KubeConfigFileClientExample:

diff --git a/examples/examples-release-20/src/main/java/io/kubernetes/client/examples/KubeConfigFileClientExample.java b/examples/examples-release-20/src/main/java/io/kubernetes/client/examples/KubeConfigFileClientExample.java
index 7a0ba4dcf..99a7cc75e 100644
--- a/examples/examples-release-20/src/main/java/io/kubernetes/client/examples/KubeConfigFileClientExample.java
+++ b/examples/examples-release-20/src/main/java/io/kubernetes/client/examples/KubeConfigFileClientExample.java
@@ -16,12 +16,13 @@ import io.kubernetes.client.openapi.ApiClient;
 import io.kubernetes.client.openapi.ApiException;
 import io.kubernetes.client.openapi.Configuration;
 import io.kubernetes.client.openapi.apis.CoreV1Api;
-import io.kubernetes.client.openapi.models.V1Pod;
-import io.kubernetes.client.openapi.models.V1PodList;
+import io.kubernetes.client.openapi.models.V1Namespace;
+import io.kubernetes.client.openapi.models.V1ObjectMeta;
 import io.kubernetes.client.util.ClientBuilder;
 import io.kubernetes.client.util.KubeConfig;
 import java.io.FileReader;
 import java.io.IOException;
+import java.nio.file.Paths;

 /**
  * A simple example of how to use the Java API from an application outside a kubernetes cluster
@@ -37,10 +38,12 @@ public class KubeConfigFileClientExample {
     // file path to your KubeConfig

     String kubeConfigPath = System.getenv("HOME") + "/.kube/config";
+    KubeConfig kubeconfig = KubeConfig.loadKubeConfig(new FileReader(kubeConfigPath));
+    kubeconfig.setFile(Paths.get(kubeConfigPath).toFile());

     // loading the out-of-cluster config, a kubeconfig from file-system
     ApiClient client =
-        ClientBuilder.kubeconfig(KubeConfig.loadKubeConfig(new FileReader(kubeConfigPath))).build();
+        ClientBuilder.kubeconfig(kubeconfig).build();

     // set the global default api-client to the in-cluster one from above
     Configuration.setDefaultApiClient(client);
@@ -48,11 +51,11 @@ public class KubeConfigFileClientExample {
     // the CoreV1Api loads default api-client from global configuration.
     CoreV1Api api = new CoreV1Api();

-    // invokes the CoreV1Api client
-    V1PodList list =
-        api.listPodForAllNamespaces().execute();
-    for (V1Pod item : list.getItems()) {
-      System.out.println(item.getMetadata().getName());
-    }
+    V1Namespace ns = new V1Namespace();
+    V1ObjectMeta meta = new V1ObjectMeta();
+    meta.setName("test");
+    ns.setMetadata(meta);
+    V1Namespace result = api.createNamespace(ns).pretty("true").execute();
+    System.out.println(result.getMetadata().getName());
   }
 }

Run the example:

cd examples/examples-release-20
mvn -X clean install exec:java -Dexec.mainClass="io.kubernetes.client.examples.KubeConfigFileClientExample"

Expected behavior

The namespace is created.

KubeConfig

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: REDACTED
    server: REDACTED
    tls-server-name: REDACTED
  name: REDACTED
contexts:
- context:
    cluster: REDACTED
    extensions:
    - extension: sandbox
      name: teleport.kube.name
    user: REDACTED
  name: REDACTED
current-context: REDACTED
kind: Config
preferences: {}
users:
- name: REDACTED
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - kube
      - credentials
      - --kube-cluster=sandbox
      - --teleport-cluster=REDACTED
      - --proxy=REDACTED
      command: tsh
      env: null
      provideClusterInfo: false

Server (please complete the following information):

OS: Linux
Environment: system
Cloud: Azure/Teleport

Additional context

My user has limited permissions, but I should be able to create a namespace:

$ kubectl auth can-i --list
Resources                                       Non-Resource URLs   Resource Names   Verbs
selfsubjectreviews.authentication.k8s.io        []                  []               [create]
selfsubjectaccessreviews.authorization.k8s.io   []                  []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                  []               [create]
namespaces.*                                    []                  []               [get list watch create delete]
nodes                                           []                  []               [get list watch]
                                                [/api/*]            []               [get]
                                                [/api]              []               [get]
                                                [/apis/*]           []               [get]
                                                [/apis]             []               [get]
                                                [/healthz]          []               [get]
                                                [/healthz]          []               [get]
                                                [/livez]            []               [get]
                                                [/livez]            []               [get]
                                                [/openapi/*]        []               [get]
                                                [/openapi]          []               [get]
                                                [/readyz]           []               [get]
                                                [/readyz]           []               [get]
                                                [/version/]         []               [get]
                                                [/version/]         []               [get]
                                                [/version]          []               [get]
                                                [/version]          []               [get]

brendandburns commented 8 months ago

What exception are you seeing?

nicolas-goudry commented 8 months ago

Here is the exception I’m seeing:

Caused by: io.kubernetes.client.openapi.ApiException: Message: Forbidden
HTTP response code: 403
HTTP response body: Forbidden

HTTP response headers: {content-length=[10], content-type=[text/plain; charset=utf-8], date=[Fri, 22 Mar 2024 16:09:32 GMT], x-content-type-options=[nosniff]}

It seems that somehow the authentication process which uses Teleport is not working and therefore the request is not authorized by the API.

brendandburns commented 8 months ago

Without access to a teleport cluster it's kind of hard to debug this.

The relevant exec code is here: https://github.com/kubernetes-client/java/blob/ef07a5b190b6a35bbfd8ee5c060c729f01694bb9/util/src/main/java/io/kubernetes/client/util/KubeConfig.java#L269

Can you turn on debug logging and see what it seems to be doing?

nicolas-goudry commented 8 months ago

Can you turn on debug logging and see what it seems to be doing?

By “debug logging” I guess you meant running maven with the -X flag. I also used client.setDebugging(true).

Here are the logs:

Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: --> POST https://REDACTED.teleport.sh/api/v1/namespaces
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: Content-Length: 57
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: Accept: application/json
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: Content-Type: application/json
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: User-Agent: Kubernetes Java Client/20.0.0-SNAPSHOT
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: 
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: {"metadata":{"annotations":{},"labels":{},"name":"test"}}
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: --> END POST (57-byte body)
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: <-- 403 Forbidden https://REDACTED.teleport.sh/api/v1/namespaces (486ms)
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: Content-Type: text/plain; charset=utf-8
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: X-Content-Type-Options: nosniff
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: Date: Sun, 24 Mar 2024 07:54:17 GMT
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: Content-Length: 10
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: 
Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: Forbidden

Mar 24, 2024 8:54:17 AM okhttp3.internal.platform.Platform log
INFO: <-- END HTTP (10-byte body)
[WARNING] 
io.kubernetes.client.openapi.ApiException: Message: Forbidden
HTTP response code: 403
HTTP response body: Forbidden

HTTP response headers: {content-length=[10], content-type=[text/plain; charset=utf-8], date=[Sun, 24 Mar 2024 07:54:17 GMT], x-content-type-options=[nosniff]}
    at io.kubernetes.client.openapi.ApiClient.handleResponse (ApiClient.java:1116)
    at io.kubernetes.client.openapi.ApiClient.execute (ApiClient.java:1029)
    at io.kubernetes.client.openapi.apis.CoreV1Api.createNamespaceWithHttpInfo (CoreV1Api.java:9213)
    at io.kubernetes.client.openapi.apis.CoreV1Api.access$19300 (CoreV1Api.java:77)
    at io.kubernetes.client.openapi.apis.CoreV1Api$APIcreateNamespaceRequest.execute (CoreV1Api.java:9307)
    at io.kubernetes.client.examples.KubeConfigFileClientExample.main (KubeConfigFileClientExample.java:59)
    at org.codehaus.mojo.exec.ExecJavaMojo.lambda$execute$0 (ExecJavaMojo.java:283)
    at java.lang.Thread.run (Thread.java:833)
[DEBUG] interrupting thread Thread[OkHttp TaskRunner,5,io.kubernetes.client.examples.KubeConfigFileClientExample]
[DEBUG] joining on thread Thread[OkHttp TaskRunner,5,io.kubernetes.client.examples.KubeConfigFileClientExample]
[WARNING] thread Thread[OkHttp TaskRunner,5,io.kubernetes.client.examples.KubeConfigFileClientExample] was interrupted but is still alive after waiting at least 15000msecs
[WARNING] thread Thread[OkHttp TaskRunner,5,io.kubernetes.client.examples.KubeConfigFileClientExample] will linger despite being asked to die via interruption
[WARNING] NOTE: 1 thread(s) did not finish despite being asked to via interruption. This is not a problem with exec:java, it is a problem with the running code. Although not serious, it should be remedied.
[WARNING] Couldn't destroy threadgroup org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=io.kubernetes.client.examples.KubeConfigFileClientExample,maxpri=10]
java.lang.IllegalThreadStateException
    at java.lang.ThreadGroup.destroy (ThreadGroup.java:803)
    at org.codehaus.mojo.exec.ExecJavaMojo.execute (ExecJavaMojo.java:331)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  23.620 s
[INFO] Finished at: 2024-03-24T08:54:32+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.1.1:java (default-cli) on project client-java-examples-release-20: An exception occurred while executing the Java class. Message: Forbidden
[ERROR] HTTP response code: 403
[ERROR] HTTP response body: Forbidden
[ERROR] 
[ERROR] HTTP response headers: {content-length=[10], content-type=[text/plain; charset=utf-8], date=[Sun, 24 Mar 2024 07:54:17 GMT], x-content-type-options=[nosniff]}
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.1.1:java (default-cli) on project client-java-examples-release-20: An exception occurred while executing the Java class. Message: Forbidden
HTTP response code: 403
HTTP response body: Forbidden

HTTP response headers: {content-length=[10], content-type=[text/plain; charset=utf-8], date=[Sun, 24 Mar 2024 07:54:17 GMT], x-content-type-options=[nosniff]}
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:333)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: org.apache.maven.plugin.MojoExecutionException: An exception occurred while executing the Java class. Message: Forbidden
HTTP response code: 403
HTTP response body: Forbidden

HTTP response headers: {content-length=[10], content-type=[text/plain; charset=utf-8], date=[Sun, 24 Mar 2024 07:54:17 GMT], x-content-type-options=[nosniff]}
    at org.codehaus.mojo.exec.ExecJavaMojo.execute (ExecJavaMojo.java:354)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: io.kubernetes.client.openapi.ApiException: Message: Forbidden
HTTP response code: 403
HTTP response body: Forbidden

HTTP response headers: {content-length=[10], content-type=[text/plain; charset=utf-8], date=[Sun, 24 Mar 2024 07:54:17 GMT], x-content-type-options=[nosniff]}
    at io.kubernetes.client.openapi.ApiClient.handleResponse (ApiClient.java:1116)
    at io.kubernetes.client.openapi.ApiClient.execute (ApiClient.java:1029)
    at io.kubernetes.client.openapi.apis.CoreV1Api.createNamespaceWithHttpInfo (CoreV1Api.java:9213)
    at io.kubernetes.client.openapi.apis.CoreV1Api.access$19300 (CoreV1Api.java:77)
    at io.kubernetes.client.openapi.apis.CoreV1Api$APIcreateNamespaceRequest.execute (CoreV1Api.java:9307)
    at io.kubernetes.client.examples.KubeConfigFileClientExample.main (KubeConfigFileClientExample.java:59)
    at org.codehaus.mojo.exec.ExecJavaMojo.lambda$execute$0 (ExecJavaMojo.java:283)
    at java.lang.Thread.run (Thread.java:833)

Let me know if I should enable any other option.

brendandburns commented 8 months ago

Can you try it adding -Dorg.slf4j.simpleLogger.defaultLogLevel=debug There are some debug level logs in the kubeconfig loader that will (hopefully) provide more info.

I'm more interested in the logs when the kubeconfig is loaded, vs. the logs when the call is made.

nicolas-goudry commented 8 months ago

I added the option, ending up with this command:

mvn -X clean install exec:java -Dexec.mainClass="io.kubernetes.client.examples.KubeConfigFileClientExample" -Dorg.slf4j.simpleLogger.defaultLogLevel=debug

There aren’t any further logs than the ones I sent earlier. I tried with -Dorg.slf4j.simpleLogger.defaultLogLevel=trace too, without success.

brendandburns commented 8 months ago

I'm specifically trying to see if these log lines are loading:

https://github.com/kubernetes-client/java/blob/ef07a5b190b6a35bbfd8ee5c060c729f01694bb9/util/src/main/java/io/kubernetes/client/util/KubeConfig.java#L307

https://github.com/kubernetes-client/java/blob/ef07a5b190b6a35bbfd8ee5c060c729f01694bb9/util/src/main/java/io/kubernetes/client/util/KubeConfig.java#L317

My bet is that the -Dorg.sl4j... is going to Maven, not to the java exec, but I'm not certain.

nicolas-goudry commented 8 months ago

My bet is that the -Dorg.sl4j... is going to Maven, not to the java exec, but I'm not certain.

That’s indeed what is happening here… I tried several solutions to pass the property down to the java exec, without any success.

In the end, I decided to manually edit the code to use the info level and use examples-release-latest to do my test. I know it’s not very conventional, but at least it worked :sweat_smile:

Here are the relevant logs you were seeking:

09:25:07.541 [io.kubernetes.client.examples.KubeConfigFileClientExample.main()] INFO io.kubernetes.client.util.KubeConfig -- Resolved /nix/store/j4dlmvfan176hzn517rzc0a9c9jb5lqs-teleport-14.3.0-client/bin/tsh to /nix/store/j4dlmvfan176hzn517rzc0a9c9jb5lqs-teleport-14.3.0-client/bin/tsh
09:25:07.642 [io.kubernetes.client.examples.KubeConfigFileClientExample.main()] INFO io.kubernetes.client.util.KubeConfig -- Obtained a client certificate from /nix/store/j4dlmvfan176hzn517rzc0a9c9jb5lqs-teleport-14.3.0-client/bin/tsh

So in the end it seems that the authentication part is working fine with the exec command… The issue seems to be elsewhere.

brendandburns commented 7 months ago

Given that it is loading the cert, this all gets injected into the request here:

https://github.com/kubernetes-client/java/blob/master/util/src/main/java/io/kubernetes/client/util/credentials/KubeconfigAuthentication.java#L45

How long do these teleport certificates last? The authenticator only loads it at config loading time, and doesn't reload ever, so if there is a short expiration time that might cause it.

nicolas-goudry commented 7 months ago

I checked by manually running the tsh command which generates the certificate: it is valid for 12 hours.

I wondered if this issue could somehow be linked to the lack of support for tls-server-name, but I get the same error without it (same kubeconfig, remove certificate-authority-data and tls-server-name and add insecure-skip-tls-verify: true).

k8s-triage-robot commented 4 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

nicolas-goudry commented 4 months ago

/remove-lifecycle stale

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

nicolas-goudry commented 1 month ago

/remove-lifecycle stale /lifecycle frozen

kubernetes-client / java

Unable to interact with cluster through Teleport #3242