GCP authentication does not support refreshing tokens

[omerkarj]

This relates to #143.

GCPAuthenticator does not implement the refresh() method. How can the client be used without manually refreshing the access token if it is expired?

for reference, initializing the client (i.e. running the example) with an expired token will result in: Caused by: java.lang.IllegalStateException: Unimplemented at io.kubernetes.client.util.authenticators.GCPAuthenticator.refresh(GCPAuthenticator.java:49) at io.kubernetes.client.util.KubeConfig.getAccessToken(KubeConfig.java:188) at io.kubernetes.client.util.credentials.KubeconfigAuthentication.<init>(KubeconfigAuthentication.java:33) at io.kubernetes.client.util.ClientBuilder.kubeconfig(ClientBuilder.java:165) at io.kubernetes.client.util.ClientBuilder.standard(ClientBuilder.java:80) at io.kubernetes.client.util.Config.defaultClient(Config.java:104) at serivces.KubernetesService.<clinit>(KubernetesService.groovy:23) ... 2 more

[brendandburns]

There's no way to fix this right now, someone needs to implement the token refresh.

[bootstraponline]

This breaks using the client. I had started on an integration then noticed the calls stopped working. Maybe mark the issue as contributions welcome if this isn't on the roadmap.

val client = Config.defaultClient()
Configuration.setDefaultApiClient(client)

val api = CoreV1Api(client)

// kubectl get service --field-selector metadata.name=esp-my-pod
val fieldSelector = "metadata.name=esp-my-pod"

val result = api.listServiceForAllNamespaces(
        null,
        fieldSelector,
        null,
        "",
        1,
        null,
        null,
        2 * 60,
        false)

Exception in thread "main" java.lang.IllegalStateException: Unimplemented
    at io.kubernetes.client.util.authenticators.GCPAuthenticator.refresh(GCPAuthenticator.java:49)
    at io.kubernetes.client.util.KubeConfig.getAccessToken(KubeConfig.java:188)
    at io.kubernetes.client.util.credentials.KubeconfigAuthentication.<init>(KubeconfigAuthentication.java:33)
    at io.kubernetes.client.util.ClientBuilder.kubeconfig(ClientBuilder.java:165)
    at io.kubernetes.client.util.ClientBuilder.standard(ClientBuilder.java:80)
    at io.kubernetes.client.util.Config.defaultClient(Config.java:104)
    at ParseProto.getIp(ParseProto.kt:59)
    at ParseProto.main(ParseProto.kt:102)

[l15k4]

Does this mean that this java client cannot authenticate using Tokens and one of the remaining methods like OAuth or basic auth must be used ?

[l15k4]

This is strange, I don't want to use http basic auth and it is not clear to me how I can set up OAuth. There is even no new OAuth() in this project.

[l15k4]

There actually is a way that doesn't throw the Unimplemented exception :

    Config.fromToken(
      "https://1.2.3.4",
      "...",
      false
    )

However there is another error https://github.com/kubernetes-client/java/issues/163

[mattnworb]

One design issue with this, at least as of 52b65f8, is that the io.kubernetes.client.util.credentials.Authentication interface is only asked to provide authentication for an ApiClient at construction time - when ClientBuilder is first constructing the ApiClient.

To support tokens that change during the lifetime of an ApiClient instance (as with a GCP token that needs to be refreshed periodically), it seems like the design would have to change to have ApiClient periodically ask the Authentication (or some other interface) for a new or current token.

Would a patch to implement such a change be welcomed?

[jglick]

I would suggest that #238 be the way forward. Although Google Cloud does not yet advertise it, you can authenticate to GKE from kubectl without using any vendor-specific plugin:

users:
- name: gcp
  user:
    exec:
      apiVersion: "client.authentication.k8s.io/v1beta1"
      command: "sh"
      args:
        - "-c"
        - |
            gcloud config config-helper --format=json | jq '{"apiVersion": "client.authentication.k8s.io/v1beta1", "kind": "ExecCredential", "status": {"token": .credential.access_token, "expirationTimestamp": .credential.token_expiry}}'

[brendandburns]

@mattnworb the trouble is that ApiClient is generated, we would have to change the generated code first in order to auto-refresh tokens...

[hmeerlo]

maybe I misunderstand the issue, but how is anyone using this library with either GKE or AWS? They both use short-lived tokens as I recall. This makes it impossible for me to create a working product that can talk to the kubernetes cluster on GKE.

[mattnworb]

@hmeerlo we have an application that has to connect to the API of several GKE clusters. We have a scheduled thread that periodically executes something like the following:

String newToken = ...;  // fetch up-to-date token for GCP Service Account that is making the call
for (ApiClient apiClient : apiClients) {
  client.setApiKey(newToken);
}

I would much preferred of course if this refresh logic could be in the kubernetes-client library itself, but having the ability to change the apiKey of an already-constructed ApiClient at least unblocks things.

[haugene]

@hmeerlo We're doing something of the same as @mattnworb. Wrapped the ApiClient in another service that checks that the token is valid before returning it to the calling code. Basically a singleton pattern with a refresh mechanism on the side.

@mattnworb How are you refreshing the tokens? We're using a service-account.json file but resorted to putting it on the file system and having gcloud binary do calls to generate a token and then reading this from file. Feels sub-optimal to say the least.

[hmeerlo]

@mattnworb @haugene thanks for your insights, at least there is some light at the end of the tunnel :-) I agree with @haugene that some insight in how you do this would be helpful

[mattnworb]

@haugene we use GoogleCredentials.getApplicationDefault() to get a token for the service account of the GCE instance where this code is running. The code calls credentials.refreshIfExpired() on each of the loops before calling apiClient.setApiKey(..).

[jhbae200]

I am using it like this.

package kubernetes.gcp;

import com.google.auth.oauth2.AccessToken;
import com.google.auth.oauth2.GoogleCredentials;
import io.kubernetes.client.util.KubeConfig;
import io.kubernetes.client.util.authenticators.Authenticator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.IOException;
import java.time.Instant;
import java.util.Date;
import java.util.Map;

public class ReplacedGCPAuthenticator implements Authenticator {
    private static final Logger log;
    private static final String ACCESS_TOKEN = "access-token";
    private static final String EXPIRY = "expiry";

    static {
        log = LoggerFactory.getLogger(io.kubernetes.client.util.authenticators.GCPAuthenticator.class);
    }

    private final GoogleCredentials credentials;

    public ReplacedGCPAuthenticator(GoogleCredentials credentials) {
        this.credentials = credentials;
    }

    public String getName() {
        return "gcp";
    }

    public String getToken(Map<String, Object> config) {
        return (String) config.get("access-token");
    }

    public boolean isExpired(Map<String, Object> config) {
        Object expiryObj = config.get("expiry");
        Instant expiry = null;
        if (expiryObj instanceof Date) {
            expiry = ((Date) expiryObj).toInstant();
        } else if (expiryObj instanceof Instant) {
            expiry = (Instant) expiryObj;
        } else {
            if (!(expiryObj instanceof String)) {
                throw new RuntimeException("Unexpected object type: " + expiryObj.getClass());
            }

            expiry = Instant.parse((String) expiryObj);
        }

        return expiry != null && expiry.compareTo(Instant.now()) <= 0;
    }

    public Map<String, Object> refresh(Map<String, Object> config) {
        try {
            AccessToken accessToken = this.credentials.refreshAccessToken();

            config.put(ACCESS_TOKEN, accessToken.getTokenValue());
            config.put(EXPIRY, accessToken.getExpirationTime());
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
        return config;
    }
}

Running in.

//GoogleCredentials.fromStream(--something credential.json filestream--)
KubeConfig.registerAuthenticator(new ReplacedGCPAuthenticator(GoogleCredentials.getApplicationDefault()));
ApiClient client = Config.defaultClient();
Configuration.setDefaultApiClient(client);
CoreV1Api api = new CoreV1Api();
V1PodList list = api.listNamespacedPod("default", null, null, null, null, null, null, null, 30, Boolean.FALSE);
for (V1Pod item : list.getItems()) {
    System.out.println(item.getMetadata().getName());
}

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[hmeerlo]

/remove-lifecycle stale

[yue9944882]

@jhbae200 how do you think about sending the patch as a PR in the repo?

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[stevenschlansker]

This issue still causes problems with running the client past authentication token revocation. I guess we will have to continue spamming this issue every 90 days to keep it from going stale...

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[stevenschlansker]

/remove-lifecycle stale

What an annoying bot. Please stop closing issues that are still open, and affect many users!

[pawelprazak]

FYI: for out-of-cluster refresh token to work, kubectl registers a plugin to the golang client, that in return calls a gcloud internal command: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/plugin/pkg/client/auth/gcp/gcp.go

the command with args is kept in the kubeconfig:

...
        cmd-args: config config-helper --format=json
        cmd-path: /usr/lib/google-cloud-sdk/bin/gcloud
...

I was trying to figure out it doesn't work with and out-of-cluster setup:

java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials.

[brendandburns]

/lifecycle frozen

[emy-lee]

I have the same problem here:

Exception in thread "main" java.lang.IllegalStateException: Unimplemented
    at io.kubernetes.client.util.authenticators.GCPAuthenticator.refresh(GCPAuthenticator.java:61)
    at io.kubernetes.client.util.KubeConfig.getAccessToken(KubeConfig.java:215)
    at io.kubernetes.client.util.credentials.KubeconfigAuthentication.<init>(KubeconfigAuthentication.java:46)
    at io.kubernetes.client.util.ClientBuilder.kubeconfig(ClientBuilder.java:276)
    at untitled4.main(untitled4.java:28)

Process finished with exit code 1

So what is the solution?

[dfernandezm]

@jhbae200 @brendandburns I'm facing this issue and have the requirement to authenticate against GKE from a Cloud Function (run this client in a Cloud Function). This means that gcloud binary cannot be bundled to support the way of refreshing tokens indicated in https://github.com/kubernetes-client/java/pull/1810.

What I did is implementing something very similar to https://github.com/kubernetes-client/java/issues/290#issuecomment-480205118, what would be the reason of not making that solution the proper refresh one?

[jhbae200]

@dfernandezm

https://cloud.google.com/kubernetes-engine/docs/how-to/api-server-authentication#environments-without-gcloud

I just did a PR supporting that feature.

[dfernandezm]

@jhbae200 thanks a lot for your quick response.

Your PR implements that piece, but that does not support the 'full' flow from out-of-cluster auth perspective IMO. Let me elaborate a bit more.

Following on the GCP docs example you link, this block in kubeconfig is automatically populated by kubectl:

users:
- name: ci-cd-pipeline-gsa
  user:
    auth-provider:
      name: gcp

with access-token and expiry, for example:

  user:
    auth-provider:
      config:
        access-token: "ya29.c. rest of token"
        expiry: "2021-11-26T13:23:13.979704Z"
      name: gcp

But this Kube Java Client does not support that functionality out-of-the-box without having gcloud around (this is what you implemented). There's no 'code-only' automated discovery of the GCP Application Credentials that can then populate the access-token and expiry in kubeconfig, making the config complete to call Kube API inside a GKE cluster.

Wouldn't this functionality (have an automated way of completing a KubeConfig with token/expiry) be desirable in order to have gcloud-free setup? As I mentioned in my comment, and as a use case, in Cloud Functions one does not have the ability to bundle gcloud. It would also make the client more standalone, not dependent on gcloud being around.

I would be happy to try and attempt a further patch with some guideline if there's a desire to include this. Let me know your thoughts.

[jhbae200]

If you want to use oauth2.0 (access token, refresh token), kubeconfig already supports oidc. After receiving the Access Token and Refresh Token as a service account, how about using it as oidc? https://accounts.google.com/.well-known/openid-configuration

[dfernandezm]

@jhbae200 Yes, I mean, this could be done via OIDC or appended to KubeConfig. In both ways this setup would avoid having to have gcloud around. I got this to work for my setup, I just wonder if it should be merged into the main GCPAuthenticator somehow, as an additional patch?

[dfernandezm]

@jhbae200 I see you have a PR opened for this already, that was quick! I can maybe remove my hacky code pretty soon! Thanks!

[FBryczak]

hello, w.r.t to gcloud based authentication in version 14.0.0: I have executed _gcloud container clusters get-credentials gkexxxxxxxxxxxxxxxxx ...

Now ~/.kube/config looks like this: users:

- name: gke_xxxxxxxxxxxxxxxxx
  user:
    auth-provider:
      config:
        cmd-args: config config-helper --format=json
        cmd-path: /home/filip/google-cloud-sdk/bin/gcloud
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

which is expected, so far so good. Now running my app results in:

Caused by: java.lang.NullPointerException: null
        at io.kubernetes.client.util.authenticators.GCPAuthenticator.isExpired(GCPAuthenticator.java:77)
        at io.kubernetes.client.util.KubeConfig.getAccessToken(KubeConfig.java:214)
        at io.kubernetes.client.util.credentials.KubeconfigAuthentication.<init>(KubeconfigAuthentication.java:57)
        at io.kubernetes.client.util.ClientBuilder.kubeconfig(ClientBuilder.java:297)
        ...
        ... 27 common frames omitted

Only after querying the cluster for example using 'kubectl get nodes' kubeconfig gets primed:

- name: gke_xxxxxxxxxxxxxxxxx
  user:
    auth-provider:
      config:
        access-token: ya29.a0ARrdaM-dRxk3op...zXsA_wCeUl7pqXuXV3g
        cmd-args: config config-helper --format=json
        cmd-path: /home/filip/google-cloud-sdk/bin/gcloud
        expiry: "2021-12-06T20:52:06Z"
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

and subsequent executions of my app work even past expiration time of the token in kubeconfig (seems like the refresh works).

Oddly enough, providing some expired token in the fields "access-token" and "expiry" works around this problem as well.

Is this an issue or I could do better?

[th0masb]

hello, w.r.t to gcloud based authentication in version 14.0.0: I have executed _gcloud container clusters get-credentials ... Is this an issue or I could do better?

I observe this issue too. Seems like a bug if the authenticator throws an exception if the "expiry" key is missing from the config when it sometimes is missing.

[honnix]

Looks like the support without gcloud has been added. However I'm still very confused whether the access token would ever be refreshed again after the first refresh, because I see the access token is then stored as client apiKey and never changes. How can one tell the SDK to refresh the access token whenever it gets expired?