The kubeconfig loader should run refresh command to update token when it is expired

[nhumrich]

I am using google container engine, and trying to use this to access the k8s api. Trying to follow the example on the readme

from kubernetes import client, config

config.load_kube_config()
api = client.CoreV1Api()
pods = api.list_pod_for_all_namespaces(watch=False)

for p in pods.items:
    print(p.metadata.name, p.status.phase)

which gives me the following error:

Traceback (most recent call last):
  File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 6, in <module>
    config.load_kube_config()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 311, in load_kube_config
    client_configuration=client_configuration).load_and_set()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 216, in load_and_set
    self._load_authentication()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 155, in _load_authentication
    if self._load_gcp_token():
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 171, in _load_gcp_token
    self.token = "Bearer %s" % self._get_google_credentials()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 124, in <lambda>
    GoogleCredentials.get_application_default()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/oauth2client/client.py", line 1271, in get_application_default
    return GoogleCredentials._get_implicit_credentials()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/oauth2client/client.py", line 1261, in _get_implicit_credentials
    raise ApplicationDefaultCredentialsError(ADC_HELP_MSG)
oauth2client.client.ApplicationDefaultCredentialsError: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

If I add the GOOGLE_APPLICATION_CREDENTIALS env-var and download a google json credential file, I then get a generic 401.

Traceback (most recent call last):
  File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 19, in <module>
    pods = api.list_pod_for_all_namespaces(watch=False)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13650, in list_pod_for_all_namespaces
    (data) = self.list_pod_for_all_namespaces_with_http_info(**kwargs)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13743, in list_pod_for_all_namespaces_with_http_info
    collection_formats=collection_formats)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 329, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 153, in __call_api
    _request_timeout=_request_timeout)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 361, in request
    headers=headers)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 240, in GET
    query_params=query_params)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/plain; charset=utf-8', 'Www-Authenticate': 'Basic realm="kubernetes-master"', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 22 May 2017 21:25:24 GMT', 'Content-Length': '13'})
HTTP response body: Unauthorized

If I try to add an api key (client.configuration.api_key['authorization'] = 'AbX.....SYh' I get another error.

Traceback (most recent call last):
  File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 19, in <module>
    pods = api.list_pod_for_all_namespaces(watch=False)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13650, in list_pod_for_all_namespaces
    (data) = self.list_pod_for_all_namespaces_with_http_info(**kwargs)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13743, in list_pod_for_all_namespaces_with_http_info
    collection_formats=collection_formats)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 329, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 153, in __call_api
    _request_timeout=_request_timeout)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 361, in request
    headers=headers)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 240, in GET
    query_params=query_params)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 22 May 2017 21:40:29 GMT', 'Content-Length': '119'})
HTTP response body: User "system:anonymous" cannot list pods at the cluster scope.: "No policy matched.\nUnknown user \"system:anonymous\""

Are there any examples of how I authenticate with kubernetes/google container engine so that I can get this working?

Note: one possible solution is to run gcloud auth application-default login but that isn't automated and only works locally.

[jonathan-kosgei]

Hi Nick, seems you're trying to use gcloud credentials to auth to the Kubernetes cluster. The two are very different, instead of using a gce serviceaccount as the credential source in load_config you need to use your kube config file. If you can run kubectl commands locally then you have one under ~/.kube/config

---- On Tue, 23 May 2017 00:43:06 +0300 notifications@github.com wrote ----

I am using google container engine, and trying to use this to access the k8s api. Trying to follow the example on the readme

from kubernetes import client, config

config.load_kube_config() api = client.CoreV1Api() pods = api.list_pod_for_all_namespaces(watch=False)

for p in pods.items: print(p.metadata.name, p.status.phase) which gives me the following error:

Traceback (most recent call last): File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 6, in config.load_kube_config() File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 311, in load_kube_config client_configuration=client_configuration).load_and_set() File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 216, in load_and_set self._load_authentication() File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 155, in _load_authentication if self._load_gcp_token(): File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 171, in _load_gcp_token self.token = "Bearer %s" % self._get_google_credentials() File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 124, in GoogleCredentials.get_application_default() File "/home/nhumrich/.local/lib/python3.6/site-packages/oauth2client/client.py", line 1271, in get_application_default return GoogleCredentials._get_implicit_credentials() File "/home/nhumrich/.local/lib/python3.6/site-packages/oauth2client/client.py", line 1261, in _get_implicit_credentials raise ApplicationDefaultCredentialsError(ADC_HELP_MSG) oauth2client.client.ApplicationDefaultCredentialsError: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information. If I add the GOOGLE_APPLICATION_CREDENTIALS env-var and download a google json credential file, I then get a generic 401.

Traceback (most recent call last): File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 19, in pods = api.list_pod_for_all_namespaces(watch=False) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13650, in list_pod_for_all_namespaces (data) = self.list_pod_for_all_namespaces_with_http_info(kwargs) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13743, in list_pod_for_all_namespaces_with_http_info collection_formats=collection_formats) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 329, in call_api _return_http_data_only, collection_formats, _preload_content, _request_timeout) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 153, in __call_api _request_timeout=_request_timeout) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 361, in request headers=headers) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 240, in GET query_params=query_params) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request raise ApiException(http_resp=r) kubernetes.client.rest.ApiException: (401) Reason: Unauthorized HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/plain; charset=utf-8', 'Www-Authenticate': 'Basic realm="kubernetes-master"', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 22 May 2017 21:25:24 GMT', 'Content-Length': '13'}) HTTP response body: Unauthorized If I try to add an api key (client.configuration.api_key['authorization'] = 'AIz.....SYU' I get another error. ``` Traceback (most recent call last): File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 19, in pods = api.list_pod_for_all_namespaces(watch=False) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13650, in list_pod_for_all_namespaces (data) = self.list_pod_for_all_namespaces_with_http_info(kwargs) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13743, in list_pod_for_all_namespaces_with_http_info collection_formats=collection_formats) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 329, in call_api _return_http_data_only, collection_formats, _preload_content, _request_timeout) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 153, in __call_api _request_timeout=_request_timeout) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 361, in request headers=headers) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 240, in GET query_params=query_params) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request raise ApiException(http_resp=r) kubernetes.client.rest.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 22 May 2017 21:40:29 GMT', 'Content-Length': '119'}) HTTP response body: User "system:anonymous" cannot list pods at the cluster scope.: "No policy matched.\nUnknown user "system:anonymous""

Are there any examples of how I authenticate with kubernetes/google container engine so that I can get this working?

Note: one possible solution is to run gcloud auth application-default login but that isn't automated and only works locally. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[nhumrich]

I tried doing that which is the first option. I also tried passing in the config file location, but that doesnt change anything, as that is the default anyways.

[mbohlool]

Does kubectl works? if yes, can you provide content of ~/.kube/config file (remove any key/token/etc. from the file, only "current context" should be enough. e.g. if your current-context is "x" provide user "x" and cluster "x".

[nhumrich]

Here is my kube config file

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ...Something goes here...
    server: https://some.url
  name: gke_infastructure-111111_us-central1-a_my-cluster
contexts:
- context:
    cluster: gke_infastructure-111111_us-central1-a_my-cluster
    user: gke_infastructure-111111_us-central1-a_my-cluster
  name: gke_infastructure-111111_us-central1-a_my-cluster
current-context: gke_infastructure-111111_us-central1-a_my-cluster
kind: Config
preferences: {}
users:
- name: gke_infastructure-111111_us-central1-a_my-cluster
  user:
    auth-provider:
      config:
        access-token: zzzzzz.some-token-here.zzzzz
        cmd-args: config config-helper --format=json
        cmd-path: /opt/google-cloud-sdk/bin/gcloud
        expiry: 2017-05-24T19:57:52Z
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

[rafitadiaz]

+1 On this one. We are having the same problem. kubectl is working fine, but we get the 401 when using the library. And our config file look like this one above.

Anyone can send an example of how to do it properly or is it impossible make it works with GKE clusters?

[mbohlool]

The kubeconfig loader should run refresh command to update token when it is expired. To solve this, we can find the corresponding code in go client (which kubectl uses) and port it to python. Added it to milestone to make sure it will be looked at for the next release.

[zweizeichen]

Just ran into this issue myself. You can use this dirty hack to get auth on GKE in the meantime. Please note that the token retrieved will expire within a rather short timeframe. This should not be used in production.

In kube_config.py add these imports:

import subprocess
import json

modify KubeConfigLoader's __init__ method like this:

if get_google_credentials:
        self._get_google_credentials = get_google_credentials
else:
        self._get_google_credentials = lambda: (
            self._get_gcp_cmd_credentials()
        )

and add the following method to the class:

def _get_gcp_cmd_credentials(self):
    cmd_path = self._user.value['auth-provider']['config']['cmd-path']
    cmd_args = self._user.value['auth-provider']['config']['cmd-args'].split(' ')
    output = subprocess.run([cmd_path]+cmd_args, stdout=subprocess.PIPE, check=False).stdout.decode('utf-8')

    return json.loads(output)['credential']['access_token']

Done!

[nhumrich]

Update: I was able to work around this issue by creating a serviceaccount in kubernetes.

I then ran kubectl describe serviceaccount myserviceaccount and that will give you a secret name, then use that secret name to run: kubectl describe secrets [secret-name] and then copy the token field. One you have the token field, all you need to do is set the api token in the client:

config.load_kube_config()
client.configuration.api_key['authorization'] = 'your token goes here'
client.configuration.api_key_prefix['authorization'] = 'Bearer'

This worked great for me. If you dont want to use the kube config file at all, you can also set the host and cert yourself:

client.configuration.api_key['authorization'] = 'your token goes here'
client.configuration.api_key_prefix['authorization'] = 'Bearer'
client.configuration.host = 'https://some.domain-or-ip.example'
client.configuration.ssl_ca_cert = 'cert/location.crt'

[zweizeichen]

Hah I'm using the API to set up the service accounts - so it is a kind of chicken and egg problem. Of course using a service account is the preferred and more durable solution.

[nhumrich]

@zweizeichen you could always use kubectl to create a single serviceaccount, then go from there.

[jeremad]

I am still experiencing that issue (version 5.0.0 on mac)

[OraserZhang]

I am still experiencing that issue (version 5.0.0 on mac)

[richerlariviere]

I am still experiencing that issue (version 6.0.0 on mac and I also tried HEAD version) in Kubernetes 1.10.1

[efim-a-efim]

Same issue here - version 6.0.0 on Windows/WSL (Ubuntu). kubernetes 1.10.5

[tomplus]

The current implementation refreshes a token when configuration is loaded. It doesn't refresh it for long living applications. PTAL https://github.com/kubernetes-client/python-base/issues/59

[mooperd]

+1 I'm confused why the library isn't able to use the Kubeconfig properly.

[tellet]

I'm facing this issue with version 7.0.1 (on mac), kubernetes 1.10.11. Workarounds don't help.

[sidpremkumar]

We are also currently experiencing the same issue +1

[OraserZhang]

您的来信收到，谢谢！

[emdete]

why is this issue closed, the bug still exists:

  File "python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 15205, in list_namespaced_event
    return self.list_namespaced_event_with_http_info(namespace, **kwargs)  # noqa: E501
  File "python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 15320, in list_namespaced_event_with_http_info
    return self.api_client.call_api(
  File "python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "python3.9/site-packages/kubernetes/client/rest.py", line 241, in GET
    return self.request("GET", url,
  File "python3.9/site-packages/kubernetes/client/rest.py", line 235, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Audit-Id': '...', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/
json', 'Date': 'Thu, 23 Feb 2023 10:15:09 GMT', 'Content-Length': '129'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

running kubectl after that refreshes the token and the api works again.

[OraserZhang]

您的来信收到，谢谢！

[akaihola]

@OraserZhang wrote twice:

您的来信收到，谢谢！

This seems to be an automated reply, it machine-translates to "Your letter was received, thank you!". Could you configure your e-mail to not auto-respond to GitHub notifications? Thanks!

这似乎是一个自动回复。你能不能把你的电子邮件配置为不自动回复GitHub的通知？谢谢!

[emdete]

can this ticket be reopened because the bug still exists?