search

Shopify / krane

A command-line tool that helps you ship changes to a Kubernetes namespace and understand the result
MIT License
1.39k stars 116 forks source link

Kubernetes API tokens are not being refreshed. #456

Open rendhalver opened 5 years ago

rendhalver commented 5 years ago

Bug report

[Description of the bug] Running kubernetes-restart with an expired API token results in the command failing. I have not tested this with the other commands so I am not sure if it's exclusive to the restart command. Kubernetes id-token's expire reasonably quickly (at least every few days). When kubectl is run with an expired token it will attempt to refresh it before contacting the api server. It definitely looks like kubernetes-restart and possibly other commands are not refreshing that token.

I tested this buy checking my ~/.kube/config file into git and running the restart. I think ran a kubectl command and diffed the kube config file and the id-token and refresh-token were indeed different. After that update the restart command succeeded.

Expected behavior: [What you expected to happen]

Restart is performed successfully.

Actual behavior: [What actually happened]

This was the output from a kubernetes-restart with an expired token.

[INFO][2019-03-27 12:31:58 +0000]
[INFO][2019-03-27 12:31:58 +0000]    -----------------------------------Phase 1: Initializing restart------------------------------------
Traceback (most recent call last):
    18: from /usr/local/bin/kubernetes-restart:23:in `<main>'
    17: from /usr/local/bin/kubernetes-restart:23:in `load'
    16: from /var/lib/gems/2.5.0/gems/kubernetes-deploy-0.25.0/exe/kubernetes-restart:28:in `<top (required)>'
    15: from /var/lib/gems/2.5.0/gems/kubernetes-deploy-0.25.0/lib/kubernetes-deploy/restart_task.rb:42:in `perform!'
    14: from /var/lib/gems/2.5.0/gems/kubernetes-deploy-0.25.0/lib/kubernetes-deploy/restart_task.rb:123:in `verify_namespace'
    13: from /var/lib/gems/2.5.0/gems/kubernetes-deploy-0.25.0/lib/kubernetes-deploy/restart_task.rb:179:in `kubeclient'
    12: from /var/lib/gems/2.5.0/gems/kubernetes-deploy-0.25.0/lib/kubernetes-deploy/kubeclient_builder.rb:17:in `build_v1_kubeclient'
    11: from /var/lib/gems/2.5.0/gems/kubernetes-deploy-0.25.0/lib/kubernetes-deploy/kubeclient_builder.rb:102:in `_build_kubeclient'
    10: from /var/lib/gems/2.5.0/gems/kubeclient-4.3.0/lib/kubeclient/config.rb:47:in `context'
     9: from /var/lib/gems/2.5.0/gems/kubernetes-deploy-0.25.0/lib/kubernetes-deploy/kubeclient_builder/kube_config.rb:16:in `fetch_user_auth_options'
     8: from /var/lib/gems/2.5.0/gems/kubeclient-4.3.0/lib/kubeclient/config.rb:162:in `fetch_user_auth_options'
     7: from /var/lib/gems/2.5.0/gems/kubeclient-4.3.0/lib/kubeclient/oidc_auth_provider.rb:24:in `token'
     6: from /var/lib/gems/2.5.0/gems/openid_connect-1.1.6/lib/openid_connect/response_object/id_token.rb:67:in `decode'
     5: from /var/lib/gems/2.5.0/gems/json-jwt-1.10.0/lib/json/jose.rb:63:in `decode'
     4: from /var/lib/gems/2.5.0/gems/json-jwt-1.10.0/lib/json/jwt.rb:104:in `decode_compact_serialized'
     3: from /var/lib/gems/2.5.0/gems/json-jwt-1.10.0/lib/json/jws.rb:195:in `decode_compact_serialized'
     2: from /var/lib/gems/2.5.0/gems/json-jwt-1.10.0/lib/json/jws.rb:26:in `verify!'
     1: from /var/lib/gems/2.5.0/gems/json-jwt-1.10.0/lib/json/jws.rb:131:in `valid?'
/var/lib/gems/2.5.0/gems/json-jwt-1.10.0/lib/json/jose.rb:29:in `with_jwk_support': JSON::JWK::Set::KidNotFound (JSON::JWK::Set::KidNotFound)

Version(s) affected: [run kubernetes-deploy --version] v0.25.0

Steps to Reproduce

  1. [First Step] Wait for your API token to expire. We discovered this on a Monday so a couple of days will do it.
  2. [Second Step] Run kubernetes-restart
benlangfeld commented 5 years ago

So, this turns out to have been fixed by https://github.com/abonas/kubeclient/pull/407.

Unfortunately, this just opens us up to another related issue: the second invocation of kubeclient in a restart, because it's using a new Kubeclient::Client instance reading fresh from the KUBECONFIG file, and the previous refresh when verifying the namespace didn't update the config file, our refresh-token is now invalid:

Note: id-tokens retrieved via this provider are not written back to the $KUBECONFIG file as they would be when using kubectl.

I think it's probably the responsibility of kubernetes-deploy to write this back to the correct KUBECONFIG file, so this ticket constitutes that feature request.

benlangfeld commented 5 years ago

See https://github.com/abonas/kubeclient/issues/409