search

dexidp / dex

OpenID Connect (OIDC) identity and OAuth 2.0 provider with pluggable connectors
https://dexidp.io
Apache License 2.0
9.52k stars 1.71k forks source link

Tokens don't refresh in personal configs #2360

Open nixon89 opened 2 years ago

nixon89 commented 2 years ago

Preflight Checklist

Version

2.30.2

Storage Type

Kubernetes

Installation Type

Official Helm chart

Expected Behavior

When id-token expires - automatically refresh via kubectl When refresh-token expires - automatically refresh via kubectl

Actual Behavior

Case 1:

When some users run: kubectl get pods

Kubectl return error (500 internal server error):

❯ kubectl get pods -v7
I1208 17:24:34.056782   42200 loader.go:372] Config loaded from file:  /Users/nixon89/.kube/config
I1208 17:24:34.057745   42200 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1208 17:24:34.057757   42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.057762   42200 round_trippers.go:442]     Accept: application/json, */*
I1208 17:24:34.057766   42200 round_trippers.go:442]     User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1208 17:24:34.057806   42200 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1208 17:24:34.057812   42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.372320   42200 round_trippers.go:457] Response Status: 200 OK in 314 milliseconds
I1208 17:24:34.372760   42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:34.372797   42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.372819   42200 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1208 17:24:34.372843   42200 round_trippers.go:442]     Authorization: Basic <masked>
I1208 17:24:34.505009   42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 132 milliseconds
I1208 17:24:34.505195   42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:34.505215   42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.505231   42200 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1208 17:24:34.629779   42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 124 milliseconds
I1208 17:24:34.629857   42200 round_trippers.go:457] Response Status:  in 572 milliseconds
I1208 17:24:34.630094   42200 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 500 Internal Server Error
Response: {"error":"invalid_request"}
I1208 17:24:34.630884   42200 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1208 17:24:34.630921   42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.630943   42200 round_trippers.go:442]     Accept: application/json, */*
I1208 17:24:34.630961   42200 round_trippers.go:442]     User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1208 17:24:34.631098   42200 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1208 17:24:34.631137   42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.672148   42200 round_trippers.go:457] Response Status: 200 OK in 40 milliseconds
I1208 17:24:34.672409   42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:34.672423   42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.672429   42200 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1208 17:24:34.672436   42200 round_trippers.go:442]     Authorization: Basic <masked>
I1208 17:24:34.788667   42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 116 milliseconds
I1208 17:24:34.788787   42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:34.788807   42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.788818   42200 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1208 17:24:34.912881   42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 124 milliseconds
I1208 17:24:34.912995   42200 round_trippers.go:457] Response Status:  in 282 milliseconds
I1208 17:24:34.913075   42200 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 500 Internal Server Error
Response: {"error":"invalid_request"}
I1208 17:24:34.913276   42200 shortcut.go:89] Error loading discovery information: Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 500 Internal Server Error
Response: {"error":"invalid_request"}
I1208 17:24:34.913589   42200 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1208 17:24:34.913616   42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.913631   42200 round_trippers.go:442]     Accept: application/json, */*
I1208 17:24:34.913648   42200 round_trippers.go:442]     User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1208 17:24:34.913714   42200 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1208 17:24:34.913727   42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.954696   42200 round_trippers.go:457] Response Status: 200 OK in 40 milliseconds
I1208 17:24:34.954865   42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:34.954875   42200 round_trippers.go:438] Request Headers:
I1208 17:24:34.954881   42200 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1208 17:24:34.954887   42200 round_trippers.go:442]     Authorization: Basic <masked>
I1208 17:24:35.071954   42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 117 milliseconds
I1208 17:24:35.072144   42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:35.072181   42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.072199   42200 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1208 17:24:35.196954   42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 124 milliseconds
I1208 17:24:35.197027   42200 round_trippers.go:457] Response Status:  in 283 milliseconds
I1208 17:24:35.197129   42200 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 500 Internal Server Error
Response: {"error":"invalid_request"}
I1208 17:24:35.197449   42200 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1208 17:24:35.197486   42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.197508   42200 round_trippers.go:442]     Accept: application/json, */*
I1208 17:24:35.197526   42200 round_trippers.go:442]     User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1208 17:24:35.197597   42200 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1208 17:24:35.197611   42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.238414   42200 round_trippers.go:457] Response Status: 200 OK in 40 milliseconds
I1208 17:24:35.238569   42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:35.238586   42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.238594   42200 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1208 17:24:35.238601   42200 round_trippers.go:442]     Authorization: Basic <masked>
I1208 17:24:35.367725   42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 129 milliseconds
I1208 17:24:35.367837   42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:35.367993   42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.368061   42200 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1208 17:24:35.482251   42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 114 milliseconds
I1208 17:24:35.482374   42200 round_trippers.go:457] Response Status:  in 284 milliseconds
I1208 17:24:35.482488   42200 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 500 Internal Server Error
Response: {"error":"invalid_request"}
I1208 17:24:35.482891   42200 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1208 17:24:35.482934   42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.482955   42200 round_trippers.go:442]     Accept: application/json, */*
I1208 17:24:35.482971   42200 round_trippers.go:442]     User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1208 17:24:35.483044   42200 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1208 17:24:35.483063   42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.523516   42200 round_trippers.go:457] Response Status: 200 OK in 40 milliseconds
I1208 17:24:35.523652   42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:35.523666   42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.523675   42200 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1208 17:24:35.523687   42200 round_trippers.go:442]     Authorization: Basic <masked>
I1208 17:24:35.634540   42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 110 milliseconds
I1208 17:24:35.634638   42200 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1208 17:24:35.634696   42200 round_trippers.go:438] Request Headers:
I1208 17:24:35.634714   42200 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1208 17:24:35.752720   42200 round_trippers.go:457] Response Status: 500 Internal Server Error in 117 milliseconds
I1208 17:24:35.752808   42200 round_trippers.go:457] Response Status:  in 269 milliseconds
I1208 17:24:35.752905   42200 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 500 Internal Server Error
Response: {"error":"invalid_request"}
I1208 17:24:35.753242   42200 helpers.go:234] Connection error: Get https://k8s-api.example.ru:30000/api?timeout=32s: failed to refresh token: oauth2: cannot fetch token: 500 Internal Server Error
Response: {"error":"invalid_request"}
F1208 17:24:35.754424   42200 helpers.go:115] Unable to connect to the server: failed to refresh token: oauth2: cannot fetch token: 500 Internal Server Error
Response: {"error":"invalid_request"}
goroutine 1 [running]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc00018e001, 0xc0007b4f00, 0xc4, 0x13d)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1021 +0xb9
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x3c90720, 0xc000000003, 0x0, 0x0, 0xc0002870a0, 0x321ed28, 0xa, 0x73, 0x100e100)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:970 +0x191
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printDepth(0x3c90720, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x2, 0xc0000a9820, 0x1, 0x1)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:733 +0x16f
k8s.io/kubernetes/vendor/k8s.io/klog/v2.FatalDepth(...)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1495
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.fatal(0xc000278c80, 0x95, 0x1)

In Dex logs error (400 bad request): level=error msg="failed to refresh identity: oidc: failed to get refresh token: oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\"}"

Case 2:

When some users run: kubectl get pods -v 7

Kubectl return error (400):

❯ kubectl get pods -v 7
I1122 12:08:01.509592   57599 loader.go:372] Config loaded from file:  /Users/nixon89/.kube/config
I1122 12:08:01.510903   57599 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1122 12:08:01.510940   57599 round_trippers.go:438] Request Headers:
I1122 12:08:01.510952   57599 round_trippers.go:442]     User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1122 12:08:01.510960   57599 round_trippers.go:442]     Accept: application/json, */*
I1122 12:08:01.511026   57599 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1122 12:08:01.511035   57599 round_trippers.go:438] Request Headers:
I1122 12:08:01.898010   57599 round_trippers.go:457] Response Status: 200 OK in 386 milliseconds
I1122 12:08:01.898320   57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:01.898342   57599 round_trippers.go:438] Request Headers:
I1122 12:08:01.898355   57599 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1122 12:08:01.898370   57599 round_trippers.go:442]     Authorization: Basic <masked>
I1122 12:08:01.945645   57599 round_trippers.go:457] Response Status: 400 Bad Request in 47 milliseconds
I1122 12:08:01.945789   57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:01.945799   57599 round_trippers.go:438] Request Headers:
I1122 12:08:01.945806   57599 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1122 12:08:01.994794   57599 round_trippers.go:457] Response Status: 400 Bad Request in 48 milliseconds
I1122 12:08:01.994898   57599 round_trippers.go:457] Response Status:  in 483 milliseconds
I1122 12:08:01.994977   57599 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}
I1122 12:08:01.995549   57599 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1122 12:08:01.995583   57599 round_trippers.go:438] Request Headers:
I1122 12:08:01.995593   57599 round_trippers.go:442]     Accept: application/json, */*
I1122 12:08:01.995600   57599 round_trippers.go:442]     User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1122 12:08:01.995650   57599 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1122 12:08:01.995659   57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.040974   57599 round_trippers.go:457] Response Status: 200 OK in 45 milliseconds
I1122 12:08:02.041220   57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:02.041236   57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.041243   57599 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1122 12:08:02.041252   57599 round_trippers.go:442]     Authorization: Basic <masked>
I1122 12:08:02.089723   57599 round_trippers.go:457] Response Status: 400 Bad Request in 48 milliseconds
I1122 12:08:02.089845   57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:02.089870   57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.089892   57599 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1122 12:08:02.141262   57599 round_trippers.go:457] Response Status: 400 Bad Request in 51 milliseconds
I1122 12:08:02.141326   57599 round_trippers.go:457] Response Status:  in 145 milliseconds
I1122 12:08:02.141403   57599 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}
I1122 12:08:02.141452   57599 shortcut.go:89] Error loading discovery information: Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}
I1122 12:08:02.141677   57599 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1122 12:08:02.141695   57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.141705   57599 round_trippers.go:442]     Accept: application/json, */*
I1122 12:08:02.141713   57599 round_trippers.go:442]     User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1122 12:08:02.141762   57599 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1122 12:08:02.141771   57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.183973   57599 round_trippers.go:457] Response Status: 200 OK in 42 milliseconds
I1122 12:08:02.184268   57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:02.184297   57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.184308   57599 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1122 12:08:02.184323   57599 round_trippers.go:442]     Authorization: Basic <masked>
I1122 12:08:02.248091   57599 round_trippers.go:457] Response Status: 400 Bad Request in 63 milliseconds
I1122 12:08:02.248255   57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:02.248291   57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.248308   57599 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1122 12:08:02.293144   57599 round_trippers.go:457] Response Status: 400 Bad Request in 44 milliseconds
I1122 12:08:02.293205   57599 round_trippers.go:457] Response Status:  in 151 milliseconds
I1122 12:08:02.293260   57599 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}
I1122 12:08:02.293491   57599 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1122 12:08:02.293508   57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.293515   57599 round_trippers.go:442]     Accept: application/json, */*
I1122 12:08:02.293522   57599 round_trippers.go:442]     User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1122 12:08:02.293560   57599 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1122 12:08:02.293565   57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.333084   57599 round_trippers.go:457] Response Status: 200 OK in 39 milliseconds
I1122 12:08:02.333250   57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:02.333258   57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.333264   57599 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1122 12:08:02.333269   57599 round_trippers.go:442]     Authorization: Basic <masked>
I1122 12:08:02.380786   57599 round_trippers.go:457] Response Status: 400 Bad Request in 47 milliseconds
I1122 12:08:02.380845   57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:02.380855   57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.380866   57599 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1122 12:08:02.428589   57599 round_trippers.go:457] Response Status: 400 Bad Request in 47 milliseconds
I1122 12:08:02.428625   57599 round_trippers.go:457] Response Status:  in 135 milliseconds
I1122 12:08:02.428659   57599 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}
I1122 12:08:02.428786   57599 round_trippers.go:432] GET https://k8s-api.example.ru:30000/api?timeout=32s
I1122 12:08:02.428795   57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.428799   57599 round_trippers.go:442]     Accept: application/json, */*
I1122 12:08:02.428803   57599 round_trippers.go:442]     User-Agent: kubectl/v1.21.5 (darwin/amd64) kubernetes/aea7bba
I1122 12:08:02.428829   57599 round_trippers.go:432] GET https://k8s-login.example.ru/.well-known/openid-configuration
I1122 12:08:02.428833   57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.468816   57599 round_trippers.go:457] Response Status: 200 OK in 39 milliseconds
I1122 12:08:02.468931   57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:02.468942   57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.468948   57599 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1122 12:08:02.468953   57599 round_trippers.go:442]     Authorization: Basic <masked>
I1122 12:08:02.512975   57599 round_trippers.go:457] Response Status: 400 Bad Request in 44 milliseconds
I1122 12:08:02.513027   57599 round_trippers.go:432] POST https://k8s-login.example.ru/token
I1122 12:08:02.513034   57599 round_trippers.go:438] Request Headers:
I1122 12:08:02.513040   57599 round_trippers.go:442]     Content-Type: application/x-www-form-urlencoded
I1122 12:08:02.562111   57599 round_trippers.go:457] Response Status: 400 Bad Request in 49 milliseconds
I1122 12:08:02.562166   57599 round_trippers.go:457] Response Status:  in 133 milliseconds
I1122 12:08:02.562213   57599 cached_discovery.go:121] skipped caching discovery info due to Get "https://k8s-api.example.ru:30000/api?timeout=32s": failed to refresh token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}
I1122 12:08:02.562275   57599 helpers.go:234] Connection error: Get https://k8s-api.example.ru:30000/api?timeout=32s: failed to refresh token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}
F1122 12:08:02.562304   57599 helpers.go:115] Unable to connect to the server: failed to refresh token: oauth2: cannot fetch token: 400 Bad Request
Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}
goroutine 1 [running]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc000138001, 0xc000746000, 0x118, 0x169)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1021 +0xb9
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x3c90720, 0xc000000003, 0x0, 0x0, 0xc0001e42a0, 0x321ed28, 0xa, 0x73, 0x100e100)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:970 +0x191
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printDepth(0x3c90720, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x2, 0xc000554c00, 0x1, 0x1)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:733 +0x16f
k8s.io/kubernetes/vendor/k8s.io/klog/v2.FatalDepth(...)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1495
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.fatal(0xc000198000, 0xe9, 0x1)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:93 +0x288
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.checkErr(0x2c9ae60, 0xc000a2ba10, 0x2b244d0)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:188 +0x935
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.CheckErr(...)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:115
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/get.NewCmdGet.func1(0xc0005b0840, 0xc000081b30, 0x1, 0x3)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/get/get.go:167 +0x159
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc0005b0840, 0xc000081b00, 0x3, 0x3, 0xc0005b0840, 0xc000081b00)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:854 +0x2c2
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc00042f080, 0xc00013a120, 0xc00013e000, 0x5)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:958 +0x375
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(...)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:895
main.main()
    _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:49 +0x21d

goroutine 18 [chan receive]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).flushDaemon(0x3c90720)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1164 +0x8b
created by k8s.io/kubernetes/vendor/k8s.io/klog/v2.init.0
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:418 +0xdf

goroutine 8 [select]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x2b243f0, 0x2c993a0, 0xc000226000, 0x1, 0xc00010cb40)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x118
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x2b243f0, 0x12a05f200, 0x0, 0x1, 0xc00010cb40)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0x2b243f0, 0x12a05f200, 0xc00010cb40)
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/util/logs.InitLogs
    /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/util/logs/logs.go:51 +0x96

goroutine 22 [IO wait]:
internal/poll.runtime_pollWait(0xc71f410, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0007b6198, 0x72, 0x1200, 0x128e, 0xffffffffffffffff)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0007b6180, 0xc0009f5300, 0x128e, 0x128e, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc0007b6180, 0xc0009f5300, 0x128e, 0x128e, 0x1281, 0xc0009c6fa0, 0xd)
    /usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00000e9c0, 0xc0009f5300, 0x128e, 0x128e, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc000b20960, 0xc0009f5300, 0x128e, 0x128e, 0xc000354a80, 0xc00006a900, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc0009c7078, 0x2c979a0, 0xc000b20960, 0x100b605, 0x27e5b40, 0x29a6c40)
    /usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc0009c6e00, 0xc71f5e0, 0xc00000e9c0, 0x5, 0xc00000e9c0, 0x400)
    /usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc0009c6e00, 0x0, 0x0, 0x2)
    /usr/local/go/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
    /usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc0009c6e00, 0xc0001db000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1276 +0x165
bufio.(*Reader).Read(0xc0003bfc20, 0xc000b42f18, 0x9, 0x9, 0x13bf2ab, 0xc000a4dc78, 0x1006c65)
    /usr/local/go/src/bufio/bufio.go:227 +0x222
io.ReadAtLeast(0x2c977c0, 0xc0003bfc20, 0xc000b42f18, 0x9, 0x9, 0x9, 0xc000554aa0, 0x6cb0f689261800, 0xc000554aa0)
    /usr/local/go/src/io/io.go:328 +0x87
io.ReadFull(...)
    /usr/local/go/src/io/io.go:347
net/http.http2readFrameHeader(0xc000b42f18, 0x9, 0x9, 0x2c977c0, 0xc0003bfc20, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/h2_bundle.go:1477 +0x89
net/http.(*http2Framer).ReadFrame(0xc000b42ee0, 0xc000a2b9e0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/h2_bundle.go:1735 +0xa5
net/http.(*http2clientConnReadLoop).run(0xc000a4dfa8, 0x0, 0x0)
    /usr/local/go/src/net/http/h2_bundle.go:8331 +0xd8
net/http.(*http2ClientConn).readLoop(0xc000582a80)
    /usr/local/go/src/net/http/h2_bundle.go:8253 +0x6f
created by net/http.(*http2Transport).newClientConn
    /usr/local/go/src/net/http/h2_bundle.go:7217 +0x6c5

In Dex logs error: Response: {"error":"invalid_request","error_description":"Refresh token is invalid or has already been claimed by another client."}

Steps To Reproduce

I don't know how reproduce this cases. Users only use kubectl/k9s. kubectl/k9s via oidc may brake after 1,2,4 days, but may brakes after 45 days.

k8s APIServer options:

--oidc-issuer-url=https://k8s-login.example.ru
--oidc-client-id=gangway
--oidc-groups-claim=groups
--oidc-username-claim=email

our setup: k8s-apiserver + dex (in oidc mode) + oidc.example.com (our custom oidc server) + gangway (for present kubeconfigs to users) + oauth2-proxy/k8s-dashboard. We hasn't problem in tandem: dex + oauth2-proxy/k8s-dashboard. Problems only with personal kubeconfigs, that uses in kubectl/k9s.

Additional Information

Kubernetes v1.19.7 Dex 2.30.2 heptio/gangway 3.3.0 kubectl 1.19..1.22

Configuration

# cat /etc/dex/config.yaml 
connectors:
- config:
    basicAuthUnsupported: true
    claimMapping: null
    clientID: kubernetes
    clientSecret: REDACTED
    insecureSkipEmailVerified: true
    issuer: https://oidc.example.ru/
    redirectURI: https://k8s-login.example.ru/callback
    scopes:
    - profile
    - email
    - offline_access
  id: example-oidc
  name: example-oidc
  type: oidc
expiry:
  idTokens: 1440m
  refreshTokens:
    absoluteLifetime: 876000h
    reuseInterval: 3s
    validIfNotUsedFor: 4392h
issuer: https://k8s-login.example.ru
logger:
  level: debug
oauth2:
  AlwaysShowLoginScreen: false
  skipApprovalScreen: true
staticClients:
- id: gangway
  name: gangway
  redirectURIs:
  - https://k8s.example.ru/callback
  - https://k8s-oauth2-proxy.example.ru/oauth2/callback
  secret: REDACTED
storage:
  config:
    inCluster: true
  type: kubernetes
web:
  http: 0.0.0.0:5556/

Logs

No response

nabokihms commented 2 years ago

Hello, Nikolay!

Some thoughts:

  1. This is a problem on the side of the oidc.example.ru. Provided response {"error":"invalid_grant"} is what oid.example.com responded to Dex. Could you check its logs to find the actual reason for a problem, because the invalid_grant error has a lot of meanings in the OIDC world?

  2. Dex rotates refresh token on every refresh request in terms of security, thus no one can steal the refresh token and use it forever. It seems that kubectl has problems persisting credentials back to the kubeconfig file sometimes.

    You can try to disable refresh tokens rotation and see if it helps:

    expiry:
  idTokens: 1440m
  refreshTokens:
    disableRotation: true
    absoluteLifetime: 876000h
    validIfNotUsedFor: 4392h
nixon89 commented 2 years ago

Hello, @nabokihms We lived 2 months with the disableRotation: true setting enabled in dex

Subjectively, there are fewer problems with non-working configs, but they have not completely disappeared.

I did the following experiment:

  1. got a personal config for test user
  2. looked at what happened in CRD refreshtokens.dex.coreos.com in the cluster. The connectorData field was filled in correctly - the testuser's refresh token from found the provider's oidc in base64
  3. made a simple call to the cluster via kubectl (everything was successful), without errors
  4. waited until 14 days have passed since the release of the personal config, and don't use this config for test user
  5. made a simple call to the cluster via kubectl (everything was successful), there were no errors
  6. waited another 18 days (because the value of validIfNotUsedFor for our oidc provider is 15 days, but in dex we still have the value validIfNotUsedFor: 4392h set), and don't use this config for test user
  7. again performed a simple call to the cluster through kubectl (got an error): 7.1 kubectl - Internal Server Error Response: {"error":"invalid_request"} 7.2 dex - level=error msg="failed to refresh identity: oidc: failed to get refresh token: oauth2: cannot fetch token: 400 Bad Request\nResponse: {\"error\":\"invalid_grant\"}"
  8. looked again at CRD refreshtokens.dex.coreos.com - and for some reason there was not even a connectorData field there for test user.

Is this the correct work of dex? I mean even with disableRotation:true in dex, the refresh token from the oidc provider was removed from the connectorData field) I expected that dex itself received a new refresh token from an external oidc provider and replace old token to new token in CRD refreshtokens.dex.coreos.com in connectorData field for test user

nabokihms commented 2 years ago

It seems that there is a problem with the concurrent call of the Refresh method of the connector. I will provide a detailed description of the problem and possible solutions.

ansh-lehri commented 2 years ago

Hello everyone, I am also facing similar kind of issue with different error. I am generating id-token and refresh-token using dex for kube-oidc-proxy check to Kubernetes GKE server. On generating new tokens, everything works fine and I am able to perform kubectl commands which first get authenticated at kube-oidc-proxy server and then get forwarded to kubernetes API server on GKE.

But once id-token gets expired, refresh-token should work and generate new id-token but running kubectl commands with expired token gives

Unable to connect to the server: Get "https://dex.xyz:32000/.well-known/openid-configuration": x509: certificate signed by unknown authority which is strange to me as new tokens always work and no error comes regarding unknown authority, but expired token give this where refresh-token should have worked... I have already entered root-ca config in oidc-proxy configurations and in dex-authenticator(in place of gangway) configurations

I am using google connector and kubernetes storage.

This is my configuration file: `issuer: https://dex.xyz:32000 storage: type: kubernetes config: inCluster: true web: https: 0.0.0.0:5556 tlsCert: /etc/dex/tls.crt tlsKey: /etc/dex/tls.key connectors:

staticClients:

ansh-lehri commented 2 years ago

It seems that there is a problem with the concurrent call of the Refresh method of the connector. I will provide a detailed description of the problem and possible solutions.

@nabokihms can you share your solution please?

nabokihms commented 2 years ago

Hello, @ansh-lehri. I think your problem differs from the one @nixon89 has. Kubectl fails to request Dex because of the TLS certificate.

The first step is to check auth flags in your kubeconfig

users:
- name: username
  user:
    auth-provider:
      config:
        ...
        idp-issuer-url: https://dex.example.com/
        idp-certificate-authority-data: ...

Which certificate will you get if you try to access the Dex URL? Is it returns a valid TLS certificate? Is the idp-certificate-authority/idp-certificate-authority-data provided/required?

sgremyachikh commented 2 years ago

@nabokihms it looks like we are facing the same problem as described by @nixon89, it is reproduced from time to time. Do you have any news about possible solutions?

nabokihms commented 2 years ago

@sgremyachikh I tried to describe the problem in detail in the linked issue #2547. I hope when we come up with a solution for calling refresh multiple times, your situation will also be fixed.