Open perezjasonr opened 3 years ago
Some followup material, after testing things out, now I really don't know where that kubectl command is getting its credentials, it doesn't appear to be its own service account. I am fortunate enough to have an istio proxy sidecar running, which mounts the same SA token:
image: <our private registry>/istio-1.7/proxyv2:1.7.2
imagePullPolicy: Always
name: istio-proxy
<other properties removed for brevity>
volumeMounts:
- mountPath: /var/run/secrets/istio
name: istiod-ca-cert
- mountPath: /var/lib/istio/data
name: istio-data
- mountPath: /etc/istio/proxy
name: istio-envoy
- mountPath: /etc/istio/pod
name: istio-podinfo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kritis-preinstall-serviceaccount-token-ttgdp
readOnly: true
note the last mount path is the kritis preinstall token. So I exec into there and using that token I am authorized just fine:
curl -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://kubernetes/apis/certificates.k8s.io -vvv
* Trying 10.96.0.1...
* TCP_NODELAY set
* Connected to kubernetes (10.96.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=kube-apiserver
* start date: Nov 27 18:06:37 2020 GMT
* expire date: Nov 27 18:07:39 2021 GMT
* subjectAltName: host "kubernetes" matched cert's "kubernetes"
* issuer: CN=kubernetes
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* Using Stream ID: 1 (easy handle 0x5651b9a13a70)
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /apis/certificates.k8s.io HTTP/2
> Host: kubernetes
> User-Agent: curl/7.61.1
> Accept: */*
> Authorization: Bearer <redacted>
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.3 (OUT), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/2 200
< cache-control: no-cache, private
< content-type: application/json
< content-length: 297
< date: Mon, 30 Nov 2020 13:55:12 GMT
<
* TLSv1.3 (IN), TLS app data, [no content] (0):
{
"kind": "APIGroup",
"apiVersion": "v1",
"name": "certificates.k8s.io",
"versions": [
{
"groupVersion": "certificates.k8s.io/v1beta1",
"version": "v1beta1"
}
],
"preferredVersion": {
"groupVersion": "certificates.k8s.io/v1beta1",
"version": "v1beta1"
}
* Connection #0 to host kubernetes left intact
Same IP is resolved and everything, I even went to the certificates api group just to list the one that the kubectl call was trying to make. So I guess my question becomes, where is this kubectl call from within pre-install getting its credentials from?
Expected Behavior
successful kubectl call from within kritis preinstall container
Actual Behavior
Steps to Reproduce the Problem
Environment, commands
I used the standalone docs, im really interested in this toolchain so demo-ing for my team.
./setup_grafeas.sh ./setup_kritis.sh
Additional info
basically, it seems like the kubectl call isn't authorized. Do see the service account exists:
as is its clusterrolebinding:
but I cant tell by looking at the golang (im a beginner/intermediate at golang at best) where its getting its kubeconf from or if its trying to use the service account token. It is noteworthy that other things I'm running that have an SA and require kube-api access arent being rejected. Any ideas?