Issues with stern / kubectl log {pod} -f

[torhagl]

It seems that logging with the follow flag doesn't work. If I use stern it's not able to match any pods either.

Setup: AKS cluster. K8s version 1.14.8. Cert manager for TLS certificates. NGINX for ingress. kube-oidc-proxy deployed from the helm chart within this repo.

Expected behaviour:

Actual behaviour:

Please let me know if you need more information regarding this issue. :)

Forgot to add my role / rolebinding as well:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: log-reader
  namespace: oidc-proxy
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/exec"]
  verbs: ["get", "list", "watch", "create"]

And the rolebinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-log-binding
  namespace: oidc-proxy
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: log-reader
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: torh@stacc.com

[JoshVanL]

/assign

[torhagl]

Closing this issue as it was a permissions issue, not an issue with the project. Solved using these permissions:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: test-ns-superadmin
  namespace: test-ns
rules:
- apiGroups:
    - "*"
  resources:
    - "*"
  verbs: 
    - "*"

Will have to dig deeper to figure out exactly which permissions are needed for this case.

[guruchatti]

I could not get tailing of logs in kubectl or stern to work with the role

Setup: EKS cluster. K8s version 1.17. kube-oidc-proxy deployed from the helm chart within this repo.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: test-role
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]

When I run the command kubectl logs <podname> -f The command hangs with no response.

when I checked the verbose logs by using the command kubectl logs <podname> -f -v=10 I could see a curl request being sent to curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.19.3 (darwin/amd64) kubernetes/1e11e4a" 'https://<apiserverURL>/api/v1/namespaces/default/pods/health-abcd/log?container=health&follow=true'

But never gets any response. Were you able to get it working or had to make any changes in the config?

[torhagl]

@ChattiGuruRaghavendra I got it working by doing the change to the role mentioned in my last comment above. Do you have verbose logging on kube-oidc-proxy? Can you check what you are getting there?

Also - are you able to do things such as kubectl get pods and other similar commands? If that's not the case, is your clusterrolebinding set up correctly to your username / group?

[guruchatti]

Hi @torhagl ,

I am able to run all the other kubectl commands on all the namespaces. The only thing that doesnot work is log tailing.

Below is the verbose log I could get from oidc-proxy pod

command run: stern -n kube-system health-abcd

Verbose log from oidc-proxy

oidc-proxy-5cd94fcff4-bgfl6 oidc-proxy I0226 02:58:59.250393 1 handlers.go:51] authenticated request: 100.108.0.13:52836 oidc-proxy-5cd94fcff4-bgfl6 oidc-proxy I0226 02:58:59.250539 1 round_trippers.go:423] curl -k -v -XGET -H "Impersonate-Group: oidc:<Group-name>" -H "Impersonate-Group: oidc:<Group-name>" -H "Impersonate-Group: oidc:<Group-name>" -H "Impersonate-Group: oidc:<Group-name>" -H "Impersonate-Group: system:authenticated" -H "X-Real-Ip: 192.168.130.162" -H "X-Forwarded-Port: 443" -H "Impersonate-User: oidc:<user-name>" -H "X-Original-Uri: /api/v1/namespaces/kube-system/pods?watch=true" -H "Authorization: Bearer <token>" -H "Accept: application/json, */*" -H "X-Forwarded-Host: <oidc-url>" -H "User-Agent: stern/v0.0.0 (darwin/amd64) kubernetes/$Format" -H "Accept-Encoding: gzip" -H "X-Forwarded-For: 192.168.132.162, 100.108.0.13" -H "X-Forwarded-Proto: https" -H "X-Scheme: https" -H "X-Request-Id: 092d3754e6b497c85afbd44b38175459" 'https://10.100.0.1:443/api/v1/namespaces/kube-system/pods?watch=true' oidc-proxy-5cd94fcff4-bgfl6 oidc-proxy I0226 02:58:59.254568 1 round_trippers.go:443] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/pods?watch=true 200 OK in 4 milliseconds oidc-proxy-5cd94fcff4-bgfl6 oidc-proxy I0226 02:58:59.254582 1 round_trippers.go:449] Response Headers: oidc-proxy-5cd94fcff4-bgfl6 oidc-proxy I0226 02:58:59.254585 1 round_trippers.go:452] Content-Type: application/json oidc-proxy-5cd94fcff4-bgfl6 oidc-proxy I0226 02:58:59.254588 1 round_trippers.go:452] Date: Fri, 26 Feb 2021 02:58:59 GMT oidc-proxy-5cd94fcff4-bgfl6 oidc-proxy I0226 02:58:59.254591 1 round_trippers.go:452] Cache-Control: no-cache, private

and after waiting for a long time, the request failed with the error: failed to set up watch: failed to set up watch: Get https://<oidc-url>/api/v1/namespaces/kube-system/pods?watch=true: unexpected EOF

[2ZZ]

I just hit this issue on v0.1.1 however it looks to be fixed in v0.3.0 👍