Can not retrieve logs or exec in non-master node'pods

[pain301]

Versions

kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:36:44Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

  Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:39:04Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:31:33Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: vagrant + virtualBox master: 192.168.1.10 worker1: 192.168.1.20 worker2: 192.168.1.30

• OS (e.g. from /etc/os-release):

  
  NAME="CentOS Linux"
  VERSION="7 (Core)"
  ID="centos"
  ID_LIKE="rhel fedora"
  VERSION_ID="7"
  PRETTY_NAME="CentOS Linux 7 (Core)"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:centos:centos:7"
  HOME_URL="https://www.centos.org/"
  BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"

- **Kernel** (e.g. `uname -a`):

Linux master 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

## What happened?
1. first step: `kubectl get pods -n kube-system -o wide` and get result as follows:

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES coredns-86c58d9df4-q9xff 1/1 Running 0 17m 10.32.0.3 master coredns-86c58d9df4-zws5r 1/1 Running 0 17m 10.32.0.4 master etcd-master 1/1 Running 0 16m 10.0.2.15 master kube-apiserver-master 1/1 Running 0 17m 10.0.2.15 master kube-controller-manager-master 1/1 Running 0 16m 10.0.2.15 master kube-proxy-ndp8r 1/1 Running 0 17m 10.0.2.15 master kube-proxy-qxcb6 1/1 Running 0 15m 10.0.2.15 worker1 kube-scheduler-master 1/1 Running 0 16m 10.0.2.15 master weave-net-8rp5f 2/2 Running 1 15m 10.0.2.15 worker1 weave-net-p9zg2 2/2 Running 0 17m 10.0.2.15 master

2. second step: I want to retrieve logs of pod `kube-proxy-qxcb6` which sechduled on node worker1 by  running `kubectl logs kube-proxy-qxcb6  -n kube-system`, and get result as follows:

Error from server (NotFound): the server could not find the requested resource ( pods/log kube-proxy-qxcb6)

## What you expected to happen?
I want to retrieve logs of pod which run on non-master node.

## How to reproduce it (as minimally and precisely as possible)?
master

sudo kubeadm init --apiserver-advertise-address 192.168.1.10 --pod-network-cidr 10.244.0.0/16

mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config

worker

sudo kubeadm join ....

## Anything else we need to know?
node status

NAME STATUS ROLES AGE VERSION master Ready master 29m v1.13.1 worker1 Ready 26m v1.13.1

component status

NAME STATUS MESSAGE ERROR scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}

**exec command `kubectl logs kube-proxy-qxcb6  -n kube-system -v=9`**

I1226 17:09:36.058615 13961 loader.go:359] Config loaded from file /home/vagrant/.kube/config I1226 17:09:36.059959 13961 loader.go:359] Config loaded from file /home/vagrant/.kube/config I1226 17:09:36.070725 13961 loader.go:359] Config loaded from file /home/vagrant/.kube/config I1226 17:09:36.071156 13961 round_trippers.go:419] curl -k -v -XGET -H "Accept: application/json, /" -H "User-Agent: kubectl/v1.13.1 (linux/amd64) kubernetes/eec55b9" 'https://192.168.1.10:6443/api/v1/namespaces/kube-system/pods/kube-proxy-qxcb6' I1226 17:09:36.083661 13961 round_trippers.go:438] GET https://192.168.1.10:6443/api/v1/namespaces/kube-system/pods/kube-proxy-qxcb6 200 OK in 12 milliseconds I1226 17:09:36.083684 13961 round_trippers.go:444] Response Headers: I1226 17:09:36.083691 13961 round_trippers.go:447] Content-Type: application/json I1226 17:09:36.083697 13961 round_trippers.go:447] Content-Length: 3889 I1226 17:09:36.083703 13961 round_trippers.go:447] Date: Wed, 26 Dec 2018 17:09:36 GMT I1226 17:09:36.083760 13961 request.go:942] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"kube-proxy-qxcb6","generateName":"kube-proxy-","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/pods/kube-proxy-qxcb6","uid":"2bb5fcab-092d-11e9-8a14-525400c042d5","resourceVersion":"651","creationTimestamp":"2018-12-26T16:42:02Z","labels":{"controller-revision-hash":"6467f97bf9","k8s-app":"kube-proxy","pod-template-generation":"1"},"annotations":{"scheduler.alpha.kubernetes.io/critical-pod":""},"ownerReferences":[{"apiVersion":"apps/v1","kind":"DaemonSet","name":"kube-proxy","uid":"d43316ff-092c-11e9-8a14-525400c042d5","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"kube-proxy","configMap":{"name":"kube-proxy","defaultMode":420}},{"name":"xtables-lock","hostPath":{"path":"/run/xtables.lock","type":"FileOrCreate"}},{"name":"lib-modules","hostPath":{"path":"/lib/modules","type":""}},{"name":"kube-proxy-token-pjgkl","secret":{"secretName":"kube-proxy-token-pjgkl","defaultMode":420}}],"containers":[{"name":"kube-proxy","image":"k8s.gcr.io/kube-proxy:v1.13.1","command":["/usr/local/bin/kube-proxy","--config=/var/lib/kube-proxy/config.conf","--hostname-override=$(NODE_NAME)"],"env":[{"name":"NODE_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.nodeName"}}}],"resources":{},"volumeMounts":[{"name":"kube-proxy","mountPath":"/var/lib/kube-proxy"},{"name":"xtables-lock","mountPath":"/run/xtables.lock"},{"name":"lib-modules","readOnly":true,"mountPath":"/lib/modules"},{"name":"kube-proxy-token-pjgkl","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"privileged":true,"procMount":"Default"}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"kube-proxy","serviceAccount":"kube-proxy","nodeName":"worker1","hostNetwork":true,"securityContext":{},"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchFields":[{"key":"metadata.name","operator":"In","values":["worker1"]}]}]}}},"schedulerName":"default-scheduler","tolerations":[{"key":"CriticalAddonsOnly","operator":"Exists"},{"operator":"Exists"},{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute"},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute"},{"key":"node.kubernetes.io/disk-pressure","operator":"Exists","effect":"NoSchedule"},{"key":"node.kubernetes.io/memory-pressure","operator":"Exists","effect":"NoSchedule"},{"key":"node.kubernetes.io/unschedulable","operator":"Exists","effect":"NoSchedule"},{"key":"node.kubernetes.io/network-unavailable","operator":"Exists","effect":"NoSchedule"}],"priorityClassName":"system-node-critical","priority":2000001000,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-12-26T16:42:02Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-12-26T16:42:04Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-12-26T16:42:04Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-12-26T16:42:02Z"}],"hostIP":"10.0.2.15","podIP":"10.0.2.15","startTime":"2018-12-26T16:42:02Z","containerStatuses":[{"name":"kube-proxy","state":{"running":{"startedAt":"2018-12-26T16:42:03Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"mirrorgooglecontainers/kube-proxy:v1.13.1","imageID":"docker-pullable://mirrorgooglecontainers/kube-proxy@sha256:95c8d3d7a4801fb6f01adaa9791e55f15f2cab320b4556fc0ed9d5876cad8364","containerID":"docker://05ca3779348dc9241b042986e8ce9b3679b8eb0ec60e3f6f805c02972b8d43d9"}],"qosClass":"BestEffort"}} I1226 17:09:36.097848 13961 loader.go:359] Config loaded from file /home/vagrant/.kube/config I1226 17:09:36.098293 13961 round_trippers.go:419] curl -k -v -XGET -H "Accept: application/json, /" -H "User-Agent: kubectl/v1.13.1 (linux/amd64) kubernetes/eec55b9" 'https://192.168.1.10:6443/api/v1/namespaces/kube-system/pods/kube-proxy-qxcb6/log' I1226 17:09:36.104363 13961 round_trippers.go:438] GET https://192.168.1.10:6443/api/v1/namespaces/kube-system/pods/kube-proxy-qxcb6/log 404 Not Found in 6 milliseconds I1226 17:09:36.104513 13961 round_trippers.go:444] Response Headers: I1226 17:09:36.104522 13961 round_trippers.go:447] Date: Wed, 26 Dec 2018 17:09:36 GMT I1226 17:09:36.104528 13961 round_trippers.go:447] Content-Type: application/json I1226 17:09:36.104534 13961 round_trippers.go:447] Content-Length: 246 I1226 17:09:36.104560 13961 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the server could not find the requested resource ( pods/log kube-proxy-qxcb6)","reason":"NotFound","details":{"name":"kube-proxy-qxcb6","kind":"pods/log"},"code":404} I1226 17:09:36.104842 13961 helpers.go:198] server response object: [{ "metadata": {}, "status": "Failure", "message": "the server could not find the requested resource ( pods/log kube-proxy-qxcb6)", "reason": "NotFound", "details": { "name": "kube-proxy-qxcb6", "kind": "pods/log" }, "code": 404 }] F1226 17:09:36.104862 13961 helpers.go:116] Error from server (NotFound): the server could not find the requested resource ( pods/log kube-proxy-qxcb6) and get something as follows:

[pain301]

Resolved.

[fabriziopandini]

@pain400 sorry for the lag in the answer Happy to know you fixed your problem!

[LyKos4]

Resolved.

What was done to resolve issue?