search

projectcalico / calico

Cloud native networking and network security
https://docs.tigera.io/calico/latest/about/
Apache License 2.0
6.01k stars 1.34k forks source link

NetworkPlugin cni failed #1689

Closed fatg1988 closed 6 years ago

fatg1988 commented 6 years ago

https://docs.projectcalico.org/v3.0/releases#v3.0.2

This manifest includes the following component versions:

calico/node:v3.0.2

calico/cni:v2.0.0

calico/kube-controllers:v2.0.0

kubelet v1.8.7 Container runtime docker initialized, version: 1.13.1, apiVersion: 1.26.0

docker_sandbox.go:343] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "default-http-backend-b9d8594c6-p2xlv_ingress-nginx": CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container "bbdfc328b4e47644b85868942b43a10fc37ebe000c4dea3be5301bf961f80b42"
Feb 12 19:13:25  kubelet[28371]: W0212 19:13:25.561515   28371 cni.go:265] CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container "bbdfc328b4e47644b85868942b43a10fc37ebe000c4dea3be5301bf961f80b42"

 [INFO][28563] calico-ipam.go 258: Releasing address using handleID ContainerID="bbdfc328b4e47644b85868942b43a10fc37ebe000c4dea3be5301bf961f80b42" HandleID="k8s-pod-network.bbdfc328b4e47644b85868942b43a10fc37ebe000c4dea3be5301bf961f80b42" Workload="77--110--k8s-default--http--backend--b9d8594c6--p2xlv-eth0"
Feb 12 19:13:26 77-110 kubelet[28371]: 2018-02-12 19:13:26.811 [INFO][28563] ipam.go 676: Releasing all IPs with handle 'k8s-pod-network.bbdfc328b4e47644b85868942b43a10fc37ebe000c4dea3be5301bf961f80b42'
Feb 12 19:13:26 77-110 kubelet[28371]: 2018-02-12 19:13:26.814 [WARNING][28563] calico-ipam.go 265: Asked to release address but it doesn't exist. Ignoring ContainerID="bbdfc328b4e47644b85868942b43a10fc37ebe000c4dea3be5301bf961f80b42" HandleID="k8s-pod-network.bbdfc328b4e47644b85868942b43a10fc37ebe000c4dea3be5301bf961f80b42" Workload="77--110--k8s-default--http--backend--b9d8594c6--p2xlv-eth0"
Feb 12 19:13:26  kubelet[28371]: 2018-02-12 19:13:26.815 [INFO][28563] calico-ipam.go 276: Releasing address using workloadID ContainerID="bbdfc328b4e47644b85868942b43a10fc37ebe000c4dea3be5301bf961f80b42" HandleID="k8s-pod-network.bbdfc328b4e47644b85868942b43a10fc37ebe000c4dea3be5301bf961f80b42" Workload="77--110--k8s-default--http--backend--b9d8594c6--p2xlv-eth0"
Feb 12 19:13:26  kubelet[28371]: 2018-02-12 19:13:26.815 [INFO][28563] ipam.go 676: Releasing all IPs with handle 'ingress-nginx.default-http-backend-b9d8594c6-p2xlv'
Feb 12 19:13:26  kubelet[28371]: 2018-02-12 19:13:26.820 [INFO][28537] k8s.go 382: Teardown processing complete. ContainerID="bbdfc328b4e47644b85868942b43a10fc37ebe000c4dea3be5301bf961f80b42"
Feb 12 19:13:26  kubelet[28371]: W0212 19:13:26.861715   28371 docker_sandbox.go:343] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "default-http-backend-b9d8594c6-p2xlv_ingress-nginx": Unexpected command output Device "eth0" does not exist.
Feb 12 19:13:26  kubelet[28371]: with error: exit status 1
Feb 12 19:13:26  kubelet[28371]: W0212 19:13:26.862661   28371 docker_sandbox.go:343] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "default-http-backend-b9d8594c6-p2xlv_ingress-nginx": CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container "bea62e76e79e9d5df702ffc57ac09854ede8f16499801476ea09fa5037b7180e"
Feb 12 19:13:26  kubelet[28371]: W0212 19:13:26.863384   28371 cni.go:265] CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container "bea62e76e79e9d5df702ffc57ac09854ede8f16499801476ea09fa5037b7180e"
tmjd commented 6 years ago

It would really help if you could fill out the Issue template that should have pre-populated the issue when you selected "New Issue". Here it is again, please fill out as much as you can as it will give context to the problem.

<!--- Provide a general summary of the issue in the Title above -->

## Expected Behavior
<!--- If you're describing a bug, tell us what should happen -->
<!--- If you're suggesting a change/improvement, tell us how it should work -->

## Current Behavior
<!--- If describing a bug, tell us what happens instead of the expected behavior -->
<!--- If suggesting a change/improvement, explain the difference from current behavior -->

## Possible Solution
<!--- Not obligatory, but suggest a fix/reason for the bug, -->
<!--- or ideas how to implement the addition or change -->

## Steps to Reproduce (for bugs)
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
1.
2.
3.
4.

## Context
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Calico version
* Orchestrator version (e.g. kubernetes, mesos, rkt):
* Operating System and version:
* Link to your project (optional):

That being said you should take a look at kubectl describe pod pod-name-xxxx as it looks like the namespace for the pod is being removed before the cni plugin runs. You could also change the log level in the CNI config to try to get more information, see https://docs.projectcalico.org/v3.0/reference/cni-plugin/configuration#logging if you need help with that.

caseydavenport commented 6 years ago

Feb 12 19:13:26 kubelet[28371]: W0212 19:13:26.862661 28371 docker_sandbox.go:343] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "default-http-backend-b9d8594c6-p2xlv_ingress-nginx": CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container "bea62e76e79e9d5df702ffc57ac09854ede8f16499801476ea09fa5037b7180e"

This is a commonly seen warning during normal operation - it's produced by the kubelet because the status mechanism is asynchronous to pod lifecycle, and sometimes races.

The log itself doesn't mean that anything went wrong though.

Feb 12 19:13:26 kubelet[28371]: 2018-02-12 19:13:26.820 [INFO][28537] k8s.go 382: Teardown processing complete. ContainerID="bbdfc328b4e47644b85868942b43a10fc37ebe000c4dea3be5301bf961f80b42"

This log means that Calico CNI did in fact complete successfully.

I'm going to close this for now, but please open another issue if you're encountering other problems.