paulmcgoldrick
commented
7 years ago I've been able to confirm this same behavior using 1.5.3_coreos.0 on rkt 1.14 (going to start bumping container os versions again with the 1.5.3, instead of 1.5.2). My test pod is as follows:
---
apiVersion: v1
kind: Pod
metadata:
name: busybox-pod
namespace: default
labels:
run: busybox
spec:
containers:
- image: busybox
command:
- sleep
- "300"
imagePullPolicy: IfNotPresent
name: busybox-container
restartPolicy: AlwaysIt seems like I was in error above assuming the new pod would be using a new network namespace, as the namespace referenced in the unit file is created every time the pod is restarted, but subsequently goes away. Also, it would seem the reporting of the IP is cached, which is leading to the correlation between kubelet-wrapper logs, and the IP "disappearing" from the kubectl describe pod output. Instead, I guess based on my observation of netns monitor, the namespace is removed much more quickly.
Here are the logs of netns monitor as I either manually kill the sleep, or allow it to expire. Note that the deletion occurs less than a minute after the instantiation of the pod, and this behavior is reproducable with any pod in my current systems. The namespaceless pod will continue to exist until it completes or is killed.
add k8s_8c8afe0d-fee5-11e6-a0c7-080027c23b90
delete k8s_8c8afe0d-fee5-11e6-a0c7-080027c23b90
add k8s_8c8afe0d-fee5-11e6-a0c7-080027c23b90
delete k8s_8c8afe0d-fee5-11e6-a0c7-080027c23b90
add k8s_8c8afe0d-fee5-11e6-a0c7-080027c23b90
delete k8s_8c8afe0d-fee5-11e6-a0c7-080027c23b90
add k8s_8c8afe0d-fee5-11e6-a0c7-080027c23b90
delete k8s_8c8afe0d-fee5-11e6-a0c7-080027c23b90
add k8s_8c8afe0d-fee5-11e6-a0c7-080027c23b90
jowy
stable Latest, 1235.9.0, and 1185.3.0 rkt Version: 1.18.0, 1.21, 1.14 matchbox 0.5.0, FLANNEL_IMAGE_TAG=v0.6.2", kubernetes 1.5.2.
When a container in a deployment is restarted, after a short but arbitrary amount of time the container loses its flannel ip.
The log appears as follows:
Upon investigation it appears that the systemd unit file related to the pod is not updated with the new netnsfor the new pod. The IP address disappearance will then coincide with the next kubelet-wrapper maintenance logs such as:
To reproduce, create a deployment with a single container.
ssh into the host the deployments pod has been started on and
After the dead pod is replaced, watch logs for kubectl-wrapper error, and maintenance tasks. IP will be unset when the latter appear.
I'm not quite sure if I've misconfigured something, nor where to go from here. Please let me know if you need more information.