Open sOblivionsCall opened 5 years ago
adding more info:
$ kubectl describe pod nginx
Name: nginx
Namespace: default
Node: yzen-erver/10.1.1.20
Start Time: Thu, 04 Apr 2019 21:42:21 -0500
Labels: name=nginx
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"name":"nginx"},"name":"nginx","namespace":"default"},"spec":{"cont...
Status: Running
IP: 10.42.24.48
Containers:
nginx:
Container ID: docker://fe6e7415c80792a70b52aafa394df5355e2e6316fc2cbf92b811233425aef68f
Image: sameersbn/nextcloud:13.0.4
Image ID: docker-pullable://sameersbn/nextcloud@sha256:6076ff7d75607439dd8234464b9526a3e63f860c0aa46677800f32f606d20e22
Port: 80/TCP
Host Port: 0/TCP
Args:
app:nginx
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 05 Apr 2019 18:16:40 -0500
Finished: Fri, 05 Apr 2019 18:17:10 -0500
Ready: False
Restart Count: 410
Liveness: http-get http://:80/ delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
NEXTCLOUD_URL: http://cloud.example.com
NEXTCLOUD_PHP_FPM_HOST: nextcloud
NEXTCLOUD_PHP_FPM_PORT: 9000
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-g7sdp (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-g7sdp:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-g7sdp
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 54m (x394 over 20h) kubelet, yzen-erver Container image "sameersbn/nextcloud:13.0.4" already present on machine
Warning Unhealthy 29m (x1207 over 20h) kubelet, yzen-erver Liveness probe failed: HTTP probe failed with statuscode: 400
Warning BackOff 4m45s (x4966 over 20h) kubelet, yzen-erver Back-off restarting failed container
$ docker version
Client:
Version: 18.09.4
API version: 1.39
Go version: go1.10.8
Git commit: d14af54266
Built: Wed Mar 27 18:35:44 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.4
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: d14af54
Built: Wed Mar 27 18:01:48 2019
OS/Arch: linux/amd64
Experimental: false
$ docker info
Containers: 13
Running: 1
Paused: 0
Stopped: 12
Images: 28
Server Version: 18.09.4
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-47-generic
Operating System: Ubuntu 18.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 15.66GiB
Name: Workstation
ID: PG5S:HKRP:IM4O:JVOL:BLPW:CL2H:Q46C:PYF4:K3ZQ:3AHC:LHAN:4EAS
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: No swap limit support
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.7-rancher1", GitCommit:"96573de545affa0a5db720a830b5d2bdf6e80b3b", GitTreeState:"clean", BuildDate:"2019-03-27T02:08:41Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
for anyone who comes across this later on, changing the liveness probe to 5 seconds for the nginx timeout seems to have resolved the issue. I must have just had a latency issue of some kind.:
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
timeoutSeconds: 5
Scratch that, issue is not resolved:
$ kubectl describe pod nginx
Name: nginx
Namespace: default
Node: yzen-erver/10.1.1.20
Start Time: Sun, 14 Apr 2019 13:18:10 -0500
Labels: name=nginx
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"name":"nginx"},"name":"nginx","namespace":"default"},"spec":{"cont...
Status: Running
IP: 10.42.198.234
Containers:
nginx:
Container ID: docker://b8e31b7322189e5fb2f0834eb1fb7eee12abd06ea21e185101ac505bfa9cfe02
Image: sameersbn/nextcloud:13.0.4
Image ID: docker-pullable://sameersbn/nextcloud@sha256:6076ff7d75607439dd8234464b9526a3e63f860c0aa46677800f32f606d20e22
Port: 80/TCP
Host Port: 0/TCP
Args:
app:nginx
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 14 Apr 2019 13:22:03 -0500
Finished: Sun, 14 Apr 2019 13:22:32 -0500
Ready: False
Restart Count: 6
Liveness: http-get http://:80/ delay=5s timeout=5s period=10s #success=1 #failure=3
Environment:
NEXTCLOUD_URL: http://cloud.example.com
NEXTCLOUD_PHP_FPM_HOST: nextcloud
NEXTCLOUD_PHP_FPM_PORT: 9000
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-g7sdp (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-g7sdp:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-g7sdp
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 5m53s default-scheduler Successfully assigned default/nginx to yzen-erver
Normal Pulled 4m21s (x4 over 5m47s) kubelet, yzen-erver Container image "sameersbn/nextcloud:13.0.4" already present on machine
Normal Created 4m21s (x4 over 5m47s) kubelet, yzen-erver Created container
Normal Killing 4m21s (x3 over 5m21s) kubelet, yzen-erver Killing container with id docker://nginx:Container failed liveness probe.. Container will be killed and recreated.
Normal Started 4m20s (x4 over 5m47s) kubelet, yzen-erver Started container
Warning Unhealthy 4m11s (x10 over 5m41s) kubelet, yzen-erver Liveness probe failed: HTTP probe failed with statuscode: 400
Warning BackOff 39s (x11 over 3m21s) kubelet, yzen-erver Back-off restarting failed container
It is very odd that the pod behaves perfectly find if i remove the liveness probe completely.
Not much to go on here but i'm adding anything i think will help. Unfortunately the container will not stay around long enough to exec into it. I am running the kubernetes pods as defined with default values on all but the nextcloud pod. Only change is below:
This is just a temporary setup until i get everything working as needed then i will tighten security ect. Relevant info below:
I'm happy to get other information for you as needed but i'm not sure what else can help.