ldimaggi
commented
7 years ago Closed burrsutter closed 6 years ago
ldimaggi
commented
7 years ago 
burrsutter
commented
7 years ago Now it looks like it finally failed https://screencast.com/t/KvpvqBXFPv though Applications still says green https://screencast.com/t/YdUxI9c99Tti
burrsutter
commented
7 years ago Happened on the Run side this time https://screencast.com/t/fxcmw9yOp0
burrsutter
commented
7 years ago LASTSEEN FIRSTSEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE
1h 1h 1 tatooinevertx12-1-deploy Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-1-deploy to ip-172-31-78-9.us-east-2.compute.internal
1h 1h 1 tatooinevertx12-1-deploy Pod spec.containers{deployment} Normal Pulling {kubelet ip-172-31-78-9.us-east-2.compute.internal} pulling image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
1h 1h 1 tatooinevertx12-1-deploy Pod spec.containers{deployment} Normal Pulled {kubelet ip-172-31-78-9.us-east-2.compute.internal} Successfully pulled image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
1h 1h 1 tatooinevertx12-1-deploy Pod spec.containers{deployment} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id 1637b0559753; Security:[seccomp=unconfined]
1h 1h 1 tatooinevertx12-1-deploy Pod spec.containers{deployment} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id 1637b0559753
1h 1h 1 tatooinevertx12-1-k6bhv Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-1-k6bhv to ip-172-31-77-238.us-east-2.compute.internal
1h 1h 1 tatooinevertx12-1-k6bhv Pod spec.containers{vertx} Normal Pulling {kubelet ip-172-31-77-238.us-east-2.compute.internal} pulling image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:c39fe760e149ab0b5f2fadf2eb1642cafd35bb061db19dd95c6ff2507bed71fb"
1h 1h 1 tatooinevertx12-1-k6bhv Pod spec.containers{vertx} Normal Pulled {kubelet ip-172-31-77-238.us-east-2.compute.internal} Successfully pulled image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:c39fe760e149ab0b5f2fadf2eb1642cafd35bb061db19dd95c6ff2507bed71fb"
1h 1h 1 tatooinevertx12-1-k6bhv Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-77-238.us-east-2.compute.internal} Created container with docker id 75ff0fb96800; Security:[seccomp=unconfined]
1h 1h 1 tatooinevertx12-1-k6bhv Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-77-238.us-east-2.compute.internal} Started container with docker id 75ff0fb96800
1h 1h 1 tatooinevertx12-1-k6bhv Pod spec.containers{vertx} Warning Unhealthy {kubelet ip-172-31-77-238.us-east-2.compute.internal} Readiness probe failed: Get http://10.129.28.187:8080/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
33m 33m 1 tatooinevertx12-1-k6bhv Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-77-238.us-east-2.compute.internal} Killing container with docker id 75ff0fb96800: Need to kill pod.
33m 33m 1 tatooinevertx12-1-k6bhv Pod Warning FailedSync {kubelet ip-172-31-77-238.us-east-2.compute.internal} Error syncing pod, skipping: error killing pod: failed to "TeardownNetwork" for "tatooinevertx12-1-k6bhv_burrsutter-stage" with TeardownNetworkError: "Failed to teardown network for pod \"4a259d3b-7172-11e7-ad51-02d7377a4b17\" using network plugins \"cni\": CNI request failed with status 400: 'Failed to execute iptables-restore: exit status 1 (iptables-restore: line 3 failed\n)\n'"
1h 1h 1 tatooinevertx12-1 ReplicationController Normal SuccessfulCreate {replication-controller } Created pod: tatooinevertx12-1-k6bhv
33m 33m 1 tatooinevertx12-1 ReplicationController Normal SuccessfulDelete {replication-controller } Deleted pod: tatooinevertx12-1-k6bhv
1h 1h 1 tatooinevertx12-2-deploy Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-2-deploy to ip-172-31-64-23.us-east-2.compute.internal
1h 1h 1 tatooinevertx12-2-deploy Pod spec.containers{deployment} Normal Pulling {kubelet ip-172-31-64-23.us-east-2.compute.internal} pulling image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
1h 1h 1 tatooinevertx12-2-deploy Pod spec.containers{deployment} Normal Pulled {kubelet ip-172-31-64-23.us-east-2.compute.internal} Successfully pulled image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
1h 1h 1 tatooinevertx12-2-deploy Pod spec.containers{deployment} Normal Created {kubelet ip-172-31-64-23.us-east-2.compute.internal} Created container with docker id e0fd9a09e538; Security:[seccomp=unconfined]
1h 1h 1 tatooinevertx12-2-deploy Pod spec.containers{deployment} Normal Started {kubelet ip-172-31-64-23.us-east-2.compute.internal} Started container with docker id e0fd9a09e538
34m 34m 1 tatooinevertx12-2-deploy Pod spec.containers{deployment} Normal Killing {kubelet ip-172-31-64-23.us-east-2.compute.internal} Killing container with docker id e0fd9a09e538: Need to kill pod.
1h 1h 1 tatooinevertx12-2-zjst4 Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-2-zjst4 to ip-172-31-65-181.us-east-2.compute.internal
40m 1h 8 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Pulling {kubelet ip-172-31-65-181.us-east-2.compute.internal} pulling image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:fe04a998532f71494da0e54850651cb4a154d6b49b2e6a8789422cccc539e9d9"
40m 1h 8 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Pulled {kubelet ip-172-31-65-181.us-east-2.compute.internal} Successfully pulled image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:fe04a998532f71494da0e54850651cb4a154d6b49b2e6a8789422cccc539e9d9"
1h 1h 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-65-181.us-east-2.compute.internal} Created container with docker id a162f4bb46e6; Security:[seccomp=unconfined]
1h 1h 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-65-181.us-east-2.compute.internal} Started container with docker id a162f4bb46e6
37m 1h 146 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Warning Unhealthy {kubelet ip-172-31-65-181.us-east-2.compute.internal} Readiness probe failed: Get http://10.130.27.16:8080/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
37m 1h 10 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Warning Unhealthy {kubelet ip-172-31-65-181.us-east-2.compute.internal} Liveness probe failed: Get http://10.130.27.16:8080/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
59m 59m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-65-181.us-east-2.compute.internal} Killing container with docker id a162f4bb46e6: pod "tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
59m 59m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-65-181.us-east-2.compute.internal} Created container with docker id 208fb93d4a66; Security:[seccomp=unconfined]
59m 59m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-65-181.us-east-2.compute.internal} Started container with docker id 208fb93d4a66
56m 56m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-65-181.us-east-2.compute.internal} Killing container with docker id 208fb93d4a66: pod "tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
56m 56m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-65-181.us-east-2.compute.internal} Created container with docker id 79168fcd4b2f; Security:[seccomp=unconfined]
56m 56m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-65-181.us-east-2.compute.internal} Started container with docker id 79168fcd4b2f
53m 53m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-65-181.us-east-2.compute.internal} Killing container with docker id 79168fcd4b2f: pod "tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
53m 53m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-65-181.us-east-2.compute.internal} Created container with docker id f892c7211b81; Security:[seccomp=unconfined]
53m 53m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-65-181.us-east-2.compute.internal} Started container with docker id f892c7211b81
50m 50m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-65-181.us-east-2.compute.internal} Killing container with docker id f892c7211b81: pod "tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
50m 50m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-65-181.us-east-2.compute.internal} Created container with docker id 303428f5e37f; Security:[seccomp=unconfined]
50m 50m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-65-181.us-east-2.compute.internal} Started container with docker id 303428f5e37f
47m 47m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-65-181.us-east-2.compute.internal} Killing container with docker id 303428f5e37f: pod "tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
47m 47m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-65-181.us-east-2.compute.internal} Created container with docker id 6d3c7cee0bfd; Security:[seccomp=unconfined]
47m 47m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-65-181.us-east-2.compute.internal} Started container with docker id 6d3c7cee0bfd
43m 43m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-65-181.us-east-2.compute.internal} Killing container with docker id 6d3c7cee0bfd: pod "tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
43m 43m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-65-181.us-east-2.compute.internal} Created container with docker id cfcc232e30c8; Security:[seccomp=unconfined]
43m 43m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-65-181.us-east-2.compute.internal} Started container with docker id cfcc232e30c8
40m 40m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-65-181.us-east-2.compute.internal} Killing container with docker id cfcc232e30c8: pod "tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
40m 40m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-65-181.us-east-2.compute.internal} Created container with docker id 5e66914f3fa8; Security:[seccomp=unconfined]
40m 40m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-65-181.us-east-2.compute.internal} Started container with docker id 5e66914f3fa8
37m 37m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-65-181.us-east-2.compute.internal} Killing container with docker id 5e66914f3fa8: pod "tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
34m 37m 15 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Warning BackOff {kubelet ip-172-31-65-181.us-east-2.compute.internal} Back-off restarting failed docker container
34m 37m 15 tatooinevertx12-2-zjst4 Pod Warning FailedSync {kubelet ip-172-31-65-181.us-east-2.compute.internal} Error syncing pod, skipping: failed to "StartContainer" for "vertx" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=vertx pod=tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)"
1h 1h 1 tatooinevertx12-2 ReplicationController Normal SuccessfulCreate {replication-controller } Created pod: tatooinevertx12-2-zjst4
34m 34m 1 tatooinevertx12-2 ReplicationController Normal SuccessfulDelete {replication-controller } Deleted pod: tatooinevertx12-2-zjst4
34m 34m 1 tatooinevertx12-3-blv2m Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-3-blv2m to ip-172-31-78-9.us-east-2.compute.internal
34m 34m 1 tatooinevertx12-3-blv2m Pod Warning FailedSync {kubelet ip-172-31-78-9.us-east-2.compute.internal} Error syncing pod, skipping: failed to "SetupNetwork" for "tatooinevertx12-3-blv2m_burrsutter-stage" with SetupNetworkError: "Failed to setup network for pod \"tatooinevertx12-3-blv2m_burrsutter-stage(0c79264c-7177-11e7-ad51-02d7377a4b17)\" using network plugins \"cni\": CNI request failed with status 400: 'Failed to execute iptables-restore: exit status 1 (iptables-restore: line 3 failed\n)\n'; Skipping pod"
33m 33m 1 tatooinevertx12-3-blv2m Pod spec.containers{vertx} Normal Pulling {kubelet ip-172-31-78-9.us-east-2.compute.internal} pulling image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:fe04a998532f71494da0e54850651cb4a154d6b49b2e6a8789422cccc539e9d9"
33m 33m 1 tatooinevertx12-3-blv2m Pod spec.containers{vertx} Normal Pulled {kubelet ip-172-31-78-9.us-east-2.compute.internal} Successfully pulled image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:fe04a998532f71494da0e54850651cb4a154d6b49b2e6a8789422cccc539e9d9"
33m 33m 1 tatooinevertx12-3-blv2m Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id ed408d29cff3; Security:[seccomp=unconfined]
33m 33m 1 tatooinevertx12-3-blv2m Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id ed408d29cff3
33m 33m 1 tatooinevertx12-3-blv2m Pod spec.containers{vertx} Warning Unhealthy {kubelet ip-172-31-78-9.us-east-2.compute.internal} Readiness probe failed: Get http://10.128.52.247:8080/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
29m 29m 1 tatooinevertx12-3-blv2m Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id ed408d29cff3: Need to kill pod.
34m 34m 1 tatooinevertx12-3-deploy Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-3-deploy to ip-172-31-65-163.us-east-2.compute.internal
34m 34m 1 tatooinevertx12-3-deploy Pod spec.containers{deployment} Normal Pulling {kubelet ip-172-31-65-163.us-east-2.compute.internal} pulling image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
34m 34m 1 tatooinevertx12-3-deploy Pod spec.containers{deployment} Normal Pulled {kubelet ip-172-31-65-163.us-east-2.compute.internal} Successfully pulled image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
34m 34m 1 tatooinevertx12-3-deploy Pod spec.containers{deployment} Normal Created {kubelet ip-172-31-65-163.us-east-2.compute.internal} Created container with docker id 721d3957cc36; Security:[seccomp=unconfined]
34m 34m 1 tatooinevertx12-3-deploy Pod spec.containers{deployment} Normal Started {kubelet ip-172-31-65-163.us-east-2.compute.internal} Started container with docker id 721d3957cc36
33m 33m 1 tatooinevertx12-3-deploy Pod spec.containers{deployment} Normal Killing {kubelet ip-172-31-65-163.us-east-2.compute.internal} Killing container with docker id 721d3957cc36: Need to kill pod.
34m 34m 1 tatooinevertx12-3 ReplicationController Normal SuccessfulCreate {replication-controller } Created pod: tatooinevertx12-3-blv2m
29m 29m 1 tatooinevertx12-3 ReplicationController Normal SuccessfulDelete {replication-controller } Deleted pod: tatooinevertx12-3-blv2m
29m 29m 1 tatooinevertx12-4-0gmpl Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-4-0gmpl to ip-172-31-77-238.us-east-2.compute.internal
29m 29m 1 tatooinevertx12-4-0gmpl Pod spec.containers{vertx} Normal Pulling {kubelet ip-172-31-77-238.us-east-2.compute.internal} pulling image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:18b7c2da193ed00cc0096eb3dd7094048bbff8856ebe932419ed0d8420813fba"
29m 29m 1 tatooinevertx12-4-0gmpl Pod spec.containers{vertx} Normal Pulled {kubelet ip-172-31-77-238.us-east-2.compute.internal} Successfully pulled image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:18b7c2da193ed00cc0096eb3dd7094048bbff8856ebe932419ed0d8420813fba"
29m 29m 1 tatooinevertx12-4-0gmpl Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-77-238.us-east-2.compute.internal} Created container with docker id 461bded8f3c8; Security:[seccomp=unconfined]
29m 29m 1 tatooinevertx12-4-0gmpl Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-77-238.us-east-2.compute.internal} Started container with docker id 461bded8f3c8
29m 29m 1 tatooinevertx12-4-0gmpl Pod spec.containers{vertx} Warning Unhealthy {kubelet ip-172-31-77-238.us-east-2.compute.internal} Readiness probe failed: Get http://10.129.28.189:8080/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
29m 29m 1 tatooinevertx12-4-deploy Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-4-deploy to ip-172-31-65-163.us-east-2.compute.internal
29m 29m 1 tatooinevertx12-4-deploy Pod spec.containers{deployment} Normal Pulling {kubelet ip-172-31-65-163.us-east-2.compute.internal} pulling image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
29m 29m 1 tatooinevertx12-4-deploy Pod spec.containers{deployment} Normal Pulled {kubelet ip-172-31-65-163.us-east-2.compute.internal} Successfully pulled image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
29m 29m 1 tatooinevertx12-4-deploy Pod spec.containers{deployment} Normal Created {kubelet ip-172-31-65-163.us-east-2.compute.internal} Created container with docker id f277c241459f; Security:[seccomp=unconfined]
29m 29m 1 tatooinevertx12-4-deploy Pod spec.containers{deployment} Normal Started {kubelet ip-172-31-65-163.us-east-2.compute.internal} Started container with docker id f277c241459f
29m 29m 1 tatooinevertx12-4-deploy Pod spec.containers{deployment} Normal Killing {kubelet ip-172-31-65-163.us-east-2.compute.internal} Killing container with docker id f277c241459f: Need to kill pod.
29m 29m 1 tatooinevertx12-4-deploy Pod Warning FailedSync {kubelet ip-172-31-65-163.us-east-2.compute.internal} Error syncing pod, skipping: failed to "TeardownNetwork" for "tatooinevertx12-4-deploy_burrsutter-stage" with TeardownNetworkError: "Failed to teardown network for pod \"ac39df2c-7177-11e7-ad51-02d7377a4b17\" using network plugins \"cni\": CNI request failed with status 400: 'Failed to execute iptables-restore: exit status 1 (iptables-restore: line 3 failed\n)\n'"
29m 29m 1 tatooinevertx12-4 ReplicationController Normal SuccessfulCreate {replication-controller } Created pod: tatooinevertx12-4-0gmpl
27m 27m 1 tatooinevertx12-5-deploy Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-5-deploy to ip-172-31-65-163.us-east-2.compute.internal
27m 27m 1 tatooinevertx12-5-deploy Pod spec.containers{deployment} Normal Pulling {kubelet ip-172-31-65-163.us-east-2.compute.internal} pulling image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
27m 27m 1 tatooinevertx12-5-deploy Pod spec.containers{deployment} Normal Pulled {kubelet ip-172-31-65-163.us-east-2.compute.internal} Successfully pulled image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
27m 27m 1 tatooinevertx12-5-deploy Pod spec.containers{deployment} Normal Created {kubelet ip-172-31-65-163.us-east-2.compute.internal} Created container with docker id 3e5469924ff3; Security:[seccomp=unconfined]
27m 27m 1 tatooinevertx12-5-deploy Pod spec.containers{deployment} Normal Started {kubelet ip-172-31-65-163.us-east-2.compute.internal} Started container with docker id 3e5469924ff3
3m 3m 1 tatooinevertx12-5-deploy Pod spec.containers{deployment} Normal Killing {kubelet ip-172-31-65-163.us-east-2.compute.internal} Killing container with docker id 3e5469924ff3: Need to kill pod.
27m 27m 1 tatooinevertx12-5-f01hf Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-5-f01hf to ip-172-31-78-9.us-east-2.compute.internal
4m 27m 8 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Pulling {kubelet ip-172-31-78-9.us-east-2.compute.internal} pulling image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:89eb08efebefdf70d98c2b9df085dacbf7862de1e20eda5a09b249f1d102285a"
4m 27m 8 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Pulled {kubelet ip-172-31-78-9.us-east-2.compute.internal} Successfully pulled image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:89eb08efebefdf70d98c2b9df085dacbf7862de1e20eda5a09b249f1d102285a"
27m 27m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id c51edec13937; Security:[seccomp=unconfined]
27m 27m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id c51edec13937
3m 26m 135 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Warning Unhealthy {kubelet ip-172-31-78-9.us-east-2.compute.internal} Readiness probe failed: Get http://10.128.52.249:8080/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
4m 23m 9 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Warning Unhealthy {kubelet ip-172-31-78-9.us-east-2.compute.internal} Liveness probe failed: Get http://10.128.52.249:8080/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
23m 23m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id c51edec13937: pod "tatooinevertx12-5-f01hf_burrsutter-stage(1061dcb8-7178-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
23m 23m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id 195efbc23129; Security:[seccomp=unconfined]
23m 23m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id 195efbc23129
20m 20m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id 195efbc23129: pod "tatooinevertx12-5-f01hf_burrsutter-stage(1061dcb8-7178-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
20m 20m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id cdfa90c9795c; Security:[seccomp=unconfined]
20m 20m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id cdfa90c9795c
17m 17m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id cdfa90c9795c: pod "tatooinevertx12-5-f01hf_burrsutter-stage(1061dcb8-7178-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
17m 17m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id 1645400dd246; Security:[seccomp=unconfined]
17m 17m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id 1645400dd246
14m 14m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id 1645400dd246: pod "tatooinevertx12-5-f01hf_burrsutter-stage(1061dcb8-7178-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
14m 14m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id f295cd90feb4; Security:[seccomp=unconfined]
14m 14m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id f295cd90feb4
10m 10m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id f295cd90feb4: pod "tatooinevertx12-5-f01hf_burrsutter-stage(1061dcb8-7178-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
10m 10m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id 12628a98d3da; Security:[seccomp=unconfined]
10m 10m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id 12628a98d3da
7m 7m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id 12628a98d3da: pod "tatooinevertx12-5-f01hf_burrsutter-stage(1061dcb8-7178-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
7m 7m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id 1152f97354a5; Security:[seccomp=unconfined]
7m 7m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id 1152f97354a5
4m 4m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id 1152f97354a5: pod "tatooinevertx12-5-f01hf_burrsutter-stage(1061dcb8-7178-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
4m 4m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id e09d8145d083; Security:[seccomp=unconfined]
4m 4m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id e09d8145d083
3m 3m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id e09d8145d083: Need to kill pod.
27m 27m 1 tatooinevertx12-5 ReplicationController Normal SuccessfulCreate {replication-controller } Created pod: tatooinevertx12-5-f01hf
3m 3m 1 tatooinevertx12-5 ReplicationController Normal SuccessfulDelete {replication-controller } Deleted pod: tatooinevertx12-5-f01hf
3m 3m 1 tatooinevertx12-6-4c5nc Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-6-4c5nc to ip-172-31-77-122.us-east-2.compute.internal
3m 3m 1 tatooinevertx12-6-4c5nc Pod spec.containers{vertx} Normal Pulling {kubelet ip-172-31-77-122.us-east-2.compute.internal} pulling image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:77d24be528e3033b56b4aa4a2c557a7c64409c3b1b040fe12687921ce1420176"
3m 3m 1 tatooinevertx12-6-4c5nc Pod spec.containers{vertx} Normal Pulled {kubelet ip-172-31-77-122.us-east-2.compute.internal} Successfully pulled image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:77d24be528e3033b56b4aa4a2c557a7c64409c3b1b040fe12687921ce1420176"
3m 3m 1 tatooinevertx12-6-4c5nc Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-77-122.us-east-2.compute.internal} Created container with docker id d3e4affc495b; Security:[seccomp=unconfined]
3m 3m 1 tatooinevertx12-6-4c5nc Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-77-122.us-east-2.compute.internal} Started container with docker id d3e4affc495b
30s 2m 15 tatooinevertx12-6-4c5nc Pod spec.containers{vertx} Warning Unhealthy {kubelet ip-172-31-77-122.us-east-2.compute.internal} Readiness probe failed: Get http://10.128.19.96:8080/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3m 3m 1 tatooinevertx12-6-deploy Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-6-deploy to ip-172-31-65-163.us-east-2.compute.internal
3m 3m 1 tatooinevertx12-6-deploy Pod spec.containers{deployment} Normal Pulling {kubelet ip-172-31-65-163.us-east-2.compute.internal} pulling image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
3m 3m 1 tatooinevertx12-6-deploy Pod spec.containers{deployment} Normal Pulled {kubelet ip-172-31-65-163.us-east-2.compute.internal} Successfully pulled image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
3m 3m 1 tatooinevertx12-6-deploy Pod spec.containers{deployment} Normal Created {kubelet ip-172-31-65-163.us-east-2.compute.internal} Created container with docker id afff7d7ec789; Security:[seccomp=unconfined]
3m 3m 1 tatooinevertx12-6-deploy Pod spec.containers{deployment} Normal Started {kubelet ip-172-31-65-163.us-east-2.compute.internal} Started container with docker id afff7d7ec789
3m 3m 9 tatooinevertx12-6 ReplicationController Warning FailedCreate {replication-controller } Error creating: pods "tatooinevertx12-6-" is forbidden: exceeded quota: compute-resources, requested: limits.cpu=1,limits.memory=512Mi, used: limits.cpu=2,limits.memory=1Gi, limited: limits.cpu=2,limits.memory=1Gi
3m 3m 1 tatooinevertx12-6 ReplicationController Normal SuccessfulCreate {replication-controller } Created pod: tatooinevertx12-6-4c5nc
1h 1h 1 tatooinevertx12 DeploymentConfig Normal DeploymentCreated {deploymentconfig-controller } Created new replication controller "tatooinevertx12-1" for version 1
1h 1h 1 tatooinevertx12 DeploymentConfig Normal DeploymentCreated {deploymentconfig-controller } Created new replication controller "tatooinevertx12-2" for version 2
34m 34m 1 tatooinevertx12 DeploymentConfig Normal DeploymentCancelled {deployments-controller } tatooinevertx12-2: Deployment "burrsutter-stage/tatooinevertx12-2" cancelled
34m 34m 1 tatooinevertx12 DeploymentConfig Normal ReplicationControllerScaled {deploymentconfig-controller } Scaled replication controller "tatooinevertx12-2" from 1 to 0
34m 34m 1 tatooinevertx12 DeploymentConfig Normal DeploymentCreated {deploymentconfig-controller } Created new replication controller "tatooinevertx12-3" for version 3
29m 29m 1 tatooinevertx12 DeploymentConfig Normal DeploymentCreated {deploymentconfig-controller } Created new replication controller "tatooinevertx12-4" for version 4
27m 27m 1 tatooinevertx12 DeploymentConfig Normal DeploymentCreated {deploymentconfig-controller } Created new replication controller "tatooinevertx12-5" for version 5
3m 3m 1 tatooinevertx12 DeploymentConfig Normal DeploymentCancelled {deploymentconfig-controller } Cancelled deployment "tatooinevertx12-5" superceded by version 6
3m 3m 6 tatooinevertx12 DeploymentConfig Normal DeploymentAwaitingCancellation {deploymentconfig-controller } Deployment of version 6 awaiting cancellation of older running deployments
3m 3m 1 tatooinevertx12 DeploymentConfig Normal DeploymentCancelled {deployments-controller } tatooinevertx12-5: Deployment "burrsutter-stage/tatooinevertx12-5" cancelled
3m 3m 1 tatooinevertx12 DeploymentConfig Normal DeploymentCreated {deploymentconfig-controller } Created new replication controller "tatooinevertx12-6" for version 6
1h 1h 1 yavinvertx1-1-54rfj Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id c0cebab9a629: Need to kill pod.
1h 1h 1 yavinvertx1-1 ReplicationController Normal SuccessfulDelete {replication-controller } Deleted pod: yavinvertx1-1-54rfj
1h 1h 1 yavinvertx1 DeploymentConfig Normal ReplicationControllerScaled {deploymentconfig-controller } Scaled replication controller "yavinvertx1-1" from 1 to 0
LASTSEEN FIRSTSEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE
1h 1h 1 tatooinevertx12-1-deploy Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-1-deploy to ip-172-31-78-9.us-east-2.compute.internal
1h 1h 1 tatooinevertx12-1-deploy Pod spec.containers{deployment} Normal Pulling {kubelet ip-172-31-78-9.us-east-2.compute.internal} pulling image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
1h 1h 1 tatooinevertx12-1-deploy Pod spec.containers{deployment} Normal Pulled {kubelet ip-172-31-78-9.us-east-2.compute.internal} Successfully pulled image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
1h 1h 1 tatooinevertx12-1-deploy Pod spec.containers{deployment} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id 1637b0559753; Security:[seccomp=unconfined]
1h 1h 1 tatooinevertx12-1-deploy Pod spec.containers{deployment} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id 1637b0559753
1h 1h 1 tatooinevertx12-1-k6bhv Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-1-k6bhv to ip-172-31-77-238.us-east-2.compute.internal
1h 1h 1 tatooinevertx12-1-k6bhv Pod spec.containers{vertx} Normal Pulling {kubelet ip-172-31-77-238.us-east-2.compute.internal} pulling image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:c39fe760e149ab0b5f2fadf2eb1642cafd35bb061db19dd95c6ff2507bed71fb"
1h 1h 1 tatooinevertx12-1-k6bhv Pod spec.containers{vertx} Normal Pulled {kubelet ip-172-31-77-238.us-east-2.compute.internal} Successfully pulled image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:c39fe760e149ab0b5f2fadf2eb1642cafd35bb061db19dd95c6ff2507bed71fb"
1h 1h 1 tatooinevertx12-1-k6bhv Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-77-238.us-east-2.compute.internal} Created container with docker id 75ff0fb96800; Security:[seccomp=unconfined]
1h 1h 1 tatooinevertx12-1-k6bhv Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-77-238.us-east-2.compute.internal} Started container with docker id 75ff0fb96800
1h 1h 1 tatooinevertx12-1-k6bhv Pod spec.containers{vertx} Warning Unhealthy {kubelet ip-172-31-77-238.us-east-2.compute.internal} Readiness probe failed: Get http://10.129.28.187:8080/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
33m 33m 1 tatooinevertx12-1-k6bhv Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-77-238.us-east-2.compute.internal} Killing container with docker id 75ff0fb96800: Need to kill pod.
33m 33m 1 tatooinevertx12-1-k6bhv Pod Warning FailedSync {kubelet ip-172-31-77-238.us-east-2.compute.internal} Error syncing pod, skipping: error killing pod: failed to "TeardownNetwork" for "tatooinevertx12-1-k6bhv_burrsutter-stage" with TeardownNetworkError: "Failed to teardown network for pod \"4a259d3b-7172-11e7-ad51-02d7377a4b17\" using network plugins \"cni\": CNI request failed with status 400: 'Failed to execute iptables-restore: exit status 1 (iptables-restore: line 3 failed\n)\n'"
1h 1h 1 tatooinevertx12-1 ReplicationController Normal SuccessfulCreate {replication-controller } Created pod: tatooinevertx12-1-k6bhv
33m 33m 1 tatooinevertx12-1 ReplicationController Normal SuccessfulDelete {replication-controller } Deleted pod: tatooinevertx12-1-k6bhv
1h 1h 1 tatooinevertx12-2-deploy Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-2-deploy to ip-172-31-64-23.us-east-2.compute.internal
1h 1h 1 tatooinevertx12-2-deploy Pod spec.containers{deployment} Normal Pulling {kubelet ip-172-31-64-23.us-east-2.compute.internal} pulling image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
1h 1h 1 tatooinevertx12-2-deploy Pod spec.containers{deployment} Normal Pulled {kubelet ip-172-31-64-23.us-east-2.compute.internal} Successfully pulled image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
1h 1h 1 tatooinevertx12-2-deploy Pod spec.containers{deployment} Normal Created {kubelet ip-172-31-64-23.us-east-2.compute.internal} Created container with docker id e0fd9a09e538; Security:[seccomp=unconfined]
1h 1h 1 tatooinevertx12-2-deploy Pod spec.containers{deployment} Normal Started {kubelet ip-172-31-64-23.us-east-2.compute.internal} Started container with docker id e0fd9a09e538
34m 34m 1 tatooinevertx12-2-deploy Pod spec.containers{deployment} Normal Killing {kubelet ip-172-31-64-23.us-east-2.compute.internal} Killing container with docker id e0fd9a09e538: Need to kill pod.
1h 1h 1 tatooinevertx12-2-zjst4 Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-2-zjst4 to ip-172-31-65-181.us-east-2.compute.internal
40m 1h 8 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Pulling {kubelet ip-172-31-65-181.us-east-2.compute.internal} pulling image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:fe04a998532f71494da0e54850651cb4a154d6b49b2e6a8789422cccc539e9d9"
40m 1h 8 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Pulled {kubelet ip-172-31-65-181.us-east-2.compute.internal} Successfully pulled image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:fe04a998532f71494da0e54850651cb4a154d6b49b2e6a8789422cccc539e9d9"
1h 1h 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-65-181.us-east-2.compute.internal} Created container with docker id a162f4bb46e6; Security:[seccomp=unconfined]
1h 1h 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-65-181.us-east-2.compute.internal} Started container with docker id a162f4bb46e6
37m 1h 146 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Warning Unhealthy {kubelet ip-172-31-65-181.us-east-2.compute.internal} Readiness probe failed: Get http://10.130.27.16:8080/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
37m 1h 10 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Warning Unhealthy {kubelet ip-172-31-65-181.us-east-2.compute.internal} Liveness probe failed: Get http://10.130.27.16:8080/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
59m 59m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-65-181.us-east-2.compute.internal} Killing container with docker id a162f4bb46e6: pod "tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
59m 59m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-65-181.us-east-2.compute.internal} Created container with docker id 208fb93d4a66; Security:[seccomp=unconfined]
59m 59m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-65-181.us-east-2.compute.internal} Started container with docker id 208fb93d4a66
56m 56m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-65-181.us-east-2.compute.internal} Killing container with docker id 208fb93d4a66: pod "tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
56m 56m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-65-181.us-east-2.compute.internal} Created container with docker id 79168fcd4b2f; Security:[seccomp=unconfined]
56m 56m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-65-181.us-east-2.compute.internal} Started container with docker id 79168fcd4b2f
53m 53m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-65-181.us-east-2.compute.internal} Killing container with docker id 79168fcd4b2f: pod "tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
53m 53m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-65-181.us-east-2.compute.internal} Created container with docker id f892c7211b81; Security:[seccomp=unconfined]
53m 53m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-65-181.us-east-2.compute.internal} Started container with docker id f892c7211b81
50m 50m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-65-181.us-east-2.compute.internal} Killing container with docker id f892c7211b81: pod "tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
50m 50m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-65-181.us-east-2.compute.internal} Created container with docker id 303428f5e37f; Security:[seccomp=unconfined]
50m 50m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-65-181.us-east-2.compute.internal} Started container with docker id 303428f5e37f
47m 47m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-65-181.us-east-2.compute.internal} Killing container with docker id 303428f5e37f: pod "tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
47m 47m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-65-181.us-east-2.compute.internal} Created container with docker id 6d3c7cee0bfd; Security:[seccomp=unconfined]
47m 47m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-65-181.us-east-2.compute.internal} Started container with docker id 6d3c7cee0bfd
43m 43m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-65-181.us-east-2.compute.internal} Killing container with docker id 6d3c7cee0bfd: pod "tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
43m 43m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-65-181.us-east-2.compute.internal} Created container with docker id cfcc232e30c8; Security:[seccomp=unconfined]
43m 43m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-65-181.us-east-2.compute.internal} Started container with docker id cfcc232e30c8
40m 40m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-65-181.us-east-2.compute.internal} Killing container with docker id cfcc232e30c8: pod "tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
40m 40m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-65-181.us-east-2.compute.internal} Created container with docker id 5e66914f3fa8; Security:[seccomp=unconfined]
40m 40m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-65-181.us-east-2.compute.internal} Started container with docker id 5e66914f3fa8
37m 37m 1 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-65-181.us-east-2.compute.internal} Killing container with docker id 5e66914f3fa8: pod "tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
34m 37m 15 tatooinevertx12-2-zjst4 Pod spec.containers{vertx} Warning BackOff {kubelet ip-172-31-65-181.us-east-2.compute.internal} Back-off restarting failed docker container
34m 37m 15 tatooinevertx12-2-zjst4 Pod Warning FailedSync {kubelet ip-172-31-65-181.us-east-2.compute.internal} Error syncing pod, skipping: failed to "StartContainer" for "vertx" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=vertx pod=tatooinevertx12-2-zjst4_burrsutter-stage(015d94fc-7173-11e7-ad51-02d7377a4b17)"
1h 1h 1 tatooinevertx12-2 ReplicationController Normal SuccessfulCreate {replication-controller } Created pod: tatooinevertx12-2-zjst4
34m 34m 1 tatooinevertx12-2 ReplicationController Normal SuccessfulDelete {replication-controller } Deleted pod: tatooinevertx12-2-zjst4
34m 34m 1 tatooinevertx12-3-blv2m Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-3-blv2m to ip-172-31-78-9.us-east-2.compute.internal
34m 34m 1 tatooinevertx12-3-blv2m Pod Warning FailedSync {kubelet ip-172-31-78-9.us-east-2.compute.internal} Error syncing pod, skipping: failed to "SetupNetwork" for "tatooinevertx12-3-blv2m_burrsutter-stage" with SetupNetworkError: "Failed to setup network for pod \"tatooinevertx12-3-blv2m_burrsutter-stage(0c79264c-7177-11e7-ad51-02d7377a4b17)\" using network plugins \"cni\": CNI request failed with status 400: 'Failed to execute iptables-restore: exit status 1 (iptables-restore: line 3 failed\n)\n'; Skipping pod"
33m 33m 1 tatooinevertx12-3-blv2m Pod spec.containers{vertx} Normal Pulling {kubelet ip-172-31-78-9.us-east-2.compute.internal} pulling image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:fe04a998532f71494da0e54850651cb4a154d6b49b2e6a8789422cccc539e9d9"
33m 33m 1 tatooinevertx12-3-blv2m Pod spec.containers{vertx} Normal Pulled {kubelet ip-172-31-78-9.us-east-2.compute.internal} Successfully pulled image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:fe04a998532f71494da0e54850651cb4a154d6b49b2e6a8789422cccc539e9d9"
33m 33m 1 tatooinevertx12-3-blv2m Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id ed408d29cff3; Security:[seccomp=unconfined]
33m 33m 1 tatooinevertx12-3-blv2m Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id ed408d29cff3
33m 33m 1 tatooinevertx12-3-blv2m Pod spec.containers{vertx} Warning Unhealthy {kubelet ip-172-31-78-9.us-east-2.compute.internal} Readiness probe failed: Get http://10.128.52.247:8080/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
29m 29m 1 tatooinevertx12-3-blv2m Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id ed408d29cff3: Need to kill pod.
34m 34m 1 tatooinevertx12-3-deploy Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-3-deploy to ip-172-31-65-163.us-east-2.compute.internal
34m 34m 1 tatooinevertx12-3-deploy Pod spec.containers{deployment} Normal Pulling {kubelet ip-172-31-65-163.us-east-2.compute.internal} pulling image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
34m 34m 1 tatooinevertx12-3-deploy Pod spec.containers{deployment} Normal Pulled {kubelet ip-172-31-65-163.us-east-2.compute.internal} Successfully pulled image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
34m 34m 1 tatooinevertx12-3-deploy Pod spec.containers{deployment} Normal Created {kubelet ip-172-31-65-163.us-east-2.compute.internal} Created container with docker id 721d3957cc36; Security:[seccomp=unconfined]
34m 34m 1 tatooinevertx12-3-deploy Pod spec.containers{deployment} Normal Started {kubelet ip-172-31-65-163.us-east-2.compute.internal} Started container with docker id 721d3957cc36
33m 33m 1 tatooinevertx12-3-deploy Pod spec.containers{deployment} Normal Killing {kubelet ip-172-31-65-163.us-east-2.compute.internal} Killing container with docker id 721d3957cc36: Need to kill pod.
34m 34m 1 tatooinevertx12-3 ReplicationController Normal SuccessfulCreate {replication-controller } Created pod: tatooinevertx12-3-blv2m
29m 29m 1 tatooinevertx12-3 ReplicationController Normal SuccessfulDelete {replication-controller } Deleted pod: tatooinevertx12-3-blv2m
29m 29m 1 tatooinevertx12-4-0gmpl Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-4-0gmpl to ip-172-31-77-238.us-east-2.compute.internal
29m 29m 1 tatooinevertx12-4-0gmpl Pod spec.containers{vertx} Normal Pulling {kubelet ip-172-31-77-238.us-east-2.compute.internal} pulling image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:18b7c2da193ed00cc0096eb3dd7094048bbff8856ebe932419ed0d8420813fba"
29m 29m 1 tatooinevertx12-4-0gmpl Pod spec.containers{vertx} Normal Pulled {kubelet ip-172-31-77-238.us-east-2.compute.internal} Successfully pulled image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:18b7c2da193ed00cc0096eb3dd7094048bbff8856ebe932419ed0d8420813fba"
29m 29m 1 tatooinevertx12-4-0gmpl Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-77-238.us-east-2.compute.internal} Created container with docker id 461bded8f3c8; Security:[seccomp=unconfined]
29m 29m 1 tatooinevertx12-4-0gmpl Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-77-238.us-east-2.compute.internal} Started container with docker id 461bded8f3c8
29m 29m 1 tatooinevertx12-4-0gmpl Pod spec.containers{vertx} Warning Unhealthy {kubelet ip-172-31-77-238.us-east-2.compute.internal} Readiness probe failed: Get http://10.129.28.189:8080/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
29m 29m 1 tatooinevertx12-4-deploy Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-4-deploy to ip-172-31-65-163.us-east-2.compute.internal
29m 29m 1 tatooinevertx12-4-deploy Pod spec.containers{deployment} Normal Pulling {kubelet ip-172-31-65-163.us-east-2.compute.internal} pulling image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
29m 29m 1 tatooinevertx12-4-deploy Pod spec.containers{deployment} Normal Pulled {kubelet ip-172-31-65-163.us-east-2.compute.internal} Successfully pulled image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
29m 29m 1 tatooinevertx12-4-deploy Pod spec.containers{deployment} Normal Created {kubelet ip-172-31-65-163.us-east-2.compute.internal} Created container with docker id f277c241459f; Security:[seccomp=unconfined]
29m 29m 1 tatooinevertx12-4-deploy Pod spec.containers{deployment} Normal Started {kubelet ip-172-31-65-163.us-east-2.compute.internal} Started container with docker id f277c241459f
29m 29m 1 tatooinevertx12-4-deploy Pod spec.containers{deployment} Normal Killing {kubelet ip-172-31-65-163.us-east-2.compute.internal} Killing container with docker id f277c241459f: Need to kill pod.
29m 29m 1 tatooinevertx12-4-deploy Pod Warning FailedSync {kubelet ip-172-31-65-163.us-east-2.compute.internal} Error syncing pod, skipping: failed to "TeardownNetwork" for "tatooinevertx12-4-deploy_burrsutter-stage" with TeardownNetworkError: "Failed to teardown network for pod \"ac39df2c-7177-11e7-ad51-02d7377a4b17\" using network plugins \"cni\": CNI request failed with status 400: 'Failed to execute iptables-restore: exit status 1 (iptables-restore: line 3 failed\n)\n'"
29m 29m 1 tatooinevertx12-4 ReplicationController Normal SuccessfulCreate {replication-controller } Created pod: tatooinevertx12-4-0gmpl
27m 27m 1 tatooinevertx12-5-deploy Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-5-deploy to ip-172-31-65-163.us-east-2.compute.internal
27m 27m 1 tatooinevertx12-5-deploy Pod spec.containers{deployment} Normal Pulling {kubelet ip-172-31-65-163.us-east-2.compute.internal} pulling image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
27m 27m 1 tatooinevertx12-5-deploy Pod spec.containers{deployment} Normal Pulled {kubelet ip-172-31-65-163.us-east-2.compute.internal} Successfully pulled image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
27m 27m 1 tatooinevertx12-5-deploy Pod spec.containers{deployment} Normal Created {kubelet ip-172-31-65-163.us-east-2.compute.internal} Created container with docker id 3e5469924ff3; Security:[seccomp=unconfined]
27m 27m 1 tatooinevertx12-5-deploy Pod spec.containers{deployment} Normal Started {kubelet ip-172-31-65-163.us-east-2.compute.internal} Started container with docker id 3e5469924ff3
3m 3m 1 tatooinevertx12-5-deploy Pod spec.containers{deployment} Normal Killing {kubelet ip-172-31-65-163.us-east-2.compute.internal} Killing container with docker id 3e5469924ff3: Need to kill pod.
27m 27m 1 tatooinevertx12-5-f01hf Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-5-f01hf to ip-172-31-78-9.us-east-2.compute.internal
4m 27m 8 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Pulling {kubelet ip-172-31-78-9.us-east-2.compute.internal} pulling image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:89eb08efebefdf70d98c2b9df085dacbf7862de1e20eda5a09b249f1d102285a"
4m 27m 8 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Pulled {kubelet ip-172-31-78-9.us-east-2.compute.internal} Successfully pulled image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:89eb08efebefdf70d98c2b9df085dacbf7862de1e20eda5a09b249f1d102285a"
27m 27m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id c51edec13937; Security:[seccomp=unconfined]
27m 27m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id c51edec13937
3m 26m 135 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Warning Unhealthy {kubelet ip-172-31-78-9.us-east-2.compute.internal} Readiness probe failed: Get http://10.128.52.249:8080/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
4m 23m 9 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Warning Unhealthy {kubelet ip-172-31-78-9.us-east-2.compute.internal} Liveness probe failed: Get http://10.128.52.249:8080/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
23m 23m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id c51edec13937: pod "tatooinevertx12-5-f01hf_burrsutter-stage(1061dcb8-7178-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
23m 23m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id 195efbc23129; Security:[seccomp=unconfined]
23m 23m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id 195efbc23129
20m 20m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id 195efbc23129: pod "tatooinevertx12-5-f01hf_burrsutter-stage(1061dcb8-7178-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
20m 20m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id cdfa90c9795c; Security:[seccomp=unconfined]
20m 20m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id cdfa90c9795c
17m 17m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id cdfa90c9795c: pod "tatooinevertx12-5-f01hf_burrsutter-stage(1061dcb8-7178-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
17m 17m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id 1645400dd246; Security:[seccomp=unconfined]
17m 17m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id 1645400dd246
14m 14m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id 1645400dd246: pod "tatooinevertx12-5-f01hf_burrsutter-stage(1061dcb8-7178-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
14m 14m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id f295cd90feb4; Security:[seccomp=unconfined]
14m 14m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id f295cd90feb4
10m 10m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id f295cd90feb4: pod "tatooinevertx12-5-f01hf_burrsutter-stage(1061dcb8-7178-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
10m 10m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id 12628a98d3da; Security:[seccomp=unconfined]
10m 10m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id 12628a98d3da
7m 7m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id 12628a98d3da: pod "tatooinevertx12-5-f01hf_burrsutter-stage(1061dcb8-7178-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
7m 7m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id 1152f97354a5; Security:[seccomp=unconfined]
7m 7m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id 1152f97354a5
4m 4m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id 1152f97354a5: pod "tatooinevertx12-5-f01hf_burrsutter-stage(1061dcb8-7178-11e7-ad51-02d7377a4b17)" container "vertx" is unhealthy, it will be killed and re-created.
4m 4m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-78-9.us-east-2.compute.internal} Created container with docker id e09d8145d083; Security:[seccomp=unconfined]
4m 4m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-78-9.us-east-2.compute.internal} Started container with docker id e09d8145d083
3m 3m 1 tatooinevertx12-5-f01hf Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id e09d8145d083: Need to kill pod.
27m 27m 1 tatooinevertx12-5 ReplicationController Normal SuccessfulCreate {replication-controller } Created pod: tatooinevertx12-5-f01hf
3m 3m 1 tatooinevertx12-5 ReplicationController Normal SuccessfulDelete {replication-controller } Deleted pod: tatooinevertx12-5-f01hf
3m 3m 1 tatooinevertx12-6-4c5nc Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-6-4c5nc to ip-172-31-77-122.us-east-2.compute.internal
3m 3m 1 tatooinevertx12-6-4c5nc Pod spec.containers{vertx} Normal Pulling {kubelet ip-172-31-77-122.us-east-2.compute.internal} pulling image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:77d24be528e3033b56b4aa4a2c557a7c64409c3b1b040fe12687921ce1420176"
3m 3m 1 tatooinevertx12-6-4c5nc Pod spec.containers{vertx} Normal Pulled {kubelet ip-172-31-77-122.us-east-2.compute.internal} Successfully pulled image "172.30.98.11:5000/burrsutter-stage/tatooinevertx12@sha256:77d24be528e3033b56b4aa4a2c557a7c64409c3b1b040fe12687921ce1420176"
3m 3m 1 tatooinevertx12-6-4c5nc Pod spec.containers{vertx} Normal Created {kubelet ip-172-31-77-122.us-east-2.compute.internal} Created container with docker id d3e4affc495b; Security:[seccomp=unconfined]
3m 3m 1 tatooinevertx12-6-4c5nc Pod spec.containers{vertx} Normal Started {kubelet ip-172-31-77-122.us-east-2.compute.internal} Started container with docker id d3e4affc495b
30s 2m 15 tatooinevertx12-6-4c5nc Pod spec.containers{vertx} Warning Unhealthy {kubelet ip-172-31-77-122.us-east-2.compute.internal} Readiness probe failed: Get http://10.128.19.96:8080/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
3m 3m 1 tatooinevertx12-6-deploy Pod Normal Scheduled {default-scheduler } Successfully assigned tatooinevertx12-6-deploy to ip-172-31-65-163.us-east-2.compute.internal
3m 3m 1 tatooinevertx12-6-deploy Pod spec.containers{deployment} Normal Pulling {kubelet ip-172-31-65-163.us-east-2.compute.internal} pulling image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
3m 3m 1 tatooinevertx12-6-deploy Pod spec.containers{deployment} Normal Pulled {kubelet ip-172-31-65-163.us-east-2.compute.internal} Successfully pulled image "registry.ops.openshift.com/openshift3/ose-deployer:v3.5.5.19"
3m 3m 1 tatooinevertx12-6-deploy Pod spec.containers{deployment} Normal Created {kubelet ip-172-31-65-163.us-east-2.compute.internal} Created container with docker id afff7d7ec789; Security:[seccomp=unconfined]
3m 3m 1 tatooinevertx12-6-deploy Pod spec.containers{deployment} Normal Started {kubelet ip-172-31-65-163.us-east-2.compute.internal} Started container with docker id afff7d7ec789
3m 3m 9 tatooinevertx12-6 ReplicationController Warning FailedCreate {replication-controller } Error creating: pods "tatooinevertx12-6-" is forbidden: exceeded quota: compute-resources, requested: limits.cpu=1,limits.memory=512Mi, used: limits.cpu=2,limits.memory=1Gi, limited: limits.cpu=2,limits.memory=1Gi
3m 3m 1 tatooinevertx12-6 ReplicationController Normal SuccessfulCreate {replication-controller } Created pod: tatooinevertx12-6-4c5nc
1h 1h 1 tatooinevertx12 DeploymentConfig Normal DeploymentCreated {deploymentconfig-controller } Created new replication controller "tatooinevertx12-1" for version 1
1h 1h 1 tatooinevertx12 DeploymentConfig Normal DeploymentCreated {deploymentconfig-controller } Created new replication controller "tatooinevertx12-2" for version 2
34m 34m 1 tatooinevertx12 DeploymentConfig Normal DeploymentCancelled {deployments-controller } tatooinevertx12-2: Deployment "burrsutter-stage/tatooinevertx12-2" cancelled
34m 34m 1 tatooinevertx12 DeploymentConfig Normal ReplicationControllerScaled {deploymentconfig-controller } Scaled replication controller "tatooinevertx12-2" from 1 to 0
34m 34m 1 tatooinevertx12 DeploymentConfig Normal DeploymentCreated {deploymentconfig-controller } Created new replication controller "tatooinevertx12-3" for version 3
29m 29m 1 tatooinevertx12 DeploymentConfig Normal DeploymentCreated {deploymentconfig-controller } Created new replication controller "tatooinevertx12-4" for version 4
27m 27m 1 tatooinevertx12 DeploymentConfig Normal DeploymentCreated {deploymentconfig-controller } Created new replication controller "tatooinevertx12-5" for version 5
3m 3m 1 tatooinevertx12 DeploymentConfig Normal DeploymentCancelled {deploymentconfig-controller } Cancelled deployment "tatooinevertx12-5" superceded by version 6
3m 3m 6 tatooinevertx12 DeploymentConfig Normal DeploymentAwaitingCancellation {deploymentconfig-controller } Deployment of version 6 awaiting cancellation of older running deployments
3m 3m 1 tatooinevertx12 DeploymentConfig Normal DeploymentCancelled {deployments-controller } tatooinevertx12-5: Deployment "burrsutter-stage/tatooinevertx12-5" cancelled
3m 3m 1 tatooinevertx12 DeploymentConfig Normal DeploymentCreated {deploymentconfig-controller } Created new replication controller "tatooinevertx12-6" for version 6
1h 1h 1 yavinvertx1-1-54rfj Pod spec.containers{vertx} Normal Killing {kubelet ip-172-31-78-9.us-east-2.compute.internal} Killing container with docker id c0cebab9a629: Need to kill pod.
1h 1h 1 yavinvertx1-1 ReplicationController Normal SuccessfulDelete {replication-controller } Deleted pod: yavinvertx1-1-54rfj
1h 1h 1 yavinvertx1 DeploymentConfig Normal ReplicationControllerScaled {deploymentconfig-controller } Scaled replication controller "yavinvertx1-1" from 1 to 0
bmicklea
commented
7 years ago I doubt this is the root cause for every situation in which we see this behaviour, but one of the easy ones is that I had run out of quota in my staging environment. The Openshift.io UI reported staging as updated in both the applications (had the correct, updated version number) and pipeline tabs, but the OSO deployment was stalled due to insufficient quota. Shutting down other running staging instances allowed me to continue.
Question - should we be aggressively shutting down staging pods for older pipelines once a new pipeline is generated in response to a GH repo change? As a development system I'd expect a ton of pipeline churn and people rarely interested in old staging pods.
rawlingsj
commented
7 years ago I doubt this is the root cause for every situation in which we see this behaviour, but one of the easy ones is that I had run out of quota in my staging environment. The Openshift.io UI reported staging as updated in both the applications (had the correct, updated version number) and pipeline tabs, but the OSO deployment was stalled due to insufficient quota. Shutting down other running staging instances allowed me to continue.
most cases I've seen of failed stage + run deployments is because of quota so I'd say that's a pretty good guess
Question - should we be aggressively shutting down staging pods for older pipelines once a new pipeline is generated in response to a GH repo change? As a development system I'd expect a ton of pipeline churn and people rarely interested in old staging pods.
I did have somewhere on my backlog to scale down staged DeploymentConfigs after a successful Runtime deployment. Would that have the desired affect?
bmicklea
commented
7 years ago re: quota: Figuring out how to better warn users through the IO UI about quota issues in OSO could be a big win for us.
re: shutdown. This one is tricky. For free accounts it probably makes sense. But we'd need a way in the pipeline and apps tabs to clearly show that the stage pod was offline once the run pod was updated. And of course we'd have to make sure the run pod was really up before we did...
Once we get teams on and doing real work then we'd need a better trigger for scaling down unused pods. Question: is there a way to detect if a pod has no traffic? If so then I'd suggest that any staging pod that doesn't have traffic for x (configurable) minutes is scaled down automatically. By default x = 120.
I also realize I wasn't completely clear in my last comment - the bigger issue is when users have multiple spaces. When I delete a space it doesn't delete any of the OSO artifacts (pipelines, stage pods, run pods, etc...). So in doing my testing I've generally been doing new spaces each time. This results in a number of these pods continuing to run and take up quota even though their associated spaces are deleted. Hopefully this is a quick fix - it would help a lot of people.
I think we need a few things then based on this conversation (this is my priority but others can/should chime in):
bmicklea
commented
7 years ago This is something we still need to figure out but isn't a critical aspect for the workshop. So I'm reopening and removing the workshop milestone.
qodfathr
commented
6 years ago This is pretty old, and I suspect no longer applicable. Plus, this discussion thread here covers a lot of additional topics, but these have generally spawned new epics and user stories. I'm going to close, but if anyone has a problem with that, please feel free to re-open.
Often in "-stage", the rolling update will fail to finish, though OSIO thinks it did.
The new pod does not seem to pass its readiness probe. https://screencast.com/t/kvrQE6LTI https://screencast.com/t/Q5SYUAZUwmWw
This happened after I ran the Helloworld flow https://vimeo.com/226958298