Closed titanous closed 8 years ago
Looks like e8c86151cc70472c9b195a77fd4ab885 got stuck in the starting state. containerinit Resume was called (or at least attempted), but no log messages after that.
Some similar failures:
c3f192a4-809a0256adb7428e8b79958ec20c9a84
stuck in starting state86e58712-f0f998eb92b94ba89e19b2c21482897c
stuck in starting stateIn both cases, the jobs have output so are running.
Another example: https://ci.flynn.io/builds/20150311205929-abfc609f
Closing, this test failure does not appear in the CI logs dating back to May 1st.
This happened in the following build: https://ci.flynn.io/builds/20150928233104-066e9837
See job 99246a34-fd75e9ef-4988-4b9a-934f-34590f161fa9
which was stuck in the starting state, thus leading to TestKeyRotation
timing out waiting for a deployment.
Although the underlying issue is not fixed, I've added a workaround in #2325.
Will re-open if I see this again.
I'm suffering with this issue too and not sure if it's this that is preventing me from fixing my cluster. I've tried stopping them but has no effect
hosting2-3f1cff2c-dee4-4cae-8532-df8115c55b00 running 29 minutes ago postgres postgres
hosting2-10f6864f-f44b-4328-b76e-03569f227924 running 30 minutes ago discoverd app
hosting2-07c27155-a8bc-4a2f-816a-fbfc42b6c80d running 30 minutes ago flannel app
hosting3-fcdfb15c-e13e-4f39-a847-e0c8c7d465ea running 30 minutes ago discoverd app
hosting3-aa8f6864-5b9c-4be5-a3c0-1411608ed471 running 30 minutes ago flannel app
hosting1-7c89111f-debb-4d27-af60-e4ad0d4311ac running 2 days ago dottodot-prerender web
hosting1-c63b5d0c-d69f-4510-86a9-92d1238c237a running 2 days ago controller web
hosting1-5b6cb230-3a54-43b9-b2ce-75be5baafd47 running 2 days ago dashboard web
hosting2-79671455-737a-4307-add5-45dbe97a8fb0 running 2 days ago router app
hosting1-6dc45290-b9ab-4bc3-a98b-bcd38f4f2ccd running 2 days ago router app
hosting3-acf80ff9-8fd6-46c2-8caa-cdd98ed91d04 running 2 days ago router app
hosting3-87bbc5c2-0495-48c2-9f22-15506335c4ee running 2 days ago controller web
hosting1-a4d9a3ed-822c-4ba8-8c1c-24a321e273c0 starting postgres postgres
hosting1-7df67f67-c3fc-4b99-8ded-4b4c48bd9f7c starting postgres postgres
hosting1-91caf687-b275-4b05-a013-6072685ab9e2 starting flannel app
hosting1-48034bf6-70d9-4b38-90df-c9fc8803f5d8 starting postgres postgres
hosting1-01b87eed-d71d-4415-8298-e41d072fa919 starting flannel app
hosting1-5c91b206-979f-4c32-afb3-8dbeb1328943 starting postgres postgres
hosting1-74dc555b-c382-47a1-b374-575507d6d24b starting postgres postgres
hosting1-b054022e-ceac-4c53-801e-6d5d1921c0a4 starting controller scheduler
hosting1-3abe6980-d8dc-4a8f-bd31-e7f0370e9540 starting postgres postgres
hosting1-43a3c4bb-989a-4931-9e66-e7605bd72b8a starting postgres postgres
hosting1-99ce7e93-ccd2-45be-99d7-4e85396b39f7 starting postgres postgres
hosting1-66817d44-6732-4826-8f2c-1cdd9d3f8dcf starting controller scheduler
hosting1-6fc55d13-60a7-42e6-907e-c6752e0b5019 starting controller scheduler
hosting1-c34831e8-9f8e-4fd9-a627-91632d1e3764 starting controller scheduler
hosting1-c666b024-55ff-41b1-8c9c-25081a581364 starting controller scheduler
hosting1-d54a72a9-91f0-4817-ada8-91236c483936 starting flannel app
hosting1-aabf8bc9-a460-4d4d-b89f-a536d6eaedce starting flannel app
hosting1-31493324-0cd5-48f7-aeab-a60701d21af4 starting postgres postgres
hosting1-34b10d88-a9d8-4512-99ec-a724c8f922e2 starting controller scheduler
hosting1-97a46b49-862b-42d4-81b9-b02a96c5870b starting postgres postgres
@dottodot that looks like a potential scheduler issue, can you please run sudo flynn-host collect-debug-info
on each node in your cluster, and then open a new issue with your above comment and the resulting gist links?
https://s3.amazonaws.com/flynn-ci-logs/20150130231002-b003fe21-build-376e150ab4de7365232b447e3333080047a3d17c-2015-01-30-23-15-07.txt