Unpredictable behaviour via BOSH CPI

[virtmerlin]

esxcloud-logs.zip bosh-task-logs.zip

Issue:

Deploying a CF manifest/releases results in a "HTTP status: '0', code: 'InternalError', message" err condition on first attempt (task 18), but completes with no errs on second attempt (task 20)

Environ is a clean deployment of Photon ctrlr 1.0/ Bosh Photon CPI 1.0

Bosh Task 19:

Director task 18 Deprecation: Ignoring cloud config. Manifest contains 'networks' section.

Started preparing deployment > Preparing deployment. Done (00:00:01)

Started preparing package compilation > Finding packages to compile. Done (00:00:01)

Started creating missing vms Started creating missing vms > consul_server-partition/0 (5ee4ede7-cb24-4ffa-a272-b2a78ee75bdf) Started creating missing vms > nats-partition/0 (922fcecb-4f5b-437d-9ab5-b08617dda215) Started creating missing vms > etcd_server-partition/0 (6737e781-cae6-4880-b1a9-f3f0776cf627) Started creating missing vms > diego_database-partition/0 (473d60e5-3bb8-4433-a5cb-465cc45cb057) Started creating missing vms > nfs_server-partition/0 (81e46dd2-cf3a-4926-99a8-1c2f0403d541) Started creating missing vms > router-partition/0 (3f377edd-c7c8-4045-a99f-64ff0242b79f) Started creating missing vms > mysql_proxy-partition/0 (94c5253a-5f02-4ba8-a6d1-824e5d8cde15) Started creating missing vms > mysql-partition/0 (68336cfd-6ec5-4021-85b5-04686fc1ef24) Started creating missing vms > ccdb-partition/0 (9f75cf98-b1e7-4e28-8964-3cbe896bace0) Started creating missing vms > uaadb-partition/0 (d1bef9eb-7bcd-478a-862c-987493f059cd) Started creating missing vms > consoledb-partition/0 (071c0e07-a095-4986-a813-f638b5ec24b0) Started creating missing vms > ha_proxy-partition/0 (7c71c8a9-675d-4511-a387-4d9ea3bb8bd1) Started creating missing vms > clock_global-partition/0 (5fe6d136-946e-4814-bb3d-adb81aa04a0d) Started creating missing vms > cloud_controller_worker-partition/0 (03025be2-0ea3-45b6-b8b9-3e2ff236464f) Started creating missing vms > cloud_controller-partition/0 (554226b6-6a1b-4ee5-9b14-297c2ad1f592) Started creating missing vms > uaa-partition/0 (e7382aad-20a3-489c-909a-c0c6d8fe5a21) Started creating missing vms > diego_brain-partition/0 (f9d813c9-4efe-4b96-8dd5-daa67cf408d3) Started creating missing vms > diego_cell-partition/0 (31c30cce-4de7-4546-851f-47bb0645f4fb) Started creating missing vms > doppler-partition/0 (c72cc37c-8f8f-4e74-b5d8-d9b7d69985a2) Started creating missing vms > loggregator_trafficcontroller-partition/0 (a8ed6fa4-5f4a-474c-80b4-ef966ee608f9) Failed creating missing vms > nfs_server-partition/0 (81e46dd2-cf3a-4926-99a8-1c2f0403d541): photon: Task 'e28c3136-f2d0-4d00-bbbf-c5cec8c89b13' is in error state: {@step=={"sequence"=>"0","state"=>"ERROR","errors"=>[photon: { HTTP status: '0', code: 'InternalError', message: 'Please contact the system administrator about request #4fb1e284-f336-4240-8fbf-ed86ad1961a1', data: 'map[]' }],"warnings"=>[],"operation"=>"UPLOAD_ISO","startedTime"=>"1471993710795","queuedTime"=>"1471993710789","endTime"=>"1471993710918","options"=>map[c11dbfe0-7fa9-4e5a-8bea-cdee6a2eaf73:iso 05b79b97-a769-438b-ab0d-314f2e949158:vm]}} (00:00:04) Done creating missing vms > nats-partition/0 (922fcecb-4f5b-437d-9ab5-b08617dda215) (00:00:52) Done creating missing vms > etcd_server-partition/0 (6737e781-cae6-4880-b1a9-f3f0776cf627) (00:00:52) Done creating missing vms > consoledb-partition/0 (071c0e07-a095-4986-a813-f638b5ec24b0) (00:00:52) Done creating missing vms > cloud_controller_worker-partition/0 (03025be2-0ea3-45b6-b8b9-3e2ff236464f) (00:00:52) Done creating missing vms > uaa-partition/0 (e7382aad-20a3-489c-909a-c0c6d8fe5a21) (00:00:56) Done creating missing vms > mysql_proxy-partition/0 (94c5253a-5f02-4ba8-a6d1-824e5d8cde15) (00:00:56) Done creating missing vms > cloud_controller-partition/0 (554226b6-6a1b-4ee5-9b14-297c2ad1f592) (00:00:56) Done creating missing vms > diego_brain-partition/0 (f9d813c9-4efe-4b96-8dd5-daa67cf408d3) (00:00:57) Done creating missing vms > ha_proxy-partition/0 (7c71c8a9-675d-4511-a387-4d9ea3bb8bd1) (00:00:58) Done creating missing vms > ccdb-partition/0 (9f75cf98-b1e7-4e28-8964-3cbe896bace0) (00:00:58) Done creating missing vms > uaadb-partition/0 (d1bef9eb-7bcd-478a-862c-987493f059cd) (00:00:58) Done creating missing vms > loggregator_trafficcontroller-partition/0 (a8ed6fa4-5f4a-474c-80b4-ef966ee608f9) (00:01:00) Done creating missing vms > consul_server-partition/0 (5ee4ede7-cb24-4ffa-a272-b2a78ee75bdf) (00:01:03) Done creating missing vms > mysql-partition/0 (68336cfd-6ec5-4021-85b5-04686fc1ef24) (00:01:03) Done creating missing vms > diego_cell-partition/0 (31c30cce-4de7-4546-851f-47bb0645f4fb) (00:01:03) Done creating missing vms > router-partition/0 (3f377edd-c7c8-4045-a99f-64ff0242b79f) (00:01:04) Done creating missing vms > doppler-partition/0 (c72cc37c-8f8f-4e74-b5d8-d9b7d69985a2) (00:01:04) Done creating missing vms > clock_global-partition/0 (5fe6d136-946e-4814-bb3d-adb81aa04a0d) (00:01:04) Done creating missing vms > diego_database-partition/0 (473d60e5-3bb8-4433-a5cb-465cc45cb057) (00:01:04) Failed creating missing vms (00:01:04)

Error 100: photon: Task 'e28c3136-f2d0-4d00-bbbf-c5cec8c89b13' is in error state: {@step=={"sequence"=>"0","state"=>"ERROR","errors"=>[photon: { HTTP status: '0', code: 'InternalError', message: 'Please contact the system administrator about request #4fb1e284-f336-4240-8fbf-ed86ad1961a1', data: 'map[]' }],"warnings"=>[],"operation"=>"UPLOAD_ISO","startedTime"=>"1471993710795","queuedTime"=>"1471993710789","endTime"=>"1471993710918","options"=>map[c11dbfe0-7fa9-4e5a-8bea-cdee6a2eaf73:iso 05b79b97-a769-438b-ab0d-314f2e949158:vm]}}

Task 18 error

BOSH Task 20:

Director task 20 Deprecation: Ignoring cloud config. Manifest contains 'networks' section.

Started preparing deployment > Preparing deployment. Done (00:00:01)

Started preparing package compilation > Finding packages to compile. Done (00:00:01)

Started creating missing vms > nfs_server-partition/0 (81e46dd2-cf3a-4926-99a8-1c2f0403d541). Done (00:00:43)

Started updating job consul_server-partition > consul_server-partition/0 (5ee4ede7-cb24-4ffa-a272-b2a78ee75bdf) (canary). Done (00:01:03) Started updating job nats-partition > nats-partition/0 (922fcecb-4f5b-437d-9ab5-b08617dda215) (canary). Done (00:00:41) Started updating job etcd_server-partition > etcd_server-partition/0 (6737e781-cae6-4880-b1a9-f3f0776cf627) (canary). Done (00:00:58) Started updating job diego_database-partition > diego_database-partition/0 (473d60e5-3bb8-4433-a5cb-465cc45cb057) (canary). Done (00:01:05) Started updating job nfs_server-partition > nfs_server-partition/0 (81e46dd2-cf3a-4926-99a8-1c2f0403d541) (canary). Done (00:01:01) Started updating job router-partition > router-partition/0 (3f377edd-c7c8-4045-a99f-64ff0242b79f) (canary). Done (00:00:42) Started updating job mysql_proxy-partition > mysql_proxy-partition/0 (94c5253a-5f02-4ba8-a6d1-824e5d8cde15) (canary). Done (00:00:48) Started updating job mysql-partition > mysql-partition/0 (68336cfd-6ec5-4021-85b5-04686fc1ef24) (canary). Done (00:03:15) Started updating job ccdb-partition > ccdb-partition/0 (9f75cf98-b1e7-4e28-8964-3cbe896bace0) (canary). Done (00:01:02) Started updating job uaadb-partition > uaadb-partition/0 (d1bef9eb-7bcd-478a-862c-987493f059cd) (canary). Done (00:01:02) Started updating job consoledb-partition > consoledb-partition/0 (071c0e07-a095-4986-a813-f638b5ec24b0) (canary). Done (00:00:59) Started updating job cloud_controller-partition > cloud_controller-partition/0 (554226b6-6a1b-4ee5-9b14-297c2ad1f592) (canary). Done (00:04:47) Started updating job ha_proxy-partition > ha_proxy-partition/0 (7c71c8a9-675d-4511-a387-4d9ea3bb8bd1) (canary). Done (00:00:44) Started updating job clock_global-partition > clock_global-partition/0 (5fe6d136-946e-4814-bb3d-adb81aa04a0d) (canary). Done (00:00:57) Started updating job cloud_controller_worker-partition > cloud_controller_worker-partition/0 (03025be2-0ea3-45b6-b8b9-3e2ff236464f) (canary). Done (00:01:17) Started updating job uaa-partition > uaa-partition/0 (e7382aad-20a3-489c-909a-c0c6d8fe5a21) (canary). Done (00:01:54) Started updating job diego_brain-partition > diego_brain-partition/0 (f9d813c9-4efe-4b96-8dd5-daa67cf408d3) (canary). Done (00:01:10) Started updating job diego_cell-partition > diego_cell-partition/0 (31c30cce-4de7-4546-851f-47bb0645f4fb) (canary). Done (00:02:32) Started updating job doppler-partition > doppler-partition/0 (c72cc37c-8f8f-4e74-b5d8-d9b7d69985a2) (canary). Done (00:00:44) Started updating job loggregator_trafficcontroller-partition > loggregator_trafficcontroller-partition/0 (a8ed6fa4-5f4a-474c-80b4-ef966ee608f9) (canary). Done (00:00:46)

Task 20 done

Started 2016-08-23 23:09:43 UTC Finished 2016-08-23 23:38:03 UTC Duration 00:28:20

[molteanu]

what release is this against?

[toliaqat]

I believe 1.0.

[molteanu]

Thanks Touseef I missed the info at the top.

Merlin - are there any http proxies on the management network?

The root cause of the failure is "Suppressed: java.net.SocketException: Broken pipe". We get this SocketException when we transfer the ISO file data from the management VM to the ESX host. I have been unable to track down the cause yet.

[virtmerlin]

no proxies

On Tue, Aug 23, 2016 at 5:39 PM, Mihnea Olteanu notifications@github.com wrote:

Thanks Touseef I missed the info at the top.

Merlin - are there any http proxies on the management network?

The root cause of the failure is "Suppressed: java.net.SocketException: Broken pipe". We get this SocketException when we transfer the ISO file data from the management VM to the ESX host. I have been unable to track down the cause yet.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/vmware/photon-controller/issues/46#issuecomment-241925358, or mute the thread https://github.com/notifications/unsubscribe-auth/AKBRYS-AZSWmvhs8bNRmz5-wjjKMazSdks5qi5LSgaJpZM4JrhCN .

Merlin Glynn mglynn@pivotal.io 001-214-551-8074

[kmjung]

This is being tracked as https://www.pivotaltracker.com/story/show/126938221