search

cloudfoundry-attic / bosh-vcloud-cpi-release

BOSH vCloud CPI
Apache License 2.0
4 stars 11 forks source link

vcd_client.rb searches for unsuccessful tasks with no time limits #10

Open amarneris opened 8 years ago

amarneris commented 8 years ago

Hello!

We recently had a failed bosh deployment on a vCloud director, which resulted in the vCloud Director returning failed tasks for days after the original deployment failure.

The following code snippet in the vcd_client.rb file (lines 165 to 175) has a check, which is not time-bound, so it checks through all failed tasks against each entity (VM/vApp).

# verify all tasks succeeded unless entity.tasks.nil? || entity.tasks.empty?

failed_tasks = entity.tasks.find_all { |task| task.status.downcase != VCloudSdk::Xml::TASK_STATUS[:SUCCESS] }

unless failed_tasks.empty? @logger.debug "Failed tasks: #{failed_tasks}" unless accept_failure failed_tasks_info = failed_tasks.map { |t| "Task #{t.urn} #{t.operation}, Details:#{t.details} raise "Some tasks failed: #{failed_tasks_info.join('; ')}" end end end

Is there a reason why this is not time bound? Would it be an acceptable pull request to add some logic that only checks tasks with an endTime of less than 24 hours?

The debug log snippet, with tasks being returned from previous failed deployments is below:

D, [2016-05-03 10:50:54 #17641] [create_vm(29edc9f2-ca78-490b-9934-8ed8a44c84f5, urn:vcloud:catalogitem:c17801db-f2f2-46b9-88c9-0ff37238ba53, {"ram"=>16384, "disk"=>47104, "cpu"=>4}, ...)] DEBUG -- DirectorJobRunner: WAIT TASK urn:vcloud:task:69f6c8ee-16bf-4d35-ab4e-181ac6208118 Updating Virtual Machine 29edc9f2-ca78-490b-9934-8ed8a44c84f5(255bb163-58a1-4999-909c-be46a1315150) running
D, [2016-05-03 10:50:55 #17641] [create_vm(f2b77f66-d031-455b-b1b5-91198162cd77, urn:vcloud:catalogitem:c17801db-f2f2-46b9-88c9-0ff37238ba53, {"ram"=>2048, "disk"=>8192, "cpu"=>1}, ...)] DEBUG -- DirectorJobRunner: REST RES 200 {:date=>"Tue, 03 May 2016 10:50:47 GMT", :x_vmware_vcloud_request_id=>"c8dc45d2-72b8-470a-ae02-8bab38fe95ba", :x_vmware_vcloud_request_execution_time=>"6312", :content_type=>"application/vnd.vmware.vcloud.vapp+xml;version=5.1", :content_encoding=>"gzip", :vary=>"Accept-Encoding, User-Agent", :connection=>"close"} <?xml version="1.0" encoding="UTF-8"?>
<VApp xmlns="http://www.vmware.com/vcloud/v1.5">
    <Tasks>
        <Task cancelRequested="false" endTime="2016-05-02T12:18:06.660+01:00" expiryTime="2016-07-31T12:17:13.467+01:00" operation="Recomposed Virtual Application cf-appdev(1d3c26f9-d23b-4450-bc4d-447cdfffb991)" operationName="vdcRecomposeVapp" serviceNamespace="com.vmware.vcloud" startTime="2016-05-02T12:17:13.467+01:00" status="error" name="task" id="urn:vcloud:task:2fc61291-fbcd-4dd0-a691-88d138731f12" href="https://api.vcd.portal.replacedString.com/api/task/2fc61291-fbcd-4dd0-a691-88d138731f12" type="application/vnd.vmware.vcloud.task+xml">
            <Owner href="https://api.vcd.portal.replacedString.com/api/vApp/vapp-1d3c26f9-d23b-4450-bc4d-447cdfffb991" name="cf-appdev" type="application/vnd.vmware.vcloud.vApp+xml"/>
            <Error majorErrorCode="500" message="[ bb85330c-1bbc-43d3-a349-1a89330deb51 ] Unable to perform this action. Contact your cloud administrator." minorErrorCode="INTERNAL_SERVER_ERROR"/>
            <User href="https://api.vcd.portal.replacedString.com/api/admin/user/d72b88d9-c564-4008-832f-ce7ec49c48b0" name="1210.84.d6ec72" type="application/vnd.vmware.admin.user+xml"/>
            <Organization href="https://api.vcd.portal.replacedString.com/api/org/a88290f6-8e8b-40a0-8f8f-ff215cea6c06" name="63-84-1-5c5420" type="application/vnd.vmware.vcloud.org+xml"/>
            <Progress>100</Progress>
            <Details>  [ bb85330c-1bbc-43d3-a349-1a89330deb51 ] Unable to perform this action. Contact your cloud administr...</Details>
        </Task>
        <Task cancelRequested="false" endTime="2016-05-03T10:58:47.510+01:00" expiryTime="2016-08-01T10:57:27.437+01:00" operation="Recomposed Virtual Application cf-appdev(1d3c26f9-d23b-4450-bc4d-447cdfffb991)" operationName="vdcRecomposeVapp" serviceNamespace="com.vmware.vcloud" startTime="2016-05-03T10:57:27.437+01:00" status="error" name="task" id="urn:vcloud:task:c2f1bd1f-1e49-44a1-a7a8-a1752a011dfb" href="https://api.vcd.portal.replacedString.com/api/task/c2f1bd1f-1e49-44a1-a7a8-a1752a011dfb" type="application/vnd.vmware.vcloud.task+xml">
            <Owner href="https://api.vcd.portal.replacedString.com/api/vApp/vapp-1d3c26f9-d23b-4450-bc4d-447cdfffb991" name="cf-appdev" type="application/vnd.vmware.vcloud.vApp+xml"/>
            <Error majorErrorCode="500" message="[ f1eb4ce8-ac3d-434a-989e-cefd3e888ab0 ] Unable to perform this action. Contact your cloud administrator." minorErrorCode="INTERNAL_SERVER_ERROR"/>
            <User href="https://api.vcd.portal.replacedString.com/api/admin/user/d72b88d9-c564-4008-832f-ce7ec49c48b0" name="1210.84.d6ec72" type="application/vnd.vmware.admin.user+xml"/>
            <Organization href="https://api.vcd.portal.replacedString.com/api/org/a88290f6-8e8b-40a0-8f8f-ff215cea6c06" name="63-84-1-5c5420" type="application/vnd.vmware.vcloud.org+xml"/>
            <Progress>100</Progress>
            <Details>  [ f1eb4ce8-ac3d-434a-989e-cefd3e888ab0 ] Unable to perform this action. Contact your cloud administr...</Details>
        </Task>
        <Task cancelRequested="false" endTime="2016-05-03T09:48:09.607+01:00" expiryTime="2016-08-01T09:46:56.967+01:00" operation="Recomposed Virtual Application cf-appdev(1d3c26f9-d23b-4450-bc4d-447cdfffb991)" operationName="vdcRecomposeVapp" serviceNamespace="com.vmware.vcloud" startTime="2016-05-03T09:46:56.967+01:00" status="error" name="task" id="urn:vcloud:task:fabf2210-a3d0-4c7b-b57e-3a7252e6dc2d" href="https://api.vcd.portal.replacedString.com/api/task/fabf2210-a3d0-4c7b-b57e-3a7252e6dc2d" type="application/vnd.vmware.vcloud.task+xml">
            <Owner href="https://api.vcd.portal.replacedString.com/api/vApp/vapp-1d3c26f9-d23b-4450-bc4d-447cdfffb991" name="cf-appdev" type="application/vnd.vmware.vcloud.vApp+xml"/>
            <Error majorErrorCode="500" message="[ 2962b45a-9a19-4226-9393-e19bb648c24e ] Unable to perform this action. Contact your cloud administrator." minorErrorCode="INTERNAL_SERVER_ERROR"/>
            <User href="https://api.vcd.portal.replacedString.com/api/admin/user/d72b88d9-c564-4008-832f-ce7ec49c48b0" name="1210.84.d6ec72" type="application/vnd.vmware.admin.user+xml"/>
            <Organization href="https://api.vcd.portal.replacedString.com/api/org/a88290f6-8e8b-40a0-8f8f-ff215cea6c06" name="63-84-1-5c5420" type="application/vnd.vmware.vcloud.org+xml"/>
            <Progress>100</Progress>
            <Details>  [ 2962b45a-9a19-4226-9393-e19bb648c24e ] Unable to perform this action. Contact your cloud administr...</Details>
        </Task>
    </Tasks>
    <LeaseSettingsSection href="https://api.vcd.portal.replacedString.com/api/vApp/vapp-1d3c26f9-d23b-4450-bc4d-447cdfffb991/leaseSettingsSection/" type="application/vnd.vmware.vcloud.leaseSettingsSection+xml" ovf:required="false">
amarneris commented 8 years ago

Hello,

there is a fix for this made by Tim Lawrence, with a pull request here: https://github.com/cloudfoundry-attic/bosh_vcloud_cpi/pull/18

Is it possible to merge it in and close this off please?

Regards, Andreas

cppforlife commented 8 years ago

will take a look at the pr today.

Sent from my iPhone

On May 12, 2016, at 8:27 AM, amarneris notifications@github.com wrote:

Hello,

there is a fix for this made by Tim Lawrence, with a pull request here: cloudfoundry-attic/bosh_vcloud_cpi#18

Is it possible to merge it in and close this off please?

Regards, Andreas

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub