search

cloudfoundry-attic / bosh-vcloud-cpi-release

BOSH vCloud CPI
Apache License 2.0
4 stars 11 forks source link

Should not fail and let director retry on vcloud bosh account limitation #6

Open poblin-orange opened 8 years ago

poblin-orange commented 8 years ago

vcloud Iaas provider typically limit the number of simultanous request for a given vcloud user In that case we get this type of API Error

<Error xmlns="http://www.vmware.com/vcloud/v1.5" minorErrorCode="OPERATION_LIMITS_EXCEEDED" message="[ 17618e03-603e-4ff0-a323-4a553e8ea452 ] The maximum number of simultaneous operations for user &quot;bosh&quot; on organization &quot;Cube4PaasQA&quot; has been reached." majorErrorCode="400" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsichemaLocation="http://www.vmware.com/vcloud/v1.5 http://normandie-cube-api.itn.ftgroup/api/v1.5/schema/master.xsd"></Error>

As the error is not permanent, CPI should retry, or let Bosh Director know that he can retry the vm creation.

cppforlife commented 8 years ago

@poblin-orange currently we set https://bosh.io/jobs/director?source=github.com/cloudfoundry/bosh&version=207#p=director.max_threads to 4 for vcloud to avoid overwhelming vcloud. do you already do that?

poblin-orange commented 8 years ago

@cppforlife yes we limit on director side. However, we used a single bosh account for our deployment (1 microbosh => 1 master bosh => 2 cloudfoundry/ service bosh ). In that case, vcloud requests can exceed the 4 per bosh director configuration.

cppforlife commented 8 years ago

Ah makes sense. Also github hid the error returned -- fixed it.

cppforlife commented 8 years ago

Related material found on google: http://infrastructureadventures.com/2012/08/11/vcloud-director-policies-part-2-quotas-limits/

jmhardison commented 8 years ago

Raising this one again. In an on-premises installation of vCloud Director, there can be a low user quota assigned. In these cases, failing on user quota being exceeded results in a overall failure both in deployment and cleanup. In cleanup, several vApps are left some empty, and some with powered off guests.

As an example, our user quota is currently 2. Any user attempting to submit more operations than allowed will receive a temporary error until other queued jobs finish.

Defining the director threads at 1 does not alleviate the issue as it appears the CPI is firing many more request at once than the defined thread count.