search

cloudfoundry / bosh-softlayer-cpi-release

An external BOSH CPI for the SoftLayer cloud written in Golang
Apache License 2.0
14 stars 20 forks source link

Problem acquiring iSCSI volumes with new Bosh VSIs #235

Closed nihillno closed 7 years ago

nihillno commented 7 years ago

When attempting a Bosh-based deployment, the following error is seen when provisioning VSIs:

Started updating job core-node > core-node/0 (fda10350-e343-4946-9127-f5ca02bebcb5) (canary). Failed: Attaching disk '24867541' to VM '32489483': Granting volume access to virtual guest 32489483: IBM02SVC421436_31 is not yet ready for mount, please try again after Volume Provisioning is complete. (00:02:11)

Error 100: Attaching disk '24867541' to VM '32489483': Granting volume access to virtual guest 32489483: IBM02SVC421436_31 is not yet ready for mount, please try again after Volume Provisioning is complete.

This has been happening fairly consistently at various times of the day over the last couple of days.

It seems the trouble is possibly with the create_disk() method in the Bosh SoftLayer CPI not calling "getVolumeStatus()" on newly-provisioned iSCSI volumes and waiting until their status is "PROVISION_COMPLETED".

cf-gitbot commented 7 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/145918729

The labels on this github issue will be updated when the story is started.

storagedaemon commented 7 years ago

It's better to check the property "isReadyToMount" (== 1) on the SoftLayer Network_Storage volume, and use that as the determining factor for whether the volume is ready to have hosts attached via the SoftLayer API.

nihillno commented 7 years ago

Hi. This issue is blocking at least one critical customer deployment for us. Would it be possible to get an ETA for a fix? Thanks.

mattcui commented 7 years ago

@nihillno Which version of CPI release are you using? The latest one v12 should have already fixed this problem.

nihillno commented 7 years ago

Ooh, it's pretty old - 3.0.5. It was updated to 12.0.1 on the environment in question - D_CIO - 25 minutes ago, as it happens. We'll see if that improves things. Many thanks!

nihillno commented 7 years ago

This is now working nicely. Thanks all!