Closed yatzek closed 2 years ago
This issue is intermittent, roughly about 50% reproducibility. You should be able to reproduce it: deploy bosh without registry and try deploying something that has a persistent disk and vm is being re-created.
Hi @yatzek This issue has been fixed by the release v40.0.0 and please have a check.
After removal of bosh registry from the config, we observe intermittent issues when attaching a persistent disk. We believe the reason for the issues is the cpi code, attach disk action, line 46:
https://github.com/cloudfoundry-incubator/bosh-alicloud-cpi-release/blob/47a510af1b73dea69987425c03033d1394b41d1a/src/bosh-alicloud-cpi/action/attach_disk.go#L46
Lines 46-63 are executed only if you do not have the bosh registry configured, the cpi stops the vm and then starts it again in line 92-109.
The vm is started for the first time in the create-vm step:
https://github.com/cloudfoundry-incubator/bosh-alicloud-cpi-release/blob/47a510af1b73dea69987425c03033d1394b41d1a/src/bosh-alicloud-cpi/action/create_vm.go#L356
and the bosh agent starts up and tries to bootstrap the config and shortly after, during the attach disk step, we stop the vm. This can corrupt bosh agent config file(s), for example, we observed empty /var/vcap/bosh/agent_state.json - which causes bosh agent to enter infinite restart loop:
We also observed empty ssh host keys, these are generated on first boot: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/7320d9060f70deb154e0a9fba688e5c7210bf8d6/stemcell_builder/stages/base_ubuntu_firstboot/assets/root/firstboot.sh#L5
When this happens you are not able to ssh into the vm.