search

cloudfoundry / bosh

Cloud Foundry BOSH is an open source tool chain for release engineering, deployment and lifecycle management of large scale distributed services.
https://bosh.io
Apache License 2.0
2.03k stars 658 forks source link

Persistent disk not mounting vcloud/vsphere #386

Closed Rikbruggink closed 11 years ago

Rikbruggink commented 11 years ago

Persistent disk isnt mounting on vcloud/vsphere using latest micro bosh stemcell downloaded from s3.amazon.com.

log output:

/var/vcap/bosh/log /current output:

2013-08-06_17:48:01.59244 #[598] INFO: Starting agent 1.5.0.pre.870... 2013-08-06_17:48:01.59249 #[598] INFO: Configuring agent... 2013-08-06_17:48:01.59384 #[598] INFO: Configuring instance 2013-08-06_17:48:01.82446 #[598] INFO: Waiting for /dev/sr0 (ENOMEDIUM): #<Errno::ENOMEDIUM: No medium found - /dev/sr0> 2013-08-06_17:48:02.32618 #[598] INFO: Waiting for /dev/sr0 (ENOMEDIUM): #<Errno::ENOMEDIUM: No medium found - /dev/sr0> 2013-08-06_17:48:02.82767 #[598] INFO: Waiting for /dev/sr0 (ENOMEDIUM): #<Errno::ENOMEDIUM: No medium found - /dev/sr0> 2013-08-06_17:48:03.32980 #[598] INFO: Waiting for /dev/sr0 (ENOMEDIUM): #<Errno::ENOMEDIUM: No medium found - /dev/sr0> 2013-08-06_17:48:03.83140 #[598] INFO: Waiting for /dev/sr0 (ENOMEDIUM): #<Errno::ENOMEDIUM: No medium found - /dev/sr0> 2013-08-06_17:48:04.33479 #[598] INFO: udevadm: #Bosh::Exec::Result:0x00000001e2d960 2013-08-06_17:48:04.33609 #[598] INFO: Waiting for /dev/sr0 (ENOMEDIUM or ENOTBLK) 2013-08-06_17:48:04.83780 #[598] INFO: Waiting for /dev/sr0 (ENOMEDIUM or ENOTBLK) 2013-08-06_17:48:05.33940 #[598] INFO: Waiting for /dev/sr0 (ENOMEDIUM or ENOTBLK) 2013-08-06_17:48:05.84105 #[598] INFO: Waiting for /dev/sr0 (ENOMEDIUM or ENOTBLK) 2013-08-06_17:48:06.34252 #[598] INFO: Waiting for /dev/sr0 (ENOMEDIUM or ENOTBLK) 2013-08-06_17:48:06.84413 #[598] INFO: failed to load infrastructure settings: No bosh cdrom env: #<Errno::ENOMEDIUM: No medium found - /dev/sr0> 2013-08-06_17:48:06.84504 #[598] INFO: loaded cached settings: {"vm"=>{"name"=>"bm-121d295b-6f2d-4d64-92c4-2ae1165fddb4", "id"=>"urn:vcloud:vm:8fe91b73-fa93-403e-8a75-0558bbb23333"}, "agent_id"=>"bm-121d295b-6f2d-4d64-92c4-2ae1165fddb4", "networks"=>{"bosh"=>{"cloud_properties"=>{"name"=>"Fundaments_LAN"}, "netmask"=>"255.255.255.0", "gateway"=>"192.168.1.1", "ip"=>"192.168.1.45", "dns"=>["93.186.179.179", "93.186.180.180"], "type"=>nil, "default"=>["dns", "gateway"], "mac"=>"00:50:56:01:01:86"}}, "disks"=>{"system"=>"0", "ephemeral"=>"1", "persistent"=>{}}, "ntp"=>["93.186.180.180"], "blobstore"=>{"provider"=>"local", "options"=>{"blobstore_path"=>"/var/vcap/micro_bosh/data/cache"}}, "mbus"=>"https://vcap:b00tstrap@0.0.0.0:6868", "env"=>{"bosh"=>{"password"=>nil}}} 2013-08-06_17:48:06.84532 #[598] INFO: Loaded settings: #<Bosh::Agent::Settings:0x00000001e27e98 @settings={"vm"=>{"name"=>"bm-121d295b-6f2d-4d64-92c4-2ae1165fddb4", "id"=>"urn:vcloud:vm:8fe91b73-fa93-403e-8a75-0558bbb23333"}, "agent_id"=>"bm-121d295b-6f2d-4d64-92c4-2ae1165fddb4", "networks"=>{"bosh"=>{"cloud_properties"=>{"name"=>"Fundaments_LAN"}, "netmask"=>"255.255.255.0", "gateway"=>"192.168.1.1", "ip"=>"192.168.1.45", "dns"=>["93.186.179.179", "93.186.180.180"], "type"=>nil, "default"=>["dns", "gateway"], "mac"=>"00:50:56:01:01:86"}}, "disks"=>{"system"=>"0", "ephemeral"=>"1", "persistent"=>{}}, "ntp"=>["93.186.180.180"], "blobstore"=>{"provider"=>"local", "options"=>{"blobstore_path"=>"/var/vcap/micro_bosh/data/cache"}}, "mbus"=>"https://vcap:b00tstrap@0.0.0.0:6868", "env"=>{"bosh"=>{"password"=>nil}}}, @cache_file="/var/vcap/bosh/settings.json"> 2013-08-06_17:48:06.85756 #[598] INFO: Configure ntp-servers: 93.186.180.180 2013-08-06_17:48:06.96281 #[598] INFO: 2013-08-06_17:48:06.96732 #[598] INFO: Swapon partition /dev/sdb1 2013-08-06_17:48:06.99238 #[598] INFO: Mount data partition /dev/sdb2 to /var/vcap/data 2013-08-06_17:48:07.12150 mke2fs 1.41.11 (14-Mar-2010) 2013-08-06_17:48:07.73500 #[598] INFO: Monit: /var/vcap/bosh/etc/monitrc:8: Warning: include files not found '/var/vcap/monit/job/*.monitrc' 2013-08-06_17:48:07.73517 #[598] INFO: Monit: Starting monit daemon with http interface at [127.0.0.1:2822] 2013-08-06_17:48:08.73368 #[598] INFO: Monit Service Connection Refused: retrying 2013-08-06_17:48:08.74064 #[598] INFO: Starting up https agent 2013-08-06_17:48:08.79310 #[598] INFO: Message processors: {"migrate_disk"=>Bosh::Agent::Message::MigrateDisk, "list_disk"=>Bosh::Agent::Message::ListDisk, "mount_disk"=>Bosh::Agent::Message::MountDisk, "unmount_disk"=>Bosh::Agent::Message::UnmountDisk, "state"=>Bosh::Agent::Message::State, "drain"=>Bosh::Agent::Message::Drain, "apply"=>Bosh::Agent::Message::Apply, "compile_package"=>Bosh::Agent::Message::CompilePackage, "fetch_logs"=>Bosh::Agent::Message::FetchLogs, "ssh"=>Bosh::Agent::Message::Ssh, "ping"=>Bosh::Agent::Message::Ping, "noop"=>Bosh::Agent::Message::Noop, "start"=>Bosh::Agent::Message::Start, "stop"=>Bosh::Agent::Message::Stop, "prepare_network_change"=>Bosh::Agent::Message::PrepareNetworkChange, "release_apply_spec"=>Bosh::Agent::Message::ReleaseApplySpec} 2013-08-06_17:48:08.84352 >> Thin web server (v1.5.1 codename Straight Razor) 2013-08-06_17:48:08.84357 >> Maximum connections set to 1024 2013-08-06_17:48:08.84360 >> Listening on 0.0.0.0:6868, CTRL+C to stop

gem list:

rik@boshv2:~/deployments$ gem list

* LOCAL GEMS *

agent_client (1.5.0.pre.870) aws-sdk (1.8.5) blobstore_client (1.5.0.pre.870, 1.5.0.pre.3) bosh_aws_cpi (1.5.0.pre.870) bosh_cli (1.5.0.pre.870, 1.5.0.pre.3) bosh_cli_plugin_micro (1.5.0.pre.870) bosh_common (1.5.0.pre.870, 1.5.0.pre.3) bosh_cpi (1.5.0.pre.870) bosh_openstack_cpi (1.5.0.pre.870) bosh_registry (1.5.0.pre.870) bosh_vcloud_cpi (0.4.9) bosh_vsphere_cpi (1.5.0.pre.870) builder (3.1.4) bundler (1.3.5) bundler-unload (1.0.1) daemons (1.1.9) eventmachine (1.0.3) excon (0.25.3) fog (1.14.0) formatador (0.2.4) highline (1.6.19) httparty (0.11.0) httpclient (2.2.4) json (1.8.0) json_pure (1.8.0) log4r (1.1.10) membrane (0.0.2) mime-types (1.23) mini_portile (0.5.1) minitar (0.5.4) multi_json (1.7.8) multi_xml (0.5.5) net-scp (1.1.2) net-ssh (2.6.8) net-ssh-gateway (1.1.0) netaddr (1.5.0) nokogiri (1.5.10) progressbar (0.9.2) rack (1.5.2) rack-protection (1.5.0) rake (10.1.0) rest-client (1.6.7) ruby-atmos-pure (1.0.5) ruby-hmac (0.4.0) ruby_vcloud_sdk (0.4.8) ruby_vim_sdk (1.5.0.pre.870) rubygems-bundler (1.2.2) rvm (1.11.3.8) sequel (3.43.0) sinatra (1.4.3) sqlite3 (1.3.7) terminal-table (1.4.5) thin (1.5.1) tilt (1.4.1) uuidtools (2.1.4) xml-fu (0.1.9) yajl-ruby (1.1.0)

micro bosh. yml

rik@boshv2:~/deployments/vcloud$ cat micro_bosh.yml

name: vcloud

logging: level: DEBUG

network: ip: 192.168.1.45 netmask: 255.255.255.0 gateway: 192.168.1.1 dns:

resources: persistent_disk: 4096

cloud: plugin: vcloud properties: agent: ntp:

jfoley commented 11 years ago

@Rikbruggink - it looks like there is an issue with loading the CDROM on this instance (failed to load infrastructure settings: No bosh cdrom env: #) which might be the reason why the persistent disk is not being loaded. It appears that the cached settings data which is being loaded does not contain any persistent disk info ("persistent"=>{}). Can you look in to why this CDROM might be failing to mount? The code which mounts the CD is in bosh_agent/lib/bosh_agent/infrastructure/vsphere/settings.rb:49-99 - which could arguably use some cleanup.

For a bit of background, on vSphere, Bosh mounts a CDROM as a way of passing data about the infrastructure from the director to the agent such as persistent disks, network settings, etc.

@aramprice / @jfoley

Rikbruggink commented 11 years ago

Hello John,

I am using the latest bosh version now where khadgils commit is merged and now using the latest microbosh stemcell the agent is responding but it now fails on mounting the persistence disk.

cli output:

root@ubuntu:~/vchsbosh/bosh/deployments# bosh micro deploy micro-bosh-stemcell-latest-vsphere-esxi-ubuntu.tgz Deploying new micro BOSH instance vcloud/micro_bosh.yml' tohttps://192.168.1.10:25555' (type 'yes' to continue): yes

Verifying stemcell... File exists and readable                                     OK Manifest not found in cache, verifying tarball... Read tarball                                                 OK Manifest exists                                              OK Stemcell image file                                          OK Writing manifest to cache... Stemcell properties                                          OK

Stemcell info

Name:    micro-bosh-stemcell Version: 896

Deploy Micro BOSH   unpacking stemcell (00:00:03)   uploading stemcell (00:02:39)   creating VM from urn:vcloud:catalogitem:8e470bfe-329c-48bb-96cb-5f570aa26759 (00:01:20)   waiting for the agent (00:00:21) Create disk                         |oooooooo                | 4/11 00:04:30  ETA: 00:07:38/usr/local/rvm/gems/ruby-1.9.3-p448/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in return!': 400 Bad Request (RestClient::BadRequest)         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/bosh_vcloud_cpi-0.5.0/lib/cloud/vcloud/vcd_client.rb:135:inblock in invoke'         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in call'         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/rest-client-1.6.7/lib/restclient/request.rb:228:inprocess_result'         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/rest-client-1.6.7/lib/restclient/request.rb:178:in block in transmit'         from /usr/local/rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/http.rb:746:instart'         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/rest-client-1.6.7/lib/restclient/request.rb:172:in transmit'         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/rest-client-1.6.7/lib/restclient/request.rb:64:inexecute'         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in execute'         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/bosh_vcloud_cpi-0.5.0/lib/cloud/vcloud/vcd_client.rb:133:ininvoke'         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/bosh_vcloud_cpi-0.5.0/lib/cloud/vcloud/vcd_client.rb:151:in invoke_and_wait'         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/bosh_vcloud_cpi-0.5.0/lib/cloud/vcloud/steps/attach_detach_disk.rb:7:inperform'         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/bosh_vcloud_cpi-0.5.0/lib/cloud/vcloud/steps.rb:37:in next'         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/bosh_vcloud_cpi-0.5.0/lib/cloud/vcloud/cloud.rb:209:inblock in attach_disk'         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/bosh_vcloud_cpi-0.5.0/lib/cloud/vcloud/steps.rb:44:in call'         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/bosh_vcloud_cpi-0.5.0/lib/cloud/vcloud/steps.rb:44:inblock in perform'         from /root/vchsbosh/bosh/bosh_common/lib/common/thread_formatter.rb:46:in with_thread_name'         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/bosh_vcloud_cpi-0.5.0/lib/cloud/vcloud/steps.rb:41:inperform'         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/bosh_vcloud_cpi-0.5.0/lib/cloud/vcloud/steps.rb:58:in perform'         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/bosh_vcloud_cpi-0.5.0/lib/cloud/vcloud/cloud.rb:269:insteps'         from /usr/local/rvm/gems/ruby-1.9.3-p448/gems/bosh_vcloud_cpi-0.5.0/lib/cloud/vcloud/cloud.rb:202:in attach_disk'         from /root/vchsbosh/bosh/bosh_cli_plugin_micro/lib/deployer/instance_manager.rb:308:inattach_disk'         from /root/vchsbosh/bosh/bosh_cli_plugin_micro/lib/deployer/instance_manager.rb:345:in update_persistent_disk'         from /root/vchsbosh/bosh/bosh_cli_plugin_micro/lib/deployer/instance_manager.rb:156:inblock in create'         from /root/vchsbosh/bosh/bosh_cli_plugin_micro/lib/deployer/instance_manager.rb:92:in step'         from /root/vchsbosh/bosh/bosh_cli_plugin_micro/lib/deployer/instance_manager.rb:155:increate'         from /root/vchsbosh/bosh/bosh_cli_plugin_micro/lib/deployer/instance_manager.rb:112:in block in create_deployment'         from /root/vchsbosh/bosh/bosh_cli_plugin_micro/lib/deployer/instance_manager.rb:105:inwith_lifecycle'         from /root/vchsbosh/bosh/bosh_cli_plugin_micro/lib/deployer/instance_manager.rb:111:in create_deployment'         from /root/vchsbosh/bosh/bosh_cli_plugin_micro/lib/bosh/cli/commands/micro.rb:172:inperform'         from /root/vchsbosh/bosh/bosh_cli/lib/cli/command_handler.rb:57:in run'         from /root/vchsbosh/bosh/bosh_cli/lib/cli/runner.rb:59:inrun'         from /root/vchsbosh/bosh/bosh_cli/lib/cli/runner.rb:18:in run'         from /root/vchsbosh/bosh/bosh_cli/bin/bosh:7:in<top (required)>'         from /usr/local/rvm/gems/ruby-1.9.3-p448/bin/bosh:19:in load'         from /usr/local/rvm/gems/ruby-1.9.3-p448/bin/bosh:19:in

'         from /usr/local/rvm/gems/ruby-1.9.3-p448/bin/ruby_noexec_wrapper:14:in eval'         from /usr/local/rvm/gems/ruby-1.9.3-p448/bin/ruby_noexec_wrapper:14:in
'

tail of debug log output from the new job:

          ovfenv:PlatformSection             ovfenv:KindVMware ESXi/ovfenv:Kind             ovfenv:Version5.1.0/ovfenv:Version             ovfenv:VendorVMware, Inc./ovfenv:Vendor             ovfenv:Localeen/ovfenv:Locale         /ovfenv:PlatformSection                    

        /ve:EthernetAdapterSection     /ovfenv:Environment                     false         false        

D, [2013-08-16T12:29:54.007956 #18376] [attachdisk(urn:vcloud:vm:a97d898e-1756-4602-8380-393372b7bb2c, urn:vcloud:disk:43eadca3-7dea-4f39-a1ec-0087a79e8df8)] DEBUG -- : REST REQ GET https://31.200.210.84/api/entity/urn:vcloud:disk:43eadca3-7dea-4f39-a1ec-0087a79e8df8 {:Accept=>"application/+xml;version=5.1", :contenttype=>"/*", :x_vcloud_authorization=>"5f5eABNkYYuMYmXC7/R1V6w5vjNZL7jRzv1MUTp4Qh0="} {"vcloud-token"=>"5f5eABNkYYuMYmXC7%2FR1V6w5vjNZL7jRzv1MUTp4Qh0%3D", "Path"=>"%2F"} D, [2013-08-16T12:29:54.038232 #18376] [attach_disk(urn:vcloud:vm:a97d898e-1756-4602-8380-393372b7bb2c, urn:vcloud:disk:43eadca3-7dea-4f39-a1ec-0087a79e8df8)] DEBUG -- : REST RES 200 {:date=>"Fri, 16 Aug 2013 10:29:57 GMT", :content_type=>"application/vnd.vmware.vcloud.entity+xml;version=5.1", :content_encoding=>"gzip", :content_length=>"311"} <?xml version="1.0" encoding="UTF-8"?>

   

D, [2013-08-16T12:29:54.039279 #18376] [attachdisk(urn:vcloud:vm:a97d898e-1756-4602-8380-393372b7bb2c, urn:vcloud:disk:43eadca3-7dea-4f39-a1ec-0087a79e8df8)] DEBUG -- : REST REQ GET https://31.200.210.84/api/disk/43eadca3-7dea-4f39-a1ec-0087a79e8df8 {:Accept=>"application/+xml;version=5.1", :contenttype=>"/*", :x_vcloud_authorization=>"5f5eABNkYYuMYmXC7/R1V6w5vjNZL7jRzv1MUTp4Qh0="} {"vcloud-token"=>"5f5eABNkYYuMYmXC7%2FR1V6w5vjNZL7jRzv1MUTp4Qh0%3D", "Path"=>"%2F"} D, [2013-08-16T12:29:54.080474 #18376] [attach_disk(urn:vcloud:vm:a97d898e-1756-4602-8380-393372b7bb2c, urn:vcloud:disk:43eadca3-7dea-4f39-a1ec-0087a79e8df8)] DEBUG -- : REST RES 200 {:date=>"Fri, 16 Aug 2013 10:29:57 GMT", :content_type=>"application/vnd.vmware.vcloud.disk+xml;version=5.1", :content_encoding=>"gzip", :content_length=>"592"} <?xml version="1.0" encoding="UTF-8"?>

                                           

D, [2013-08-16T12:29:54.081155 #18376] [attach_disk(urn:vcloud:vm:a97d898e-1756-4602-8380-393372b7bb2c, urn:vcloud:disk:43eadca3-7dea-4f39-a1ec-0087a79e8df8)] DEBUG -- : STEP VCloudCloud::Steps::AttachDetachDisk D, [2013-08-16T12:29:54.082632 #18376] [attach_disk(urn:vcloud:vm:a97d898e-1756-4602-8380-393372b7bb2c, urn:vcloud:disk:43eadca3-7dea-4f39-a1ec-0087a79e8df8)] DEBUG -- : REST REQ POST https://31.200.210.84/api/vApp/vm-a97d898e-1756-4602-8380-393372b7bb2c/disk/action/attach {:Accept=>"application/*+xml;version=5.1", :content_type=>"application/vnd.vmware.vcloud.diskAttachOrDetachParams+xml", :x_vcloud_authorization=>"5f5eABNkYYuMYmXC7/R1V6w5vjNZL7jRzv1MUTp4Qh0="} {"vcloud-token"=>"5f5eABNkYYuMYmXC7%2FR1V6w5vjNZL7jRzv1MUTp4Qh0%3D", "Path"=>"%2F"}   D, [2013-08-16T12:29:54.174318 #18376] [attach_disk(urn:vcloud:vm:a97d898e-1756-4602-8380-393372b7bb2c, urn:vcloud:disk:43eadca3-7dea-4f39-a1ec-0087a79e8df8)] DEBUG -- : REST RES 400 {:date=>"Fri, 16 Aug 2013 10:29:57 GMT", :content_type=>"application/vnd.vmware.vcloud.error+xml;version=5.1", :content_length=>"446"} <?xml version="1.0" encoding="UTF-8"?>

E, [2013-08-16T12:29:54.174909 #18376] [attach_disk(urn:vcloud:vm:a97d898e-1756-4602-8380-393372b7bb2c, urn:vcloud:disk:43eadca3-7dea-4f39-a1ec-0087a79e8df8)] ERROR -- : FAIL attach_disk(urn:vcloud:vm:a97d898e-1756-4602-8380-393372b7bb2c, urn:vcloud:disk:43eadca3-7dea-4f39-a1ec-0087a79e8df8) 400 Bad Request D, [2013-08-16T12:29:54.175056 #18376] [attach_disk(urn:vcloud:vm:a97d898e-1756-4602-8380-393372b7bb2c, urn:vcloud:disk:43eadca3-7dea-4f39-a1ec-0087a79e8df8)] DEBUG -- : REV ROLLBACK VCloudCloud::Steps::AttachDetachDisk D, [2013-08-16T12:29:54.175156 #18376] [attach_disk(urn:vcloud:vm:a97d898e-1756-4602-8380-393372b7bb2c, urn:vcloud:disk:43eadca3-7dea-4f39-a1ec-0087a79e8df8)] DEBUG -- : REV CLEANUP VCloudCloud::Steps::AttachDetachDisk

mkocher commented 11 years ago

The bosh core team is not maintaining the vCloud CPI at this point, it's moved over to https://github.com/vchs/bosh_vcloud_cpi, I'm going to close this but you're welcome to open an issue over there and point to this one.

This discussion sounds like it might also get better results on the bosh-users mailing list.