cloudstack xen compatible stemcell

[poblin-orange]

We do not have yet cloudstack compatible official stemcell.

For now development of the CPI were based on openstack-kvm-ubuntu, go agent 3300 stemcell, converted to VHD.

Here is the list of CloudStack specifics that the stemcell would require:

• specific default agent.json config:

CPI bootstraps the vms by setting UserData on the virtual machine (registry endpoint, etc ...) Cloudstack has a specific metadata http url, referring to its vrouter vm (not a fixed ip, as in openstack). A workaround was to change metadata url in /var/vcap/bosh/agent.json, but its not sufficient, as the url depends on the vrouter IP address by default.

• hypervisor support to ease vm live migrations (xen tools)

[gberche-orange]

Configuration of the vrouter is discussed into http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201508.mbox/%3CD1E8F7FA.63241%25chiradeep.vittal@citrix.com%3E but it seems that this would not work with vrouter in advanced zone, or require the vrouter to be configured as the default gateway (which is not yet the case within orange network setting)

[chiradeep]

Static route to send 169.254.169.254 to VR in case VR is not default router can be specified in DHCP config (dhcp option 121). See https://github.com/apache/cloudstack/blob/94f1ebb60af320166206c213e18b97a177bc145a/systemvm/patches/debian/config/opt/cloud/bin/edithosts.sh#L202

[poblin-orange]

@chirad does the solution you propose imply custom cloudstack system vm, or can be activated with cloudstack properties ?

[chiradeep]

Not a custom system vm, but a change to the scripts that are downloaded to the systemvm. Alternatively, customize your templates (or cloud-init) to add a route to the systemvm. E.g. dhclient script: /etc/dhclient-exit-hooks: srvr=$(grep server-identifier /var/lib/dhcp/dhclient.leases | tail -1 | awk -F" " '{print $3}') srvr=${srvr:0:-1} ip route add 169.254.169.254 via $srvr

see here how to canonically determine VR address https://github.com/number5/cloud-init/blob/master/cloudinit/sources/DataSourceCloudStack.py#L224

[poblin-orange]

probably we should add this step to the cloudstack stemcell, to avoid iaas specific configuration.

[chiradeep]

Agreed.

[poblin-orange]

Current progress. preparing a PR so cloudstack xen stemcell is generated with other stemcells

Inspired from softlayer PR to generate stemcell => commit on bosh-orange to generate cloudstack-xen https://github.com/Orange-OpenSource/bosh/commit/fd3d5a645c6f41ca438f23ddc1cfc6b7a68e1941

generation OK see http://84.39.33.56:8080/pipelines/cloudstack-stemcell/jobs/build-ubuntu-trusty/builds/21

stemcell upload to bosh ok, template registration in cloudstack is KO : Failed post download script: vhd check of /mnt/SecStorage/d9b0e175-26da-36ce-afe7-911c38b1e27f/template/tmpl/6/260/dnld6935681358325791994tmp_.tmp failed

=> should fix qemu vhd generation for cloudstack xen (difference with hyperV vhd)

[dpassante]

Hello,

Vhd-util semble être le seul moyen de générer un vhd valide pour XenServer et la version officielle ne semble pas compiler sur un environnement non-xenserver. Il existe une version patchée ici : https://github.com/rubiojr/vhd-util-convert qui permet la conversion au format VHD.

Si l’image est au format qcow : qemu-img convert -O raw root.qcow root.img

Pour la convertir en vhd: vhd-util convert -i root.img -s0 -t1 -o 0.vhd vhd-util convert -i root.img -s1 -t2 -i 0.vhd -o bosh-stemcell-3147-cloudstack-xen-ubuntu-trusty-go_agent.vhd

Verification: $ vhd-util check -n bosh-stemcell-3147-cloudstack-xen-ubuntu-trusty-go_agent.vhd bosh-stemcell-3147-cloudstack-xen-ubuntu-trusty-go_agent.vhd is valid

Il existe aussi un problème avec la date inscrite dans le vhd qui peut être vue comme future par CloudStack (pb timezone). Contournable avec faketime. sudo apt-get install faketime (ou à compiler depuis les sources dispo sur github) faketime -f "-12h" vhd-util…

Bon Week end

A+ David

[poblin-orange]

stemcell build now uses vhd-util. The resulting vhd template is zipped (bzip), no tar accepted by cloudstack. http://84.39.33.56:8080/pipelines/cloudstack-stemcell/jobs/build-ubuntu-trusty/builds/34 The template registering in cloudstack is now ok. Still vr_metadata iptable patch is ko

[poblin-orange]

vr_metadata fixed, by moving it to /etc/network/if-up.d (/var/lib/dhcp lease file not yet available from dhcp hook ?). Stemcell ok with micro-bosh

[poblin-orange]

stemcell 3165 tested successfully on cloudfoundry and diego deployments https://s3.amazonaws.com/orange-cloudstack-xen-stemcell/bosh-stemcell-3165-cloudstack-xen-ubuntu-trusty-go_agent.tgz

generation pipeline http://84.39.33.56:8080/pipelines/cloudstack-stemcell

[poblin-orange]

stemcell 3173 tested successfully

Following up with a bosh PR

[poblin-orange]

align with bosh 255 pipeline https://concourse.cw-lz-api.elpaaso.net/pipelines/cloudstack-stemcell downloadable 3195 stemcell https://s3.amazonaws.com/orange-candidate-cloudstack-xen-stemcell/bosh-stemcell/cloudstack/bosh-stemcell-3215-cloudstack-xen-ubuntu-trusty-go_agent.tgz

[poblin-orange]

vagrant manual build for 3262.3, including xen tools 6.5 https://orange-candidate-cloudstack-xen-stemcell.s3.amazonaws.com/bosh-stemcell/cloudstack/bosh-stemcell-3262.3-cloudstack-xen-ubuntu-trusty-go_agent.tgz tested ok with bosh 257.x + cloudstack 4.7 + xen 6.5

NB: from 3262.3 stemcell, the password authentication is DISABLED => must use bosh ssh, direct ssh requires the private key configured in cloudstack /cpi see https://github.com/cloudfoundry-community/bosh-cloudstack-cpi-release/issues/14, http://bosh.io/jobs/cloudstack_cpi?source=github.com/cloudfoundry-community/bosh-cloudstack-cpi-release&version=12#p=cloudstack.private_key

[poblin-orange]

docker manual build for 3262.5 xen tools 6.5, with vhd dynamic format (see #22) http://orange-candidate-cloudstack-xen-stemcell.s3.amazonaws.com/bosh-stemcell/cloudstack/bosh-stemcell-3262.5-cloudstack-xen-ubuntu-trusty-go_agent.tgz

[poblin-orange]

docker manual build for 3262.9 xentools 6.5 including fix for #38 (garden issue with xentools)

http://orange-candidate-cloudstack-xen-stemcell.s3.amazonaws.com/bosh-stemcell/cloudstack/bosh-stemcell-3262.9-cloudstack-xen-ubuntu-trusty-go_agent.tgz

rebased on https://github.com/cloudfoundry/bosh/releases/tag/stable-3262.9

[poblin-orange]

docker manual build for latest stemcell

https://orange-candidate-cloudstack-xen-stemcell.s3.amazonaws.com/bosh-stemcell/cloudstack/bosh-stemcell-3262.22-cloudstack-xen-ubuntu-trusty-go_agent.tgz rebased on https://github.com/cloudfoundry/bosh/releases/tag/stable-3262.21

https://orange-candidate-cloudstack-xen-stemcell.s3.amazonaws.com/bosh-stemcell/cloudstack/bosh-stemcell-3263.10-cloudstack-xen-ubuntu-trusty-go_agent.tgz

kernel 4.4, rebased on https://github.com/cloudfoundry/bosh/releases/tag/stable-3263.10

[lcacciagioni]

where can I put all the likes!!! Thanks guys for the awesome work!!!

[poblin-orange]

@lcacciagioni thks. Would love to hear from you if you used the cpi (feedbacks / usage context / improvment suggestions).