Cloud-init not executing with instructions

[asimkhawaja]

Hi Guys, I am running the yaml file below using plumbery in a CaaS VM. Looks like my cloud-config params from fittings file is being picked up by plumbery but none of the instructions are actually being executed on the machine. I have verfied by logging into the machine and checking as well.

asim_fittings.yaml

information:
    - "Setup mysql servers for testing on Ubuntu /Redhat "
links:
  documentation: https://plumbery.readthedocs.org/en/latest/tutorial.apache2.cluster.html

defaults:

  cloud-config:
    disable_root: false
    ssh_pwauth: true

---
# London
locationId: AU10
regionId: dd-au

blueprints:

  - mysql:

      domain:
        name: BaaS_Test_AU10

        service: advanced

      ethernet:
        name: "baas_vlan"
        subnet: 10.0.0.0

      nodes:
        - mysqltest_6:
           information : "mysql base node for ubuntu"
           appliance : 'Ubuntu 14'
           cloud-config:
             hostname: "mysqltest6"
             package_upgrade: true
             packages:
               - git
               - ntp
               - mysql-server-5.6
             final_message: "Should be done in $UPTIME seconds"
             write_files:

                # a script in awk to update the /etc/hosts file
                #
                - path: /root/master_db.sql
                  content: |
                    CREATE DATABASE db01;
                    USE db01;
                    CREATE USER 'dbuser'@'localhost' IDENTIFIED BY '{{ dbuser.secret }}';
                    GRANT ALL PRIVILEGES ON db01.* TO 'dbuser'@'localhost';
                    CREATE TABLE persons (id INT AUTO_INCREMENT,
                      name VARCHAR(30),
                      datum TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
                      PRIMARY KEY(id));
                    INSERT INTO persons(name) VALUES ('Marko');
                    INSERT INTO persons(name) VALUES ('John');
                    INSERT INTO persons(name) VALUES ('Eric');
                    INSERT INTO persons(name) VALUES ('Clive');
                    INSERT INTO persons(name) VALUES ('Maria');

             runcmd:
               - echo "============Finishing Cloud config ===="
               - ls -l

output log:

root@10-0-0-8:~# python -m plumbery -d asim_fittings.yaml deploy
Default values:
- cloud-config: {'ssh_pwauth': True, 'disable_root': False}
Adding facility
- regionId: dd-au
- locationId: AU10
Loading secrets from 'f42ea87e61a991c7c206345494820020.secrets'
Building all blueprints
Getting driver for 'dd-au / None'
Getting location 'AU10'
Plumbing at 'AU10' Australia - Melbourne MCP2 (Australia)
- available blueprints: 'mysql'
Building infrastructure of blueprint 'mysql'
Listing network domains
- found 2 network domains
Creating network domain 'BaaS_Test_AU10'
- already there
Listing Ethernet networks
- found 2 Ethernet networks
Creating Ethernet network 'baas_vlan'
- already there
Building nodes of blueprint 'mysql'
Creating node 'mysqltest_6'
- using image 'Ubuntu 14.04 2 CPU'
- in progress
Importing 'polishers.configure'
Instantiating 'ConfigurePolisher'
Using polisher 'configure'
Polishing all blueprints
Plumbing at 'AU10' Australia - Melbourne MCP2 (Australia)
- available blueprints: 'mysql'
Polishing blueprint 'mysql'
Configuring blueprint 'mysql'
- waiting for nodes to be deployed
- nodes have been deployed
Configuring node 'mysqltest_6'
- no change in compute
Starting nodes from all blueprints
Plumbing at 'AU10' Australia - Melbourne MCP2 (Australia)
- available blueprints: 'mysql'
Starting node 'mysqltest_6'
- in progress
Importing 'polishers.prepare'
Instantiating 'PreparePolisher'
Using polisher 'prepare'
Polishing all blueprints
Plumbing at 'AU10' Australia - Melbourne MCP2 (Australia)
- available blueprints: 'mysql'
Polishing blueprint 'mysql'
- beachheading at 'AU10'
preparing node 'mysqltest_6'
- using cloud-config
- preparing meta-data
- preparing user-data
- serializing object before expansion
- generating dbuser.secret
- 'dbuser.secret' -> 's7yH859Oi'
#cloud-config
disable_root: false
package_upgrade: true
hostname: mysqltest6
write_files:
  - content: |
        CREATE DATABASE db01;
        USE db01;
        CREATE USER 'dbuser'@'localhost' IDENTIFIED BY 's7yH859Oi';
        GRANT ALL PRIVILEGES ON db01.* TO 'dbuser'@'localhost';
        CREATE TABLE persons (id INT AUTO_INCREMENT,
          name VARCHAR(30),
          datum TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
          PRIMARY KEY(id));
        INSERT INTO persons(name) VALUES ('Marko');
        INSERT INTO persons(name) VALUES ('John');
        INSERT INTO persons(name) VALUES ('Eric');
        INSERT INTO persons(name) VALUES ('Clive');
        INSERT INTO persons(name) VALUES ('Maria');

    path: /root/master_db.sql
runcmd:
  - echo "============Finishing Cloud config ===="
  - ls -l
final_message: Should be done in $UPTIME seconds
packages:
  - git
  - ntp
  - mysql-server-5.6
ssh_pwauth: true

- preparing remote install of cloud-init
- preparing reboot to trigger cloud-init
Connecting to server
starting thread (client mode): 0xa5ec6910L
Local version/idstring: SSH-2.0-paramiko_2.0.0
Remote version/idstring: SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
Connected (version 2.0, client OpenSSH_6.6.1p1)
kex algos:[u'curve25519-sha256@libssh.org', u'ecdh-sha2-nistp256', u'ecdh-sha2-nistp384', u'ecdh-sha2-nistp521', u'diffie-hellman-group-exchange-sha256', u'diffie-hellman-group-exchange-sha1', u'diffie-hellman-group14-sha1', u'diffie-hellman-group1-sha1'] server key:[u'ssh-rsa', u'ssh-dss', u'ecdsa-sha2-nistp256'] client encrypt:[u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'arcfour256', u'arcfour128', u'aes128-gcm@openssh.com', u'aes256-gcm@openssh.com', u'chacha20-poly1305@openssh.com', u'aes128-cbc', u'3des-cbc', u'blowfish-cbc', u'cast128-cbc', u'aes192-cbc', u'aes256-cbc', u'arcfour', u'rijndael-cbc@lysator.liu.se'] server encrypt:[u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'arcfour256', u'arcfour128', u'aes128-gcm@openssh.com', u'aes256-gcm@openssh.com', u'chacha20-poly1305@openssh.com', u'aes128-cbc', u'3des-cbc', u'blowfish-cbc', u'cast128-cbc', u'aes192-cbc', u'aes256-cbc', u'arcfour', u'rijndael-cbc@lysator.liu.se'] client mac:[u'hmac-md5-etm@openssh.com', u'hmac-sha1-etm@openssh.com', u'umac-64-etm@openssh.com', u'umac-128-etm@openssh.com', u'hmac-sha2-256-etm@openssh.com', u'hmac-sha2-512-etm@openssh.com', u'hmac-ripemd160-etm@openssh.com', u'hmac-sha1-96-etm@openssh.com', u'hmac-md5-96-etm@openssh.com', u'hmac-md5', u'hmac-sha1', u'umac-64@openssh.com', u'umac-128@openssh.com', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-ripemd160', u'hmac-ripemd160@openssh.com', u'hmac-sha1-96', u'hmac-md5-96'] server mac:[u'hmac-md5-etm@openssh.com', u'hmac-sha1-etm@openssh.com', u'umac-64-etm@openssh.com', u'umac-128-etm@openssh.com', u'hmac-sha2-256-etm@openssh.com', u'hmac-sha2-512-etm@openssh.com', u'hmac-ripemd160-etm@openssh.com', u'hmac-sha1-96-etm@openssh.com', u'hmac-md5-96-etm@openssh.com', u'hmac-md5', u'hmac-sha1', u'umac-64@openssh.com', u'umac-128@openssh.com', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-ripemd160', u'hmac-ripemd160@openssh.com', u'hmac-sha1-96', u'hmac-md5-96'] client compress:[u'none', u'zlib@openssh.com'] server compress:[u'none', u'zlib@openssh.com'] client lang:[u''] server lang:[u''] kex follows?False
Kex agreed: diffie-hellman-group1-sha1
Cipher agreed: aes128-ctr
MAC agreed: hmac-sha2-256
Compression agreed: none
kex engine KexGroup1 specified hash_algo <built-in function openssl_sha1>
Switch to new keys ...
Adding ssh-rsa host key for 2402:9900:211:1199:fa6:e425:1afa:7f0c: 7cb27ae7eb5e536984922f2ca3e68ce4
Trying key 931ae5f467adb6b075985f55a0fa5789 from /root/.ssh/id_rsa
userauth is OK
Authentication (publickey) failed.
userauth is OK
Authentication (password) successful!
Uploading file
[chan 0] Max packet in: 32768 bytes
[chan 0] Max packet out: 32768 bytes
Secsh channel 0 opened.
[chan 0] Sesch channel 0 request ok
[chan 0] Opened sftp connection (server version 3)
[chan 0] stat('/')
[chan 0] normalize('/')
[chan 0] mkdir('/var', 511)
[chan 0] stat('/var')
[chan 0] normalize('/var')
[chan 0] mkdir('/var/lib', 511)
[chan 0] stat('/var/lib')
[chan 0] normalize('/var/lib')
[chan 0] mkdir('/var/lib/cloud', 511)
[chan 0] stat('/var/lib/cloud')
[chan 0] normalize('/var/lib/cloud')
[chan 0] mkdir('/var/lib/cloud/seed', 511)
[chan 0] stat('/var/lib/cloud/seed')
[chan 0] normalize('/var/lib/cloud/seed')
[chan 0] mkdir('/var/lib/cloud/seed/nocloud-net', 511)
[chan 0] stat('/var/lib/cloud/seed/nocloud-net')
[chan 0] normalize('/var/lib/cloud/seed/nocloud-net')
[chan 0] open('/var/lib/cloud/seed/nocloud-net/meta-data', 'w')
[chan 0] open('/var/lib/cloud/seed/nocloud-net/meta-data', 'w') -> 00000000
[chan 0] close(00000000)
[chan 0] sftp session closed.
[chan 0] EOF sent (0)
Uploading file
[chan 1] Max packet in: 32768 bytes
[chan 0] EOF received (0)
[chan 1] Max packet out: 32768 bytes
Secsh channel 1 opened.
[chan 1] Sesch channel 1 request ok
[chan 1] Opened sftp connection (server version 3)
[chan 1] stat('/')
[chan 1] normalize('/')
[chan 1] mkdir('/var', 511)
[chan 1] stat('/var')
[chan 1] normalize('/var')
[chan 1] mkdir('/var/lib', 511)
[chan 1] stat('/var/lib')
[chan 1] normalize('/var/lib')
[chan 1] mkdir('/var/lib/cloud', 511)
[chan 1] stat('/var/lib/cloud')
[chan 1] normalize('/var/lib/cloud')
[chan 1] mkdir('/var/lib/cloud/seed', 511)
[chan 1] stat('/var/lib/cloud/seed')
[chan 1] normalize('/var/lib/cloud/seed')
[chan 1] mkdir('/var/lib/cloud/seed/nocloud-net', 511)
[chan 1] stat('/var/lib/cloud/seed/nocloud-net')
[chan 1] normalize('/var/lib/cloud/seed/nocloud-net')
[chan 1] open('/var/lib/cloud/seed/nocloud-net/user-data', 'w')
[chan 1] open('/var/lib/cloud/seed/nocloud-net/user-data', 'w') -> 00000000
[chan 1] close(00000000)
[chan 1] sftp session closed.
[chan 1] EOF sent (1)
Uploading file
[chan 1] EOF received (1)
[chan 2] Max packet in: 32768 bytes
[chan 2] Max packet out: 32768 bytes
Secsh channel 2 opened.
[chan 2] Sesch channel 2 request ok
[chan 2] Opened sftp connection (server version 3)
[chan 2] stat('.')
[chan 2] normalize('.')
[chan 2] open('/root/prepare.cloud-init.sh', 'w')
[chan 2] open('/root/prepare.cloud-init.sh', 'w') -> 00000000
[chan 2] chmod(00000000, 493)
[chan 2] close(00000000)
[chan 2] sftp session closed.
[chan 2] EOF sent (2)
Executing command
[chan 2] EOF received (2)
[chan 3] Max packet in: 32768 bytes
[chan 3] Max packet out: 32768 bytes
Secsh channel 3 opened.
[chan 3] Sesch channel 3 request ok
[chan 3] EOF received (3)
[chan 3] EOF sent (3)
Command finished
Closing server connection
EOF in transport thread
- done
Importing 'polishers.information'
Instantiating 'InformationPolisher'
Using polisher 'information'
Polishing all blueprints
Plumbing at 'AU10' Australia - Melbourne MCP2 (Australia)
- available blueprints: 'mysql'
Polishing blueprint 'mysql'
- examinating node 'mysqltest_6'
Showing information
About this fittings plan:
- Setup mysql servers for testing on Ubuntu /Redhat
- documentation: https://plumbery.readthedocs.org/en/latest/tutorial.apache2.cluster.html
About 'mysqltest_6':
- state: rebooting
Worked for you for 3 seconds locally, and for 7 minutes in the cloud

[bernard357]

Your log is actually looking good. Cloud-init always applies commands while rebooting the machine, and things done at this stage are not reflected by plumbery. This is inherent to the way cloud-config is working, and not related to plumbery. Maybe you could ssh to the node that has been created, and check the /var/log/cloud-config-init.log file. This is where cloud-config is logging results of your commands.

[bernard357]

Sorry the correct log file is /var/log/cloud-init-output.log

[asimkhawaja]

Hey Bernard, there appears to be no cloud-init log files in the /var/log dir

further more I can also confirm that the prepare.cloud-init.sh file is not and cloud init is not installed on the machine.

What is plumbery supposed to be doing once it puts the prepare.cloud-init.sh onto the machine?

[asimkhawaja]

@bernard357 , on further investigation and based on the output above it is clear now that the prepare-cloud.init.sh file never runs, and we never get cloud-init installed. Since cloud-init is not installed none of the instructions are executed. I have now confirmed that if I install cloud-init manually on the VM and re-run plumbery the packages and instructions get followed.

I am now trying to figure out when and how prepare.cloud-init.sh should run, but any guidance you can provide would be most appreciated

[asimkhawaja]

@bernard357 --- one potential cause of this issue is that the prepare.clould-init.sh has windows line endings so if you try to execute it in linux it throws a bad interpreter error!

[root@10-0-0-9 ~]# ls
anaconda-ks.cfg  c.sh  prepare.cloud-init.sh
[root@10-0-0-9 ~]# ./prepare.cloud-init.sh
: No such file or directory
[root@10-0-0-9 ~]# dos2unix prepare.cloud-init.sh
dos2unix: converting file prepare.cloud-init.sh to Unix format ...
[root@10-0-0-9 ~]# ./prepare.cloud-init.sh
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.steadfast.net
 * extras: mirror.steadfast.net
 * updates: mirror.steadfast.net
Resolving Dependencies
--> Running transaction check
---> Package cloud-init.x86_64 0:0.7.5-10.el7.centos.1 will be installed
--> Processing Dependency: python-requests for package: cloud-init-0.7.5-10.el7.centos.1.x86_64

[asimkhawaja]

Looks like the problem is infact with prepare-.cloud-init.sh

Apparently what is checked into source control is ok However the file that is available as per the build 0.80 using pip has prepare.cloud-init.sh that has incorrect line endings , i.e line endings for windows that will cause problems on linux machines!

[bernard357]

@asimkhawaja, after the upload of prepare.cloud-init.sh plumbery just runs it to install cloud-init. Your feedback on newlines is surprising since we are using plumbery on Ubuntu and CentOS images on a daily basis. Which image are you using here?

[asimkhawaja]

I am testing on Cetos 7 and ubuntu 14

Here is the docker file that is being used to run plumbery:

FROM centos:7
#comment
MAINTAINER A.K Asim Khawaja
#set up data directoty for mongodb
RUN mkdir -p /data/db
RUN mkdir -p /var/log/mongodb.log
RUN mkdir -p /root/Development/mp3checker
#COPY mp3checker /root/Development/mp3checker/
#COPY sampleMusic /root/Downloads
RUN cd /root/Development
RUN ls -lh 
#RUN cd /root/Development/mp3checker/node_webUI; npm install
#The follwing redis line does not work
#RUN /usr/bin/redis-server "--daemonize yes"

#installing python requirements

RUN yum install -y epel-release
RUN yum install -y git
RUN yum install -y gcc
RUN yum install -y python
RUN yum install -y python-pip python-devel openssl-devel 
RUN yum install -y python-cffi
RUN yum install -y libffi-devel
RUN yum install -y mercurial

RUN pip install --upgrade pip
RUN pip install redis mutagen pyechonest jsonpickle
RUN pip install pyOpenSSL
RUN pip install pycrypto
RUN pip install -e git+https://git-wip-us.apache.org/repos/asf/libcloud.git@trun
k#egg=apache-libcloud
RUN pip install plumbery

[asimkhawaja]

@bernard357 I am going to change my pip installed plumber preare.cloud-init.sh file and then re-run plumbery deployed to confirm it is infact the pip install package that is suffering from the problem.

[bernard357]

Ok. FYI: on my Mac the prepare.cloud-init.sh file only has newlines, as expected on any Unix machine.

[asimkhawaja]

hmm so how is the build for pip happening? is that on a windows or linux machine?

[asimkhawaja]

Confirmed,having changed the line-endings on my pip install plumbery installation and re-running plumbery results in a VM where cloud-init is executing. So I would propose, that somwhere in the build pip package process we are introducing windows characters into that file

[bernard357]

@asimkhawaja this is great feedback indeed, since some funky process was used for previous generation of PIP release. Maybe you would like to use for plumbery a command similar to what you put for libcloud? pip install -e git+https://github.com/DimensionDataCBUSydney/plumbery.git#egg=plumbery

[asimkhawaja]

pip install through git does much better, Thanks Bernard

[bernard357]

thanks @asimkhawaja

[tonybaloney]

fixed in pypi https://pypi.python.org/pypi?name=plumbery&version=0.8.1&:action=display