search

riptano / ComboAMI

The AMI takes a set of input parameters via the EC2 user-data to install, RAID, ring, and launch a DataStax Enterprise/Community cluster.
69 stars 59 forks source link

Exception seen in ds1_launcher.py #102

Closed tritonresearch closed 9 years ago

tritonresearch commented 9 years ago

Hi there,

I am trying to launch an EC2 Cassandra cluster from ami-f9a2b690, following the directions at http://docs.datastax.com/en/cassandra/2.2/cassandra/install/installAMILaunch.html.

I am running with the user data: --totalnodes 1 --version community --release 2.2 --clustername my-cluster, but the installation fails. Here is ami.log below. Any advice would be much appreciated.

Thanks!

[EXEC] 11/24/15-19:26:30 git pull: Updating 5f722d6..575cea6 Fast-forward README.md | 32 +-- SWITCHES.md | 7 +- ami_ids.json | 22 ++ community_cloud/README.md | 5 + community_cloud/cloudformation/README.md | 35 +++ .../cloudformation/cassandra-vpc.template | 103 ++++++++ ds0_updater.py | 9 +- ds1_launcher.py | 22 +- ds2_configure.py | 266 ++++++++++++++++---- ds4_motd.py | 8 +- logger.py | 2 +- presetup/setup.md | 4 +- repo_keys/DataStax.key | 32 +-- wait_for_first_node.sh | 9 + 14 files changed, 446 insertions(+), 110 deletions(-) create mode 100644 ami_ids.json create mode 100644 community_cloud/README.md create mode 100644 community_cloud/cloudformation/README.md create mode 100644 community_cloud/cloudformation/cassandra-vpc.template create mode 100755 wait_for_first_node.sh

[EXEC] 11/24/15-19:26:30 git reset --hard: HEAD is now at 575cea6 Merge branch 'dev-2.5' into 2.5

[EXEC:E] 11/24/15-19:26:30 gpg --import /home/ubuntu/datastax_ami/repo_keys/DataStax_AMI.7123CDFD.key: gpg: directory /root/.gnupg' created gpg: new configuration file/root/.gnupg/gpg.conf' created gpg: WARNING: options in /root/.gnupg/gpg.conf' are not yet active during this run gpg: keyring/root/.gnupg/secring.gpg' created gpg: keyring `/root/.gnupg/pubring.gpg' created gpg: /root/.gnupg/trustdb.gpg: trustdb created gpg: key 7123CDFD: public key "Joaquin Casares (DataStax AMI) joaquin@datastax.com" imported gpg: Total number processed: 1 gpg: imported: 1 (RSA: 1)

[EXEC] 11/24/15-19:26:30 git log --pretty="format:%G?" --show-signature HEAD^..HEAD: gpg: Signature made Fri Aug 14 16:08:55 2015 UTC using RSA key ID 7123CDFD gpg: Good signature from "Joaquin Casares (DataStax AMI) joaquin@datastax.com" gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 6273 EED3 28B4 B09B 2087 1C0E E711 FCAF 7123 CDFD G gpg: Signature made Fri Aug 14 15:55:19 2015 UTC using RSA key ID 7123CDFD gpg: Good signature from "Joaquin Casares (DataStax AMI) joaquin@datastax.com" gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 6273 EED3 28B4 B09B 2087 1C0E E711 FCAF 7123 CDFD G [EXEC] 11/24/15-19:26:31 sudo rm ds2_configure.py [INFO] Deleting ds2_configure.py now. This AMI will never change any configs after this first run. [EXEC] 11/24/15-19:26:31 sudo apt-key add /home/ubuntu/datastax_ami/repo_keys/Launchpad_VLC.C2518248EEA14886.key: OK

[EXEC] 11/24/15-19:26:31 sudo apt-key add /home/ubuntu/datastax_ami/repo_keys/Ubuntu_Archive.40976EAF437D05B5.key: OK

[EXEC] 11/24/15-19:26:31 sudo rm -rf /etc/motd [EXEC] 11/24/15-19:26:31 sudo touch /etc/motd [INFO] Started with user data set to: [INFO] --totalnodes 1 --version community --release 2.2 --clustername tron-east [INFO] Using user data: [INFO] --totalnodes 1 --version community --release 2.2 --clustername tron-east [INFO] Using instance type: m3.medium [INFO] meta-data:instance-type: m3.medium [INFO] meta-data:local-ipv4: 172.31.43.224 [INFO] meta-data:public-hostname: ec2-54-86-171-19.compute-1.amazonaws.com [INFO] meta-data:ami-launch-index: 0 [INFO] meta-data:reservation-id: r-146c0bbd [EXEC] 11/24/15-19:26:31 sudo ls: ami.conf ami_ids.json AMI_LICENSE.txt ami.log CHANGES.md community_cloud conf.py conf.pyc ds0_updater.py ds0_utils.py ds0_utils.pyc ds1_launcher.py ds1_launcher.pyc ds2_configure.pyc ds3_after_init.py ds4_motd.py EXPANDING.md FILES.md logger.py logger.pyc presetup README.md reflector.php repo_keys SWITCHES.md tokentool.html tokentoolv2.py wait_for_first_node.sh

[INFO] Using cluster name: tron-east [INFO] Using cluster size: 1 [INFO] Using seed indexes: [0, 1, 1] [PIPE] 11/24/15-19:26:31 echo "deb http://debian.datastax.com/community stable main" | sudo tee /etc/apt/sources.list.d/datastax.sources.list: deb http://debian.datastax.com/community stable main

[PIPE] 11/24/15-19:26:31 echo "deb http://debian.datastax.com/community stable main" | sudo tee /etc/apt/sources.list.d/datastax.sources.list: deb http://debian.datastax.com/community stable main

[EXEC] 11/24/15-19:26:31 sudo apt-key add /home/ubuntu/datastax_ami/repo_keys/DataStax.key: OK

[EXEC] 11/24/15-19:27:15 sudo apt-get update: Hit http://us-east-1.ec2.archive.ubuntu.com precise Release.gpg Get:1 http://us-east-1.ec2.archive.ubuntu.com precise-updates Release.gpg [198 B] Get:2 http://debian.datastax.com stable Release.gpg [280 B] Get:3 http://ppa.launchpad.net precise Release.gpg [316 B] Get:4 http://security.ubuntu.com precise-security Release.gpg [198 B] Hit http://us-east-1.ec2.archive.ubuntu.com precise Release Get:5 http://security.ubuntu.com precise-security Release [54.3 kB] Get:6 http://debian.datastax.com stable Release [7,168 B] Get:7 http://ppa.launchpad.net precise Release [13.0 kB] Get:8 http://us-east-1.ec2.archive.ubuntu.com precise-updates Release [196 kB] Hit http://us-east-1.ec2.archive.ubuntu.com precise/main Sources Hit http://us-east-1.ec2.archive.ubuntu.com precise/universe Sources Hit http://us-east-1.ec2.archive.ubuntu.com precise/main amd64 Packages Hit http://us-east-1.ec2.archive.ubuntu.com precise/universe amd64 Packages Hit http://us-east-1.ec2.archive.ubuntu.com precise/main i386 Packages Hit http://us-east-1.ec2.archive.ubuntu.com precise/universe i386 Packages Hit http://us-east-1.ec2.archive.ubuntu.com precise/main TranslationIndex Get:9 http://security.ubuntu.com precise-security/main Sources [136 kB] Hit http://us-east-1.ec2.archive.ubuntu.com precise/universe TranslationIndex Hit http://us-east-1.ec2.archive.ubuntu.com precise/main Translation-en Hit http://us-east-1.ec2.archive.ubuntu.com precise/universe Translation-en Get:10 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main Sources [494 kB] Get:11 http://debian.datastax.com stable/main amd64 Packages [30.9 kB] Get:12 http://ppa.launchpad.net precise/main Sources [1,592 B] Get:13 http://security.ubuntu.com precise-security/universe Sources [43.8 kB] Get:14 http://security.ubuntu.com precise-security/main amd64 Packages [563 kB] Get:15 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe Sources [122 kB] Get:16 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main amd64 Packages [956 kB] Get:17 http://ppa.launchpad.net precise/main amd64 Packages [3,368 B] Get:18 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe amd64 Packages [270 kB] Get:19 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main i386 Packages [1,010 kB] Get:20 http://debian.datastax.com stable/main i386 Packages [30.9 kB] Get:21 http://ppa.launchpad.net precise/main i386 Packages [3,368 B] Get:22 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe i386 Packages [279 kB] Get:23 http://security.ubuntu.com precise-security/universe amd64 Packages [125 kB] Get:24 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main TranslationIndex [10.6 kB] Get:25 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe TranslationIndex [8,333 B] Get:26 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main Translation-en [408 kB] Get:27 http://security.ubuntu.com precise-security/main i386 Packages [621 kB] Get:28 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe Translation-en [162 kB] Get:29 http://ppa.launchpad.net precise/main TranslationIndex [199 B] Ign http://debian.datastax.com stable/main TranslationIndex Get:30 http://security.ubuntu.com precise-security/universe i386 Packages [133 kB] Get:31 http://security.ubuntu.com precise-security/main TranslationIndex [208 B] Get:32 http://ppa.launchpad.net precise/main Translation-en [1,556 B] Get:33 http://security.ubuntu.com precise-security/universe TranslationIndex [205 B] Get:34 http://security.ubuntu.com precise-security/main Translation-en [240 kB] Get:35 http://security.ubuntu.com precise-security/universe Translation-en [79.9 kB] Ign http://debian.datastax.com stable/main Translation-en_US Ign http://debian.datastax.com stable/main Translation-en Fetched 6,005 kB in 9s (614 kB/s) Reading package lists...

[INFO] Update loop... [EXEC] 11/24/15-19:27:22 sudo apt-get update: Hit http://us-east-1.ec2.archive.ubuntu.com precise Release.gpg Hit http://us-east-1.ec2.archive.ubuntu.com precise-updates Release.gpg Hit http://us-east-1.ec2.archive.ubuntu.com precise Release Hit http://us-east-1.ec2.archive.ubuntu.com precise-updates Release Hit http://us-east-1.ec2.archive.ubuntu.com precise/main Sources Hit http://us-east-1.ec2.archive.ubuntu.com precise/universe Sources Hit http://us-east-1.ec2.archive.ubuntu.com precise/main amd64 Packages Hit http://us-east-1.ec2.archive.ubuntu.com precise/universe amd64 Packages Hit http://us-east-1.ec2.archive.ubuntu.com precise/main i386 Packages Hit http://us-east-1.ec2.archive.ubuntu.com precise/universe i386 Packages Hit http://us-east-1.ec2.archive.ubuntu.com precise/main TranslationIndex Hit http://us-east-1.ec2.archive.ubuntu.com precise/universe TranslationIndex Hit http://us-east-1.ec2.archive.ubuntu.com precise-updates/main Sources Hit http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe Sources Hit http://us-east-1.ec2.archive.ubuntu.com precise-updates/main amd64 Packages Hit http://security.ubuntu.com precise-security Release.gpg Hit http://debian.datastax.com stable Release.gpg Hit http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe amd64 Packages Hit http://ppa.launchpad.net precise Release.gpg Hit http://us-east-1.ec2.archive.ubuntu.com precise-updates/main i386 Packages Hit http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe i386 Packages Hit http://us-east-1.ec2.archive.ubuntu.com precise-updates/main TranslationIndex Hit http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe TranslationIndex Hit http://us-east-1.ec2.archive.ubuntu.com precise/main Translation-en Hit http://us-east-1.ec2.archive.ubuntu.com precise/universe Translation-en Hit http://us-east-1.ec2.archive.ubuntu.com precise-updates/main Translation-en Hit http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe Translation-en Hit http://security.ubuntu.com precise-security Release Hit http://debian.datastax.com stable Release Hit http://ppa.launchpad.net precise Release Hit http://security.ubuntu.com precise-security/main Sources Hit http://debian.datastax.com stable/main amd64 Packages Hit http://ppa.launchpad.net precise/main Sources Hit http://security.ubuntu.com precise-security/universe Sources Hit http://debian.datastax.com stable/main i386 Packages Hit http://ppa.launchpad.net precise/main amd64 Packages Hit http://security.ubuntu.com precise-security/main amd64 Packages Hit http://ppa.launchpad.net precise/main i386 Packages Ign http://debian.datastax.com stable/main TranslationIndex Hit http://security.ubuntu.com precise-security/universe amd64 Packages Hit http://ppa.launchpad.net precise/main TranslationIndex Hit http://security.ubuntu.com precise-security/main i386 Packages Hit http://ppa.launchpad.net precise/main Translation-en Hit http://security.ubuntu.com precise-security/universe i386 Packages Hit http://security.ubuntu.com precise-security/main TranslationIndex Hit http://security.ubuntu.com precise-security/universe TranslationIndex Hit http://security.ubuntu.com precise-security/main Translation-en Hit http://security.ubuntu.com precise-security/universe Translation-en Ign http://debian.datastax.com stable/main Translation-en_US Ign http://debian.datastax.com stable/main Translation-en Reading package lists...

[INFO] Performing deployment install... [EXEC] 11/24/15-19:27:27 sudo mv /etc/security/limits.d/cassandra.conf /etc/security/limits.d/cassandra.conf.bak [EXEC] 11/24/15-19:27:28 sudo apt-get install -y python-cql datastax-agent cassandra=2.2 dsc22=2.2-1: Reading package lists... Building dependency tree... Reading state information...

[ERROR] 11/24/15-19:27:28 sudo service cassandra stop: cassandra: unrecognized service

[EXEC] 11/24/15-19:27:28 sudo rm -rf /var/lib/cassandra [EXEC] 11/24/15-19:27:28 sudo rm -rf /var/log/cassandra [EXEC] 11/24/15-19:27:28 sudo mkdir -p /var/lib/cassandra [EXEC] 11/24/15-19:27:28 sudo mkdir -p /var/log/cassandra [ERROR] 11/24/15-19:27:28 sudo chown -R cassandra:cassandra /var/lib/cassandra: chown: invalid user: `cassandra:cassandra'

[ERROR] 11/24/15-19:27:28 sudo chown -R cassandra:cassandra /var/log/cassandra: chown: invalid user: `cassandra:cassandra'

[EXEC] 11/24/15-19:27:28 sudo mv /etc/security/limits.d/cassandra.conf.bak /etc/security/limits.d/cassandra.conf [INFO] Installing OpsCenter... [EXEC] 11/24/15-19:28:29 sudo apt-get install -y opscenter libssl0.9.8: Reading package lists... Building dependency tree... Reading state information... The following NEW packages will be installed: libssl0.9.8 opscenter 0 upgraded, 2 newly installed, 0 to remove and 147 not upgraded. Need to get 78.3 MB of archives. After this operation, 104 MB of additional disk space will be used. Get:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise-updates/universe libssl0.9.8 amd64 0.9.8o-7ubuntu3.2 [868 kB] Get:2 http://debian.datastax.com/community/ stable/main opscenter all 5.2.2 [77.4 MB] Fetched 78.3 MB in 14s (5,266 kB/s) Selecting previously unselected package libssl0.9.8. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 87758 files and directories currently installed.) Unpacking libssl0.9.8 (from .../libssl0.9.8_0.9.8o-7ubuntu3.2_amd64.deb) ... Selecting previously unselected package opscenter. Unpacking opscenter (from .../opscenter_5.2.2_all.deb) ... Processing triggers for ureadahead ... Setting up libssl0.9.8 (0.9.8o-7ubuntu3.2) ... Setting up opscenter (5.2.2) ... Generating a 1024 bit RSA private key ........................++++++ .......................++++++

writing new private key to '/var/lib/opscenter/ssl/opscenter.key'

MAC verified OK Certificate was added to keystore Warning: Overwriting existing alias agent_key in destination keystore [Storing /var/lib/opscenter/ssl/agentKeyStore.p12] MAC verified OK Processing triggers for libc-bin ... ldconfig deferred processing now taking place

[EXEC] 11/24/15-19:28:29 sudo service opscenterd stop [INFO] Reflector loop... [INFO] 11/24/15-19:28:29 Reflector: Received 1 of 1 responses from: [u'172.31.43.224'] [INFO] Seed list: set([u'172.31.43.224']) [INFO] OpsCenter: 172.31.43.224 [INFO] Options: Namespace(analyticsnodes=0, base64postscript=None, bootstrap=False, cfsreplication=None, clustername='tron-east', customreservation=None, email=None, hadoop=False, heapsize=None, multiregion=False, opscenter=None, opscenterinterface=None, opscenterip=None, opscenteronly=False, opscenterssl=False, password=None, raidonly=False, realtimenodes=1, reflector=None, release='2.2', rpcbinding=False, searchnodes=0, seed_indexes=[0, 1, 1], seeds=None, totalnodes=1, username=None, version='community', vnodes=False) [ERROR] Exception seen in ds1_launcher.py: Traceback (most recent call last): File "/home/ubuntu/datastax_ami/ds1_launcher.py", line 22, in initial_configurations ds2_configure.run() File "/home/ubuntu/datastax_ami/ds2_configure.py", line 1176, in run File "/home/ubuntu/datastax_ami/ds2_configure.py", line 577, in construct_yaml IOError: [Errno 2] No such file or directory: '/etc/cassandra/cassandra.yaml'

tritonresearch commented 9 years ago

I seem to have fixed this by adding the proper release: --release 2.2.3. However, the error handling was extremely confusing in this case.

mlococo commented 8 years ago

Yeah, the error handling for many edge-cases is not what I would wish. Thanks for the report and the workaround.