search

riptano / ComboAMI

The AMI takes a set of input parameters via the EC2 user-data to install, RAID, ring, and launch a DataStax Enterprise/Community cluster.
69 stars 59 forks source link

Exception in ds1_launcher.py #7

Closed TokMedia closed 12 years ago

TokMedia commented 12 years ago

I launched 3 node cassandra ec2 cluster using ami-6139e708 with user data: --clustername TokCluster --totalnodes 3 --version community --release 1.0

When launched had the error: Exception seen in ds1_launcher.py. Please check ~/datastax_ami/ami.log for more info.

This is the content of ami.log

[EXEC] 11/07/12-11:16:00 git pull: Updating 38b1733..0b637b9 Fast-forward ds1_launcher.py | 9 ++- ds2_configure.py | 218 +++++++++++++++++++++++++++++++++++++++------------- ds4_motd.py | 11 +-- presetup/pre_1.sh | 4 +- presetup/pre_2.py | 4 +- tokentoolv2.py | 4 +- 6 files changed, 180 insertions(+), 70 deletions(-)

[EXEC] 11/07/12-11:16:01 sudo rm ds2_configure.py [INFO] Deleting ds2_configure.py now. This AMI will never change any configs after this first run. [EXEC] 11/07/12-11:16:01 gpg --keyserver pgp.mit.edu --recv-keys 40976EAF437D05B5: gpgkeys: key 40976EAF437D05B5 not found on keyserver

[PIPE] 11/07/12-11:16:01 gpg --export --armor 40976EAF437D05B5 | sudo apt-key add - [EXEC] 11/07/12-11:16:01 sudo rm -rf /etc/motd [EXEC] 11/07/12-11:16:01 sudo touch /etc/motd [INFO] Started with user data set to: [INFO] --clustername TokCluster --totalnodes 3 --version community --release 1.0 [INFO] Using cluster name: TokCluster [INFO] Using cluster size: 3 [INFO] Using seed indexes: [0, 3, 3] [PIPE] 11/07/12-11:16:01 echo "deb http://debian.datastax.com/community stable main" | sudo tee -a /etc/apt/sources.list.d/datastax.sources.list: deb http://debian.datastax.com/community stable main

[PIPE] 11/07/12-11:16:01 echo "deb http://debian.datastax.com/community stable main" | sudo tee -a /etc/apt/sources.list.d/datastax.sources.list: deb http://debian.datastax.com/community stable main

[PIPE] 11/07/12-11:16:02 curl -s http://installer.datastax.com/downloads/ubuntuarchive.repo_key | sudo apt-key add -: OK

[PIPE] 11/07/12-11:16:02 curl -s http://installer.datastax.com/downloads/ubuntuarchive.repo_key | sudo apt-key add -: OK

[PIPE] 11/07/12-11:16:02 curl -s http://opscenter.datastax.com/debian/repo_key | sudo apt-key add -: OK

[PIPE] 11/07/12-11:16:02 curl -s http://opscenter.datastax.com/debian/repo_key | sudo apt-key add -: OK

[PIPE] 11/07/12-11:16:02 curl -s http://debian.datastax.com/debian/repo_key | sudo apt-key add -: OK

[PIPE] 11/07/12-11:16:02 curl -s http://debian.datastax.com/debian/repo_key | sudo apt-key add -: OK

[EXEC] 11/07/12-11:16:12 sudo apt-get update: Get:1 http://debian.datastax.com stable Release.gpg [490B] Get:2 http://us-east-1.ec2.archive.ubuntu.com maverick Release.gpg [198B] Ign http://debian.datastax.com/community/ stable/main Translation-en Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ maverick/main Translation-en Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ maverick/universe Translation-en Get:3 http://debian.datastax.com stable Release [2859B] Get:4 http://us-east-1.ec2.archive.ubuntu.com maverick-updates Release.gpg [198B] Get:5 http://security.ubuntu.com maverick-security Release.gpg [198B] Ign http://security.ubuntu.com/ubuntu/ maverick-security/main Translation-en Ign http://security.ubuntu.com/ubuntu/ maverick-security/universe Translation-en Get:6 http://archive.canonical.com lucid Release.gpg [198B] Ign http://archive.canonical.com/ lucid/partner Translation-en Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ maverick-updates/main Translation-en Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ maverick-updates/universe Translation-en Get:7 http://debian.datastax.com stable/main amd64 Packages [5522B] Get:8 http://us-east-1.ec2.archive.ubuntu.com maverick Release [39.8kB] Get:9 http://security.ubuntu.com maverick-security Release [39.8kB] Get:10 http://archive.canonical.com lucid Release [8215B] Get:11 http://us-east-1.ec2.archive.ubuntu.com maverick-updates Release [39.8kB] Get:12 http://us-east-1.ec2.archive.ubuntu.com maverick/main Sources [829kB] Get:13 http://archive.canonical.com lucid/partner amd64 Packages [13.0kB] Get:14 http://security.ubuntu.com maverick-security/main Sources [106kB] Get:15 http://us-east-1.ec2.archive.ubuntu.com maverick/universe Sources [4179kB] Get:16 http://security.ubuntu.com maverick-security/universe Sources [34.2kB] Get:17 http://security.ubuntu.com maverick-security/main amd64 Packages [330kB] Get:18 http://us-east-1.ec2.archive.ubuntu.com maverick/main amd64 Packages [1491kB] Get:19 http://us-east-1.ec2.archive.ubuntu.com maverick/universe amd64 Packages [5771kB] Get:20 http://security.ubuntu.com maverick-security/universe amd64 Packages [116kB] Get:21 http://us-east-1.ec2.archive.ubuntu.com maverick-updates/main Sources [179kB] Get:22 http://us-east-1.ec2.archive.ubuntu.com maverick-updates/universe Sources [65.2kB] Get:23 http://us-east-1.ec2.archive.ubuntu.com maverick-updates/main amd64 Packages [466kB] Get:24 http://us-east-1.ec2.archive.ubuntu.com maverick-updates/universe amd64 Packages [202kB] Fetched 13.9MB in 8s (1667kB/s) Reading package lists...

[EXEC] 11/07/12-11:16:14 sudo apt-get update: Hit http://us-east-1.ec2.archive.ubuntu.com maverick Release.gpg Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ maverick/main Translation-en Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ maverick/universe Translation-en Get:1 http://debian.datastax.com stable Release.gpg [490B] Hit http://us-east-1.ec2.archive.ubuntu.com maverick-updates Release.gpg Ign http://debian.datastax.com/community/ stable/main Translation-en Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ maverick-updates/main Translation-en Get:2 http://debian.datastax.com stable Release [2859B] Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ maverick-updates/universe Translation-en Hit http://us-east-1.ec2.archive.ubuntu.com maverick Release Hit http://security.ubuntu.com maverick-security Release.gpg Ign http://security.ubuntu.com/ubuntu/ maverick-security/main Translation-en Ign http://security.ubuntu.com/ubuntu/ maverick-security/universe Translation-en Get:3 http://debian.datastax.com stable/main amd64 Packages [5522B] Hit http://archive.canonical.com lucid Release.gpg Ign http://archive.canonical.com/ lucid/partner Translation-en Hit http://us-east-1.ec2.archive.ubuntu.com maverick-updates Release Hit http://us-east-1.ec2.archive.ubuntu.com maverick/main Sources Hit http://us-east-1.ec2.archive.ubuntu.com maverick/universe Sources Hit http://security.ubuntu.com maverick-security Release Hit http://archive.canonical.com lucid Release Hit http://us-east-1.ec2.archive.ubuntu.com maverick/main amd64 Packages Hit http://us-east-1.ec2.archive.ubuntu.com maverick/universe amd64 Packages Hit http://us-east-1.ec2.archive.ubuntu.com maverick-updates/main Sources Hit http://security.ubuntu.com maverick-security/main Sources Hit http://us-east-1.ec2.archive.ubuntu.com maverick-updates/universe Sources Hit http://us-east-1.ec2.archive.ubuntu.com maverick-updates/main amd64 Packages Hit http://us-east-1.ec2.archive.ubuntu.com maverick-updates/universe amd64 Packages Hit http://archive.canonical.com lucid/partner amd64 Packages Hit http://security.ubuntu.com maverick-security/universe Sources Hit http://security.ubuntu.com maverick-security/main amd64 Packages Hit http://security.ubuntu.com maverick-security/universe amd64 Packages Fetched 8871B in 0s (22.2kB/s) Reading package lists...

[INFO] Performing deployment install... [EXEC] 11/07/12-11:16:20 sudo apt-get install -y cassandra=1.0 dsc: Reading package lists... Building dependency tree... Reading state information...

[ERROR] 11/07/12-11:16:21 sudo service cassandra stop: cassandra: unrecognized service

[EXEC] 11/07/12-11:16:21 sudo rm -rf /var/lib/cassandra [EXEC] 11/07/12-11:16:21 sudo rm -rf /var/log/cassandra [EXEC] 11/07/12-11:16:21 sudo mkdir -p /var/lib/cassandra [EXEC] 11/07/12-11:16:21 sudo mkdir -p /var/log/cassandra [ERROR] 11/07/12-11:16:21 sudo chown -R cassandra:cassandra /var/lib/cassandra: chown: invalid user: `cassandra:cassandra'

[ERROR] 11/07/12-11:16:21 sudo chown -R cassandra:cassandra /var/log/cassandra: chown: invalid user: `cassandra:cassandra'

[INFO] Installing OpsCenter... [EXEC] 11/07/12-11:16:41 sudo apt-get -y install opscenter-free: Reading package lists... Building dependency tree... Reading state information... The following NEW packages will be installed: opscenter-free 0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded. Need to get 47.7MB of archives. After this operation, 70.5MB of additional disk space will be used. Get:1 http://debian.datastax.com/community/ stable/main opscenter-free all 2.1.2-1 [47.7MB] Fetched 47.7MB in 9s (5117kB/s) Selecting previously deselected package opscenter-free. (Reading database ... 36947 files and directories currently installed.) Unpacking opscenter-free (from .../opscenter-free_2.1.2-1_all.deb) ... Processing triggers for ureadahead ... Setting up opscenter-free (2.1.2-1) ... Generating a 1024 bit RSA private key ........++++++ .................++++++

writing new private key to '/var/lib/opscenter/ssl/opscenter.key'

MAC verified OK Certificate was added to keystore

[EXEC] 11/07/12-11:16:41 sudo service opscenterd stop [INFO] Reflector loop... [INFO] Seed list: set([u'10.83.46.56']) [INFO] OpsCenter: 10.83.46.56 [INFO] Options: {'username': None, 'cfsreplication': None, 'reflector': None, 'clustername': 'TokCluster', 'analyticsnodes': 0, 'seed_indexes': [0, 3, 3], 'realtimenodes': 3, 'opscenter': None, 'totalnodes': 3, 'searchnodes': 0, 'opscenterinterface': None, 'version': 'community', 'dev': None, 'release': '1.0', 'password': None, 'email': None, 'raidonly': None, 'heapsize': None} [ERROR] Exception seen in ds1_launcher.py: Traceback (most recent call last): File "/home/ubuntu/datastax_ami/ds1_launcher.py", line 20, in initial_configurations ds2_configure.run() File "/home/ubuntu/datastax_ami/ds2_configure.py", line 799, in run File "/home/ubuntu/datastax_ami/ds2_configure.py", line 389, in construct_yaml IOError: [Errno 2] No such file or directory: '/etc/cassandra/cassandra.yaml'

TokMedia commented 12 years ago

Solved, it, should have been --release 1.0.11

joaquincasares commented 12 years ago

Great to hear this was resolved!

TokMedia commented 12 years ago

I am having another issue with the AMI which may be related,

I installed new cluster using DataStax AMI with --release 1.0.11 The instances have: python-cql 1.0.10-1 python-thrift 0.6.0-2~riptano1 cassandra 1.0.11

cqlsh fails with errors, both when I try version 2 and version 3: cqlsh --debug --cqlversion=2 localhost 9160 Using CQL driver: <module 'cql' from '/usr/lib/pymodules/python2.6/cql/init.pyc'> Using thrift lib: <module 'thrift' from '/usr/lib/pymodules/python2.6/thrift/init.pyc'> Connection error: Invalid method name: 'set_cql_version'

cqlsh --debug --cql3 localhost 9160 Using CQL driver: <module 'cql' from '/usr/lib/pymodules/python2.6/cql/init.pyc'> Using thrift lib: <module 'thrift' from '/usr/lib/pymodules/python2.6/thrift/init.pyc'> Connection error: Invalid method name: 'set_cql_version'

Any idea?

Thanks,

Tamar Fraenkel Senior Software Engineer, TOK Media

[image: Inline image 1]

tamar@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956

On Wed, Nov 7, 2012 at 8:29 PM, Joaquin Casares notifications@github.comwrote:

Great to hear this was resolved!

— Reply to this email directly or view it on GitHubhttps://github.com/riptano/ComboAMI/issues/7#issuecomment-10159326.

joaquincasares commented 12 years ago

Does the issue appear when the --release tag isn't used?

If that's the case, this may be more of a packaging bug, which I can report as well.

The AMI is more of a platform for installing the packages.

TokMedia commented 12 years ago

Hi! I guess that if I use the AMI without the --release tag, and I get Cassandra 1.1.5 the bug will disappear as it will have set_cql_version, but haven't tested it.

Can you report the bug? In the meantime, do you have a way for me to upgrade or downgrade one of the components so that they all be compatible?

Thanks,

Tamar Fraenkel Senior Software Engineer, TOK Media

[image: Inline image 1]

tamar@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956

On Thu, Nov 8, 2012 at 10:07 PM, Joaquin Casares notifications@github.comwrote:

Does the issue appear when the --release tag isn't used?

If that's the case, this may be more of a packaging bug, which I can report as well.

The AMI is more of a platform for installing the packages.

— Reply to this email directly or view it on GitHubhttps://github.com/riptano/ComboAMI/issues/7#issuecomment-10202756.

joaquincasares commented 11 years ago

Hello Tamar,

Sorry for the late response. This message got lost within my inbox.

If you're still seeing issues with the community packages, feel free to report a bug here: https://issues.apache.org/jira/browse/CASSANDRA

As far as keeping all components compatible while using the AMI, I strongly suggest launching new AMI's with the appropriate settings. This will ensure stronger reproducibility.

Thanks, Joaquin

TokMedia commented 11 years ago

Thanks for the response. If I do accept your offer, and lunch a new AMI (assuming I want to still be on 1.0.x, and I don't want to loose data), how would I do that? I have 3 node ring with replication factor of 2 and I use WCL=RCL=Quorum. So would you suggest adding 3 new nodes, and after the data spread around removing the old ones one by one? Thanks, Tamar

Tamar Fraenkel Senior Software Engineer, TOK Media

[image: Inline image 1]

tamar@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956

On Tue, Mar 5, 2013 at 2:19 AM, Joaquin Casares notifications@github.comwrote:

Hello Tamar,

Sorry for the late response. This message got lost within my inbox.

If you're still seeing issues with the community packages, feel free to report a bug here: https://issues.apache.org/jira/browse/CASSANDRA

As far as keeping all components compatible while using the AMI, I strongly suggest launching new AMI's with the appropriate settings. This will ensure stronger reproducibility.

Thanks, Joaquin

— Reply to this email directly or view it on GitHubhttps://github.com/riptano/ComboAMI/issues/7#issuecomment-14415987 .

joaquincasares commented 11 years ago

Hello Tamar,

Yes, that's correct. When launching the new AMIs just ensure release is set properly.

If you hit any operational issues, feel free to work through them here: http://www.datastax.com/support-forums/

Thanks, Joaquin

sumitkarn commented 9 years ago

Use latest http://thecloudmarket.com/image/ami-59ff4332--datastax-auto-clustering-ami-test-2-6-3-1404-hvm