cgcloud create -IT toil-box issue

[niko-balanis]

Been having trouble creating a toil-box, tried this with toil-box ,toil-latest-box and toil-legacy-box get the same error. However i had no problems creating a generic-ubuntu-trusty-box so it seems specific to the toil boxes. I apologize if this is something simple or this is the wrong place to post. Below is a readout of the full output following the cgcloud create command . Thank you for any help you can provide. I eagerly am looking forward to using cgcloud. Thanks again.

Fatal error: sudo() received nonzero return code 100 while executing!

cgcloud create -k Niko1 -IT toil-latest-box INFO: Using zone 'us-west-1b' and namespace '/Niko/' INFO: Looking up default image for role toil-latest-box and virtualization type hvm, ... INFO: ... found ami-ed8fa78d. INFO: Setting up security group ... INFO: ... finished setting up sg-214f8647. INFO: Creating t2.micro instance(s) ... INFO: ... got InvalidParameterValue, trying again in 5s ... INFO: ... got InvalidParameterValue, trying again in 5s ... INFO: ... created i-077c424c3dae7dcbc. INFO: Tagging instance ... INFO: ... instance tagged {'cluster_name': u'i-077c424c3dae7dcbc', 'ebs_volume_size': '0', 'leader_instance_id': u'i-077c424c3dae7dcbc', 'Name': u'Niko_toil-latest-box', 'enable_agent': 'True', 'generation': '0', 'cluster_ordinal': '0'}. INFO: ... waiting for instance i-077c424c3dae7dcbc ... INFO: ... running, waiting for assignment of public IP ... INFO: ... assigned, waiting for SSH port ... INFO: ... open ... INFO: ... testing SSH ... INFO: ... SSH working ..., INFO: ... instance ready. INFO: Waiting for cloud-init to finish ...... cloud-init done. [52.53.224.80] Executing task '__add_per_boot_script' [52.53.224.80] put: -> /var/lib/cloud/scripts/per-boot/cgcloud-done [52.53.224.80] Executing task '_setup_package_repos' [52.53.224.80] run: test -e /usr/lib/apt/methods/https [52.53.224.80] sudo: apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D [52.53.224.80] out: Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.lFwU2QNT3J --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D [52.53.224.80] out: gpg: requesting key 2C52609D from hkp server p80.pool.sks-keyservers.net [52.53.224.80] out: gpg: key 2C52609D: public key "Docker Release Tool (releasedocker) docker@docker.com" imported [52.53.224.80] out: gpg: Total number processed: 1 [52.53.224.80] out: gpg: imported: 1 (RSA: 1) [52.53.224.80] out:

[52.53.224.80] sudo: echo deb https://apt.dockerproject.org/repo ubuntu-trusty main > /etc/apt/sources.list.d/docker.list [52.53.224.80] sudo: apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF [52.53.224.80] out: Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.3prwefbA8y --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver keyserver.ubuntu.com --recv E56151BF [52.53.224.80] out: gpg: requesting key E56151BF from hkp server keyserver.ubuntu.com [52.53.224.80] out: gpg: key E56151BF: public key "Mesosphere Archive Automatic Signing Key support@mesosphere.io" imported [52.53.224.80] out: gpg: Total number processed: 1 [52.53.224.80] out: gpg: imported: 1 (RSA: 1) [52.53.224.80] out:

[52.53.224.80] sudo: echo "deb http://repos.mesosphere.io/ubuntu trusty main" > /etc/apt/sources.list.d/mesosphere.list [52.53.224.80] sudo: add-apt-repository -y ppa:fkrull/deadsnakes-python2.7 [52.53.224.80] out: gpg: keyring /tmp/tmpcsmb836c/secring.gpg' created [52.53.224.80] out: gpg: keyring/tmp/tmpcsmb836c/pubring.gpg' created [52.53.224.80] out: gpg: "tag:launchpad.net:2008:redacted" not a key ID: skipping [52.53.224.80] out:

[52.53.224.80] Executing task '_sync_package_repos' [52.53.224.80] put: -> /etc/apt/apt.conf.d/99timeout [52.53.224.80] sudo: DEBIAN_FRONTEND=readline apt-get -q -y update [52.53.224.80] out: Ign http://us-west-1.ec2.archive.ubuntu.com trusty InRelease [52.53.224.80] out: Get:1 http://us-west-1.ec2.archive.ubuntu.com trusty-updates InRelease [65.9 kB] [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty-backports InRelease [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty Release.gpg [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty Release [52.53.224.80] out: Get:2 http://us-west-1.ec2.archive.ubuntu.com trusty-updates/main Sources [402 kB] [52.53.224.80] out: Get:3 http://us-west-1.ec2.archive.ubuntu.com trusty-updates/restricted Sources [6,331 B] [52.53.224.80] out: Get:4 http://us-west-1.ec2.archive.ubuntu.com trusty-updates/universe Sources [187 kB] [52.53.224.80] out: Get:5 http://us-west-1.ec2.archive.ubuntu.com trusty-updates/multiverse Sources [7,769 B] [52.53.224.80] out: Get:6 http://us-west-1.ec2.archive.ubuntu.com trusty-updates/main amd64 Packages [1,004 kB] [52.53.224.80] out: Get:7 http://repos.mesosphere.io trusty InRelease [5,279 B] [52.53.224.80] out: Get:8 http://us-west-1.ec2.archive.ubuntu.com trusty-updates/restricted amd64 Packages [17.1 kB] [52.53.224.80] out: Get:9 http://us-west-1.ec2.archive.ubuntu.com trusty-updates/universe amd64 Packages [419 kB] [52.53.224.80] out: Get:10 http://repos.mesosphere.io trusty/main amd64 Packages [15.3 kB] [52.53.224.80] out: Get:11 http://us-west-1.ec2.archive.ubuntu.com trusty-updates/multiverse amd64 Packages [14.3 kB] [52.53.224.80] out: Get:12 http://us-west-1.ec2.archive.ubuntu.com trusty-updates/main Translation-en [493 kB] [52.53.224.80] out: Get:13 http://us-west-1.ec2.archive.ubuntu.com trusty-updates/multiverse Translation-en [7,430 B] [52.53.224.80] out: Get:14 http://us-west-1.ec2.archive.ubuntu.com trusty-updates/restricted Translation-en [3,978 B] [52.53.224.80] out: Get:15 http://us-west-1.ec2.archive.ubuntu.com trusty-updates/universe Translation-en [225 kB] [52.53.224.80] out: Get:16 http://us-west-1.ec2.archive.ubuntu.com trusty-backports/main Sources [9,712 B] [52.53.224.80] out: Get:17 http://ppa.launchpad.net trusty InRelease [15.5 kB] [52.53.224.80] out: Get:18 http://security.ubuntu.com trusty-security InRelease [65.9 kB] [52.53.224.80] out: Get:19 http://us-west-1.ec2.archive.ubuntu.com trusty-backports/restricted Sources [28 B] [52.53.224.80] out: Get:20 http://us-west-1.ec2.archive.ubuntu.com trusty-backports/universe Sources [35.3 kB] [52.53.224.80] out: Get:21 http://us-west-1.ec2.archive.ubuntu.com trusty-backports/multiverse Sources [1,898 B] [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty-backports/main amd64 Packages [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty-backports/restricted amd64 Packages [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty-backports/universe amd64 Packages [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty-backports/multiverse amd64 Packages [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty-backports/main Translation-en [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty-backports/multiverse Translation-en [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty-backports/restricted Translation-en [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty-backports/universe Translation-en [52.53.224.80] out: Get:22 http://us-west-1.ec2.archive.ubuntu.com trusty/main Sources [1,064 kB] [52.53.224.80] out: Get:23 http://us-west-1.ec2.archive.ubuntu.com trusty/restricted Sources [5,433 B] [52.53.224.80] out: Get:24 http://us-west-1.ec2.archive.ubuntu.com trusty/universe Sources [6,399 kB] [52.53.224.80] out: Ign http://ppa.launchpad.net trusty InRelease [52.53.224.80] out: Ign http://repos.mesosphere.io trusty/main Translation-en_US [52.53.224.80] out: Ign http://repos.mesosphere.io trusty/main Translation-en [52.53.224.80] out: Get:25 http://ppa.launchpad.net trusty/main amd64 Packages [3,002 B] [52.53.224.80] out: Get:26 https://apt.dockerproject.org ubuntu-trusty/main amd64 Packages [52.53.224.80] out: Get:27 https://apt.dockerproject.org ubuntu-trusty/main Translation-en_US [52.53.224.80] out: Ign https://apt.dockerproject.org ubuntu-trusty/main Translation-en_US [52.53.224.80] out: Ign https://apt.dockerproject.org ubuntu-trusty/main Translation-en [52.53.224.80] out: Get:28 http://us-west-1.ec2.archive.ubuntu.com trusty/multiverse Sources [174 kB] [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty/main amd64 Packages [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty/restricted amd64 Packages [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty/universe amd64 Packages [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty/multiverse amd64 Packages [52.53.224.80] out: Get:29 http://security.ubuntu.com trusty-security/main Sources [139 kB] [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty/main Translation-en [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty/multiverse Translation-en [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty/restricted Translation-en [52.53.224.80] out: Hit http://us-west-1.ec2.archive.ubuntu.com trusty/universe Translation-en [52.53.224.80] out: Ign http://us-west-1.ec2.archive.ubuntu.com trusty/main Translation-en_US [52.53.224.80] out: Ign http://us-west-1.ec2.archive.ubuntu.com trusty/multiverse Translation-en_US [52.53.224.80] out: Ign http://us-west-1.ec2.archive.ubuntu.com trusty/restricted Translation-en_US [52.53.224.80] out: Ign http://us-west-1.ec2.archive.ubuntu.com trusty/universe Translation-en_US [52.53.224.80] out: Get:30 http://ppa.launchpad.net trusty/main Translation-en [1,780 B] [52.53.224.80] out: Get:31 http://security.ubuntu.com trusty-security/universe Sources [61.0 kB] [52.53.224.80] out: Get:32 http://security.ubuntu.com trusty-security/main amd64 Packages [648 kB] [52.53.224.80] out: Ign http://ppa.launchpad.net trusty/main Translation-en_US [52.53.224.80] out: Get:33 http://security.ubuntu.com trusty-security/universe amd64 Packages [179 kB] [52.53.224.80] out: Get:34 http://security.ubuntu.com trusty-security/main Translation-en [347 kB] [52.53.224.80] out: Get:35 http://security.ubuntu.com trusty-security/universe Translation-en [103 kB] [52.53.224.80] out: Fetched 12.2 MB in 4s (2,498 kB/s) [52.53.224.80] out: Reading package lists... [52.53.224.80] out: W: GPG error: http://ppa.launchpad.net trusty InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 5BB92C09DB82666C [52.53.224.80] out:

[52.53.224.80] Executing task '__setup_application_user' [52.53.224.80] sudo: useradd --home /home/mesosbox --create-home --user-group --shell /bin/bash mesosbox [52.53.224.80] Executing task '_list_packages_to_install' [52.53.224.80] run: uname -r [52.53.224.80] out: 3.13.0-125-generic [52.53.224.80] out:

[52.53.224.80] run: uname -r [52.53.224.80] out: 3.13.0-125-generic [52.53.224.80] out:

[52.53.224.80] run: cat /sys/module/apparmor/parameters/enabled [52.53.224.80] out: Y [52.53.224.80] out:

[52.53.224.80] Executing task '_install_packages' [52.53.224.80] sudo: DEBIAN_FRONTEND=readline apt-get -q -y --no-install-recommends install linux-image-extra-3.13.0-125-generic linux-image-extra-virtual gcc python make docker-engine=1.9.1-0~trusty python2.7 libcurl4-openssl-dev libffi-dev python-pip htop apparmor mesos=0.25.1-* mdadm python-dev [52.53.224.80] out: Reading package lists... [52.53.224.80] out: Building dependency tree... [52.53.224.80] out: Reading state information... [52.53.224.80] out: python is already the newest version. [52.53.224.80] out: apparmor is already the newest version. [52.53.224.80] out: mdadm is already the newest version. [52.53.224.80] out: The following extra packages will be installed: [52.53.224.80] out: binutils comerr-dev cpp cpp-4.8 crda gcc-4.8 gcj-4.8-jre-headless [52.53.224.80] out: gcj-4.8-jre-lib gcj-jre-headless krb5-multidev libapr1 libaprutil1 libasan0 [52.53.224.80] out: libasound2 libasound2-data libatomic1 libc-dev-bin libc6-dev libcloog-isl4 [52.53.224.80] out: libexpat1-dev libgcc-4.8-dev libgcj-common libgcj14 libgcrypt11-dev libgmp10 [52.53.224.80] out: libgnutls-dev libgnutlsxx27 libgomp1 libgpg-error-dev libgssrpc4 [52.53.224.80] out: libidn11-dev libisl10 libitm1 libkadm5clnt-mit9 libkadm5srv-mit9 libkdb5-7 [52.53.224.80] out: libkrb5-dev libldap2-dev libmpc3 libmpfr4 libnl-3-200 libnl-genl-3-200 [52.53.224.80] out: libp11-kit-dev libpython-dev libpython2.7 libpython2.7-dev [52.53.224.80] out: libpython2.7-minimal libpython2.7-stdlib libquadmath0 librtmp-dev [52.53.224.80] out: libserf-1-1 libssl-dev libsvn1 libtasn1-6-dev libtsan0 linux-firmware [52.53.224.80] out: linux-image-3.13.0-126-generic linux-image-extra-3.13.0-126-generic [52.53.224.80] out: linux-image-generic linux-libc-dev pkg-config python-chardet-whl [52.53.224.80] out: python-colorama python-colorama-whl python-distlib python-distlib-whl [52.53.224.80] out: python-html5lib python-html5lib-whl python-pip-whl python-requests-whl [52.53.224.80] out: python-setuptools python-setuptools-whl python-six-whl python-urllib3-whl [52.53.224.80] out: python2.7-dev python2.7-minimal python3-pkg-resources wireless-regdb [52.53.224.80] out: zlib1g-dev [52.53.224.80] out: Suggested packages: [52.53.224.80] out: binutils-doc doc-base cpp-doc gcc-4.8-locales gcc-multilib manpages-dev [52.53.224.80] out: autoconf automake1.9 libtool flex bison gdb gcc-doc gcc-4.8-multilib [52.53.224.80] out: gcc-4.8-doc libgcc1-dbg libgomp1-dbg libitm1-dbg libatomic1-dbg libasan0-dbg [52.53.224.80] out: libtsan0-dbg libquadmath0-dbg fastjar gcj-4.8-jdk libgcj14-awt gcj-jdk [52.53.224.80] out: krb5-doc libasound2-plugins alsa-utils glibc-doc libcurl4-doc libcurl3-dbg [52.53.224.80] out: libgcj14-dbg libgcrypt11-doc gnutls26-doc gnutls-bin krb5-user fdutils [52.53.224.80] out: linux-doc-3.13.0 linux-source-3.13.0 linux-tools [52.53.224.80] out: linux-headers-3.13.0-126-generic make-doc python-genshi python-lxml [52.53.224.80] out: python2.7-doc binfmt-support python3-setuptools [52.53.224.80] out: Recommended packages: [52.53.224.80] out: iw aufs-tools cgroupfs-mount cgroup-lite git libssl-doc zookeeper zookeeperd [52.53.224.80] out: zookeeper-bin build-essential python-dev-all python-wheel [52.53.224.80] out: The following NEW packages will be installed: [52.53.224.80] out: binutils comerr-dev cpp cpp-4.8 crda docker-engine gcc gcc-4.8 [52.53.224.80] out: gcj-4.8-jre-headless gcj-4.8-jre-lib gcj-jre-headless htop krb5-multidev [52.53.224.80] out: libapr1 libaprutil1 libasan0 libasound2 libasound2-data libatomic1 [52.53.224.80] out: libc-dev-bin libc6-dev libcloog-isl4 libcurl4-openssl-dev libexpat1-dev [52.53.224.80] out: libffi-dev libgcc-4.8-dev libgcj-common libgcj14 libgcrypt11-dev libgmp10 [52.53.224.80] out: libgnutls-dev libgnutlsxx27 libgomp1 libgpg-error-dev libgssrpc4 [52.53.224.80] out: libidn11-dev libisl10 libitm1 libkadm5clnt-mit9 libkadm5srv-mit9 libkdb5-7 [52.53.224.80] out: libkrb5-dev libldap2-dev libmpc3 libmpfr4 libnl-3-200 libnl-genl-3-200 [52.53.224.80] out: libp11-kit-dev libpython-dev libpython2.7-dev libquadmath0 librtmp-dev [52.53.224.80] out: libserf-1-1 libssl-dev libsvn1 libtasn1-6-dev libtsan0 linux-firmware [52.53.224.80] out: linux-image-3.13.0-126-generic linux-image-extra-3.13.0-125-generic [52.53.224.80] out: linux-image-extra-3.13.0-126-generic linux-image-extra-virtual [52.53.224.80] out: linux-image-generic linux-libc-dev make mesos pkg-config python-chardet-whl [52.53.224.80] out: python-colorama python-colorama-whl python-dev python-distlib [52.53.224.80] out: python-distlib-whl python-html5lib python-html5lib-whl python-pip [52.53.224.80] out: python-pip-whl python-requests-whl python-setuptools python-setuptools-whl [52.53.224.80] out: python-six-whl python-urllib3-whl python2.7-dev python3-pkg-resources [52.53.224.80] out: wireless-regdb zlib1g-dev [52.53.224.80] out: The following packages will be upgraded: [52.53.224.80] out: libpython2.7 libpython2.7-minimal libpython2.7-stdlib python2.7 [52.53.224.80] out: python2.7-minimal [52.53.224.80] out: 5 upgraded, 86 newly installed, 0 to remove and 4 not upgraded. [52.53.224.80] out: Need to get 229 MB of archives. [52.53.224.80] out: After this operation, 789 MB of additional disk space will be used. [52.53.224.80] out: WARNING: The following packages cannot be authenticated! [52.53.224.80] out: python2.7 libpython2.7 libpython2.7-stdlib python2.7-minimal [52.53.224.80] out: libpython2.7-minimal libpython2.7-dev python2.7-dev [52.53.224.80] out: E: There are problems and -y was used without --force-yes [52.53.224.80] out:

Fatal error: sudo() received nonzero return code 100 while executing!

Requested: DEBIAN_FRONTEND=readline apt-get -q -y --no-install-recommends install linux-image-extra-3.13.0-125-generic linux-image-extra-virtual gcc python make docker-engine=1.9.1-0~trusty python2.7 libcurl4-openssl-dev libffi-dev python-pip htop apparmor mesos=0.25.1- mdadm python-dev Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "DEBIAN_FRONTEND=readline apt-get -q -y --no-install-recommends install linux-image-extra-3.13.0-125-generic linux-image-extra-virtual gcc python make docker-engine=1.9.1-0~trusty python2.7 libcurl4-openssl-dev libffi-dev python-pip htop apparmor mesos=0.25.1- mdadm python-dev"

Aborting. INFO: Terminating instance ...

[ejacox]

Hi @graeberlab, what are trying to do with the Toil box? We are in the process of moving away from using cgclould and we should be able to suggest an alternative.

[niko-balanis]

Hello Edwin, thanks for the reply, I am trying to use the toil-rnaseq pipeline. I built a semi-bare bones xeon box here to run the toil-rnaseq pipeline which works great, but I needed access to a computer with a little more computing power and more RAM for some very large files we have so I decided to try cgcloud/aws out. However I actually saw mention of cgcloud being deprecated after I posted my question so I kind of gave up trying to figure out whatever mistake I made. In the meantime I just decided to use AWS online instance spot creation tool , ssh into it and install everything manually for the toil pipeline that way. Im running the toil-rnaseq pipeline fine that way it seems, ill know for sure in a day or two if it doesnt error out. But what are the other alternatives to cgcloud? Its my first foray into cloud computing so I'd be very interested to know and pass on the knowledge to others in my lab.

Thank you,

Nikolas Balanis Graeber Lab, UCLA

[ejacox]

Try "toil launch-cluster ...", see http://toil.readthedocs.io/en/3.9.1/running/cloud.html.

We are updating the documentation and a newer version that might be easier to follow should be out next week.

[oskotsky]

add "--force -yes" to DEBIAN_FRONTEND in the file cgcloud/lib/python2.7/site-packages/cgcloud/core/ubuntu_box.py. So it looks like: DEBIAN_FRONTEND=readline apt-get -q -y --force-yes

[arkal]

oskotsky's answer is what i did to get cgcloud to work for me.