search

aws / aws-parallelcluster

AWS ParallelCluster is an AWS supported Open Source cluster management tool to deploy and manage HPC clusters in the AWS cloud.
https://github.com/aws/aws-parallelcluster
Apache License 2.0
833 stars 312 forks source link

pcluster createami command fails #1301

Closed ssbotelh closed 5 years ago

ssbotelh commented 5 years ago

Environment:

I'm trying to create a custom ParallelCluster AMI off of an existing AMI following instructions from here: https://aws-parallelcluster.readthedocs.io/en/latest/tutorials/02_ami_customization.html

The exact command is: pcluster createami --ami-id ami-0345b9024daa8254b --os ubuntu1604

After some time, pcluster exits with code 1, and no AMI is created. The log has a bit more info, but nothing too descriptive to me. The last few lines are:

custom-ubuntu1604: WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
custom-ubuntu1604: ---- End output of "bash"  "/tmp/chef-script20190907-2822-1odyih" ----
custom-ubuntu1604: Ran "bash"  "/tmp/chef-script20190907-2822-1odyih" returned 1
==> custom-ubuntu1604: Terminating the source AWS instance...
==> custom-ubuntu1604: Cleaning up any extra volumes...
==> custom-ubuntu1604: No volumes to clean up, skipping
==> custom-ubuntu1604: Deleting temporary security group...
==> custom-ubuntu1604: Deleting temporary keypair...
Build 'custom-ubuntu1604' errored: Error executing Chef: Non-zero exit status: 1

==> Some builds didn't complete successfully and had errors:
--> custom-ubuntu1604: Error executing Chef: Non-zero exit status: 1

==> Builds finished but no artifacts were created.

Any ideas? Please let me know if/what any extra details would be helpful in debugging. Thanks!

lukeseawalker commented 5 years ago

Hi @ssbotelh, can you please post the complete log? Also be sure that ami-0345b9024daa8254b is not already a custom ParalleCluster AMI, even if created with older version of ParallelCluster

ssbotelh commented 5 years ago

Hi @lukeseawalker, thanks for the response. Yes, the base AMI I used was definitely not created with ParallelCluster. Please see the full log below. Thanks.

ssbotelh commented 5 years ago 
+ main --os ubuntu1604 --partition region --region us-west-2 --custom
+ requirements_check
+ packer build --help
+ '[' 0 -ne 0 ']'
+ berks vendor --help
+ '[' 0 -ne 0 ']'
+ parse_options --os ubuntu1604 --partition region --region us-west-2 --custom
+ _os=
+ _partition=
+ _region=
+ _custom=false
+ _public=false
+ _build_date=
+ '[' 7 -eq 0 ']'
+ '[' 7 -gt 0 ']'
+ case "$1" in
+ _os=ubuntu1604
+ shift
+ shift
+ '[' 5 -gt 0 ']'
+ case "$1" in
+ _partition=region
+ shift
+ shift
+ '[' 3 -gt 0 ']'
+ case "$1" in
+ _region=us-west-2
+ shift
+ shift
+ '[' 1 -gt 0 ']'
+ case "$1" in
+ _custom=true
+ shift
+ '[' 0 -gt 0 ']'
+ check_options
+ set -e
+ available_os='centos6 centos7 alinux ubuntu1404 ubuntu1604'
++ dirname /tmp/tmpdXj80U/aws-parallelcluster-cookbook-2.4.1/amis/build_ami.sh
+ cwd=/tmp/tmpdXj80U/aws-parallelcluster-cookbook-2.4.1/amis
++ mktemp -d
+ tmp_dir=/tmp/tmp.gnn1muXh0J
+ export VENDOR_PATH=/tmp/tmp.gnn1muXh0J/vendor/cookbooks
+ VENDOR_PATH=/tmp/tmp.gnn1muXh0J/vendor/cookbooks
+ '[' true == true ']'
+ only=custom-ubuntu1604
+ '[' xubuntu1604 == x ']'
+ '[' xus-west-2 == x ']'
+ '[' false == true ']'
+ '[' region == commercial ']'
+ '[' region == govcloud ']'
+ '[' region == china ']'
+ '[' region == region ']'
+ export AWS_REGION=us-west-2
+ AWS_REGION=us-west-2
+ '[' us-west-2 == all ']'
+ export BUILD_FOR=us-west-2
+ BUILD_FOR=us-west-2
+ do_command
+ RC=0
+ rm -rf /tmp/tmp.gnn1muXh0J/vendor/cookbooks
+ berks vendor /tmp/tmp.gnn1muXh0J/vendor/cookbooks --berksfile /tmp/tmpdXj80U/aws-parallelcluster-cookbook-2.4.1/amis/../Berksfile
Resolving cookbook dependencies...
Fetching 'aws-parallelcluster' from source at .
Fetching cookbook index from https://supermarket.chef.io...
Using apt (7.0.0)
Using aws-parallelcluster (2.4.1) from source at .
Using mingw (2.1.0)
Using nfs (2.5.1)
Using ohai (5.3.0)
Using openssh (2.6.3)
Using poise (2.8.2)
Using poise-archive (1.5.0)
Using poise-languages (2.1.2)
Using poise-python (1.7.0)
Using pyenv (3.1.0)
Using selinux (2.1.1)
Using seven_zip (3.1.1)
Using sysctl (1.0.5)
Using tar (2.1.1)
Using ulimit (1.0.0)
Using windows (6.0.0)
Using yum (5.1.0)
Using yum-epel (3.1.0)
Using build-essential (8.1.1)
Using hostname (0.4.2)
Using hostsfile (3.0.1)
Using line (1.0.6)
Using iptables (4.5.0)
Vendoring apt (7.0.0) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/apt
Vendoring aws-parallelcluster (2.4.1) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/aws-parallelcluster
Vendoring build-essential (8.1.1) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/build-essential
Vendoring hostname (0.4.2) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/hostname
Vendoring hostsfile (3.0.1) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/hostsfile
Vendoring iptables (4.5.0) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/iptables
Vendoring line (1.0.6) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/line
Vendoring mingw (2.1.0) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/mingw
Vendoring nfs (2.5.1) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/nfs
Vendoring ohai (5.3.0) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/ohai
Vendoring openssh (2.6.3) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/openssh
Vendoring poise (2.8.2) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/poise
Vendoring poise-archive (1.5.0) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/poise-archive
Vendoring poise-languages (2.1.2) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/poise-languages
Vendoring poise-python (1.7.0) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/poise-python
Vendoring pyenv (3.1.0) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/pyenv
Vendoring selinux (2.1.1) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/selinux
Vendoring seven_zip (3.1.1) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/seven_zip
Vendoring sysctl (1.0.5) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/sysctl
Vendoring tar (2.1.1) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/tar
Vendoring ulimit (1.0.0) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/ulimit
Vendoring windows (6.0.0) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/windows
Vendoring yum (5.1.0) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/yum
Vendoring yum-epel (3.1.0) to /tmp/tmp.gnn1muXh0J/vendor/cookbooks/yum-epel
+ '[' x == x ']'
++ date +%Y%m%d%H%M
+ export BUILD_DATE=201909091554
+ BUILD_DATE=201909091554
+ export AWS_TIMEOUT_SECONDS=3600
+ AWS_TIMEOUT_SECONDS=3600
+ case ${_os} in
+ packer build -color=false -var-file=/tmp/tmpdXj80U/aws-parallelcluster-cookbook-2.4.1/amis/packer_variables.json -only=custom-ubuntu1604 /tmp/tmpdXj80U/aws-parallelcluster-cookbook-2.4.1/amis/packer_ubuntu1604.json
==> custom-ubuntu1604: Prevalidating AMI Name: base_cuda_aws-parallelcluster-2.4.1-ubuntu-1604-lts-hvm-201909091554
custom-ubuntu1604: Found Image ID: ami-0345b9024daa8254b
==> custom-ubuntu1604: Creating temporary keypair: packer_5d7675bf-b850-5f03-6690-6b3a799e646c
==> custom-ubuntu1604: Creating temporary security group for this instance: packer_5d7675c0-04f5-0f12-ff87-41c1d5fa8b8e
==> custom-ubuntu1604: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
==> custom-ubuntu1604: Launching a source AWS instance...
==> custom-ubuntu1604: Adding tags to source instance
custom-ubuntu1604: Adding tag: "Name": "Packer Builder ami-0345b9024daa8254b"
custom-ubuntu1604: Adding tag: "parallelcluster_version": "aws-parallelcluster-2.4.1"
custom-ubuntu1604: Adding tag: "build_date": "201909091554"
custom-ubuntu1604: Instance ID: i-0bc155bd12a4bdceb
==> custom-ubuntu1604: Waiting for instance (i-0bc155bd12a4bdceb) to become ready...
==> custom-ubuntu1604: Using ssh communicator to connect: 34.214.6.180
==> custom-ubuntu1604: Waiting for SSH to become available...
==> custom-ubuntu1604: Connected to SSH!
==> custom-ubuntu1604: Provisioning with shell script: /tmp/packer-shell283484417
custom-ubuntu1604: Waiting for cloud-init...
custom-ubuntu1604: Waiting for cloud-init...
custom-ubuntu1604: Waiting for cloud-init...
custom-ubuntu1604: Waiting for cloud-init...
custom-ubuntu1604: Waiting for cloud-init...
custom-ubuntu1604: Waiting for cloud-init...
custom-ubuntu1604: Waiting for cloud-init...
custom-ubuntu1604: Waiting for cloud-init...
custom-ubuntu1604: Waiting for cloud-init...
custom-ubuntu1604: Waiting for cloud-init...
custom-ubuntu1604: Waiting for cloud-init...
==> custom-ubuntu1604: Provisioning with shell script: /tmp/packer-shell010794464
custom-ubuntu1604: build-essential - Informational list of build-essential packages
custom-ubuntu1604: crossbuild-essential-arm64 - Informational list of cross-build-essential packages
custom-ubuntu1604: crossbuild-essential-armhf - Informational list of cross-build-essential packages
custom-ubuntu1604: crossbuild-essential-powerpc - Informational list of cross-build-essential packages
custom-ubuntu1604: crossbuild-essential-ppc64el - Informational list of cross-build-essential packages
custom-ubuntu1604: devscripts - scripts to make the life of a Debian Package maintainer easier
custom-ubuntu1604: crossbuild-essential-armel - Informational list of cross-build-essential packages
custom-ubuntu1604: crossbuild-essential-s390x - Informational list of cross-build-essential packages
custom-ubuntu1604: dh-buildinfo - Debhelper addon to track package versions used to build a package
custom-ubuntu1604: sbuild - Tool for building Debian binary packages from Debian sources
custom-ubuntu1604: Get:1 file:/var/cuda-repo-10-1-local-10.1.243-418.87.00  InRelease
custom-ubuntu1604: Ign:1 file:/var/cuda-repo-10-1-local-10.1.243-418.87.00  InRelease
custom-ubuntu1604: Get:2 file:/var/cuda-repo-10-1-local-10.1.243-418.87.00  Release [574 B]
custom-ubuntu1604: Hit:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial InRelease
custom-ubuntu1604: Get:2 file:/var/cuda-repo-10-1-local-10.1.243-418.87.00  Release [574 B]
custom-ubuntu1604: Get:4 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
custom-ubuntu1604: Get:5 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
custom-ubuntu1604: Get:6 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
custom-ubuntu1604: Get:8 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main Sources [339 kB]
custom-ubuntu1604: Get:9 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe Sources [260 kB]
custom-ubuntu1604: Get:10 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [1,021 kB]
custom-ubuntu1604: Get:11 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [398 kB]
custom-ubuntu1604: Get:12 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [764 kB]
custom-ubuntu1604: Get:13 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [319 kB]
custom-ubuntu1604: Get:14 http://security.ubuntu.com/ubuntu xenial-security/main Sources [154 kB]
custom-ubuntu1604: Get:15 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [732 kB]
custom-ubuntu1604: Fetched 4,313 kB in 7s (562 kB/s)
custom-ubuntu1604: Reading package lists... Done
==> custom-ubuntu1604: Provisioning with shell script: /tmp/packer-shell764960171
custom-ubuntu1604: Reading package lists... Done
custom-ubuntu1604: Building dependency tree
custom-ubuntu1604: Reading state information... Done
custom-ubuntu1604: build-essential is already the newest version (12.1ubuntu2).
custom-ubuntu1604: jq is already the newest version (1.5+dfsg-1ubuntu0.1).
custom-ubuntu1604: The following additional packages will be installed:
custom-ubuntu1604:   libcurl3-gnutls
custom-ubuntu1604: The following packages will be upgraded:
custom-ubuntu1604:   curl libcurl3-gnutls wget
custom-ubuntu1604: 3 upgraded, 0 newly installed, 0 to remove and 186 not upgraded.
custom-ubuntu1604: Need to get 621 kB of archives.
custom-ubuntu1604: After this operation, 0 B of additional disk space will be used.
custom-ubuntu1604: Get:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 curl amd64 7.47.0-1ubuntu2.13 [139 kB]
custom-ubuntu1604: Get:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libcurl3-gnutls amd64 7.47.0-1ubuntu2.13 [184 kB]
custom-ubuntu1604: Get:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 wget amd64 1.17.1-1ubuntu1.5 [299 kB]
custom-ubuntu1604: Fetched 621 kB in 0s (19.5 MB/s)
custom-ubuntu1604: (Reading database ... 149851 files and directories currently installed.)
custom-ubuntu1604: Preparing to unpack .../curl_7.47.0-1ubuntu2.13_amd64.deb ...
custom-ubuntu1604: Unpacking curl (7.47.0-1ubuntu2.13) over (7.47.0-1ubuntu2.11) ...
custom-ubuntu1604: Preparing to unpack .../libcurl3-gnutls_7.47.0-1ubuntu2.13_amd64.deb ...
custom-ubuntu1604: Unpacking libcurl3-gnutls:amd64 (7.47.0-1ubuntu2.13) over (7.47.0-1ubuntu2.11) ...
custom-ubuntu1604: Preparing to unpack .../wget_1.17.1-1ubuntu1.5_amd64.deb ...
custom-ubuntu1604: Unpacking wget (1.17.1-1ubuntu1.5) over (1.17.1-1ubuntu1.4) ...
custom-ubuntu1604: Processing triggers for man-db (2.7.5-1) ...
custom-ubuntu1604: Processing triggers for libc-bin (2.23-0ubuntu10) ...
custom-ubuntu1604: Processing triggers for install-info (6.1.0.dfsg.1-5) ...
custom-ubuntu1604: Setting up libcurl3-gnutls:amd64 (7.47.0-1ubuntu2.13) ...
custom-ubuntu1604: Setting up curl (7.47.0-1ubuntu2.13) ...
custom-ubuntu1604: Setting up wget (1.17.1-1ubuntu1.5) ...
custom-ubuntu1604: Processing triggers for libc-bin (2.23-0ubuntu10) ...
==> custom-ubuntu1604: Provisioning with shell script: /tmp/packer-shell697333852
==> custom-ubuntu1604: Uploading /tmp/tmp.gnn1muXh0J/vendor/cookbooks => /tmp/cookbooks
==> custom-ubuntu1604: Provisioning with shell script: /tmp/packer-shell237795978
==> custom-ubuntu1604: Provisioning with shell script: /tmp/packer-shell680217388
custom-ubuntu1604:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
custom-ubuntu1604:                                  Dload  Upload   Total   Spent    Left  Speed
custom-ubuntu1604: 100 23097  100 23097    0     0   132k      0 --:--:-- --:--:-- --:--:--  131k
custom-ubuntu1604: ubuntu 16.04 x86_64
custom-ubuntu1604: Getting information for chef stable 14.2.0 for ubuntu...
custom-ubuntu1604: downloading https://www.chef.io/stable/chef/metadata?v=14.2.0&p=ubuntu&pv=16.04&m=x86_64
custom-ubuntu1604:   to file /tmp/install.sh.2829/metadata.txt
custom-ubuntu1604: trying wget...
custom-ubuntu1604: sha1 1a907db833e5a2feeab7c79845f5a8ae61f8f555
custom-ubuntu1604: sha256   6832fd71a42c014e78d5f2f9d749b674df78bd4b1e1c28c66d224caf5ca8623d
custom-ubuntu1604: url  https://packages.chef.io/files/stable/chef/14.2.0/ubuntu/16.04/chef_14.2.0-1_amd64.deb
custom-ubuntu1604: version  14.2.0
custom-ubuntu1604: downloaded metadata file looks valid...
custom-ubuntu1604: downloading https://packages.chef.io/files/stable/chef/14.2.0/ubuntu/16.04/chef_14.2.0-1_amd64.deb
custom-ubuntu1604:   to file /tmp/install.sh.2829/chef_14.2.0-1_amd64.deb
custom-ubuntu1604: trying wget...
custom-ubuntu1604: Comparing checksum with sha256sum...
custom-ubuntu1604: Installing chef 14.2.0
custom-ubuntu1604: installing with dpkg...
custom-ubuntu1604: (Reading database ... 149851 files and directories currently installed.)
custom-ubuntu1604: Preparing to unpack .../chef_14.2.0-1_amd64.deb ...
custom-ubuntu1604: Unpacking chef (14.2.0-1) over (14.2.0-1) ...
custom-ubuntu1604: Setting up chef (14.2.0-1) ...
custom-ubuntu1604: Thank you for installing Chef!
==> custom-ubuntu1604: Provisioning with shell script: /tmp/packer-shell504458760
custom-ubuntu1604: Successfully installed ridley-5.1.1
custom-ubuntu1604: Successfully installed berkshelf-7.0.4
custom-ubuntu1604: 2 gems installed
==> custom-ubuntu1604: Provisioning with chef-solo
custom-ubuntu1604: Creating directory: /tmp/packer-chef-solo
custom-ubuntu1604: Creating configuration file 'solo.rb'
custom-ubuntu1604: Creating JSON attribute file
custom-ubuntu1604: Executing Chef: sudo chef-client -z --no-color -c /tmp/packer-chef-solo/solo.rb -j /tmp/packer-chef-solo/node.json
custom-ubuntu1604: Starting Chef Client, version 14.2.0
custom-ubuntu1604: resolving cookbooks for run list: ["aws-parallelcluster::default"]
custom-ubuntu1604: Synchronizing Cookbooks:
custom-ubuntu1604:   - aws-parallelcluster (2.4.1)
custom-ubuntu1604:   - build-essential (8.1.1)
custom-ubuntu1604:   - poise-python (1.7.0)
custom-ubuntu1604:   - tar (2.1.1)
custom-ubuntu1604:   - selinux (2.1.1)
custom-ubuntu1604:   - nfs (2.5.1)
custom-ubuntu1604:   - sysctl (1.0.5)
custom-ubuntu1604:   - yum (5.1.0)
custom-ubuntu1604:   - yum-epel (3.1.0)
custom-ubuntu1604:   - openssh (2.6.3)
custom-ubuntu1604:   - apt (7.0.0)
custom-ubuntu1604:   - hostname (0.4.2)
custom-ubuntu1604:   - line (1.0.6)
custom-ubuntu1604:   - ulimit (1.0.0)
custom-ubuntu1604:   - pyenv (3.1.0)
custom-ubuntu1604:   - seven_zip (3.1.1)
custom-ubuntu1604:   - mingw (2.1.0)
custom-ubuntu1604:   - poise (2.8.2)
custom-ubuntu1604:   - poise-languages (2.1.2)
custom-ubuntu1604:   - ohai (5.3.0)
custom-ubuntu1604:   - iptables (4.5.0)
custom-ubuntu1604:   - hostsfile (3.0.1)
custom-ubuntu1604:   - windows (6.0.0)
custom-ubuntu1604:   - poise-archive (1.5.0)
custom-ubuntu1604: Installing Cookbook Gems:
custom-ubuntu1604: Compiling Cookbooks...
custom-ubuntu1604: Converging 138 resources
custom-ubuntu1604: Recipe: apt::default
custom-ubuntu1604:   * file[/var/lib/apt/periodic/update-success-stamp] action nothing (skipped due to action :nothing)
custom-ubuntu1604:   * apt_update[periodic] action periodic (up to date)
custom-ubuntu1604:   * execute[apt-get update] action nothing (skipped due to action :nothing)
custom-ubuntu1604:   * execute[apt-get autoremove] action nothing (skipped due to action :nothing)
custom-ubuntu1604:   * execute[apt-get autoclean] action nothing (skipped due to action :nothing)
custom-ubuntu1604:   * directory[/var/cache/local] action create (up to date)
custom-ubuntu1604:   * directory[/var/cache/local/preseeding] action create (up to date)
custom-ubuntu1604:   * template[/etc/apt/apt.conf.d/10dpkg-options] action create (up to date)
custom-ubuntu1604:   * template[/etc/apt/apt.conf.d/10recommends] action create (up to date)
custom-ubuntu1604:   * apt_package[apt-transport-https] action install (up to date)
custom-ubuntu1604: Recipe: aws-parallelcluster::base_install
custom-ubuntu1604:   * directory[/etc/parallelcluster] action create (up to date)
custom-ubuntu1604:   * directory[/opt/parallelcluster] action create (up to date)
custom-ubuntu1604:   * directory[/opt/parallelcluster/sources] action create (up to date)
custom-ubuntu1604:   * directory[/opt/parallelcluster/scripts] action create (up to date)
custom-ubuntu1604:   * directory[/opt/parallelcluster/licenses] action create (up to date)
custom-ubuntu1604:   * build_essential[] action install
custom-ubuntu1604:     * apt_package[autoconf, binutils-doc, bison, build-essential, flex, gettext, ncurses-dev] action install (up to date)
custom-ubuntu1604:      (up to date)
custom-ubuntu1604: Recipe: aws-parallelcluster::_setup_python
custom-ubuntu1604:   * pyenv_user_install[root] action install
custom-ubuntu1604:     * apt_update[update] action periodic (up to date)
custom-ubuntu1604:     * build_essential[build packages] action install
custom-ubuntu1604:       * apt_package[autoconf, binutils-doc, bison, build-essential, flex, gettext, ncurses-dev] action install (up to date)
custom-ubuntu1604:        (up to date)
custom-ubuntu1604:     * apt_package[make, libssl-dev, zlib1g-dev, libbz2-dev, libreadline-dev, libsqlite3-dev, wget, curl, llvm, libncurses5-dev, libncursesw5-dev, xz-utils, tk-dev, libffi-dev, liblzma-dev, python-openssl, git] action install
custom-ubuntu1604:       - install version 3.11.0-1ubuntu1.2 of package libsqlite3-dev
custom-ubuntu1604:       - install version 1:3.8-33ubuntu3.1 of package llvm
custom-ubuntu1604:       - install version 6.0+20160213-1ubuntu1 of package libncursesw5-dev
custom-ubuntu1604:       - install version 8.6.0+9 of package tk-dev
custom-ubuntu1604:       - install version 0.15.1-2ubuntu0.2 of package python-openssl
custom-ubuntu1604:     * template[/etc/profile.d/pyenv.sh] action create
custom-ubuntu1604:       - create new file /etc/profile.d/pyenv.sh
custom-ubuntu1604:       - update content in file /etc/profile.d/pyenv.sh from none to a1c685
custom-ubuntu1604:       --- /etc/profile.d/pyenv.sh    2019-09-09 15:58:10.378696014 +0000
custom-ubuntu1604:       +++ /etc/profile.d/.chef-pyenv20190909-2937-clen0a.sh  2019-09-09 15:58:10.378696014 +0000
custom-ubuntu1604:       @@ -1 +1,13 @@
custom-ubuntu1604:       +# Prefer a user pyenv over a system wide install
custom-ubuntu1604:       +if [ -s "${HOME}/.pyenv/bin" ]; then
custom-ubuntu1604:       +    pyenv_root="${HOME}/.pyenv"
custom-ubuntu1604:       +elif [ -s "" ]; then
custom-ubuntu1604:       +    pyenv_root=""
custom-ubuntu1604:       +    export PYENV_ROOT=${pyenv_root}
custom-ubuntu1604:       +fi
custom-ubuntu1604:       +
custom-ubuntu1604:       +if [ -n "$pyenv_root" ]; then
custom-ubuntu1604:       +    export PATH="${pyenv_root}/bin:$PATH"
custom-ubuntu1604:       +    eval "$(pyenv init -)"
custom-ubuntu1604:       +fi
custom-ubuntu1604:       - change mode from '' to '0755'
custom-ubuntu1604:       - change owner from '' to 'root'
custom-ubuntu1604:     * git[/root/.pyenv] action sync
custom-ubuntu1604:       - clone from https://github.com/pyenv/pyenv.git into /root/.pyenv
custom-ubuntu1604:       - checkout ref 6563b64d0bca945ffa791f9dfcde75ec947a594d branch master
custom-ubuntu1604:     * ruby_block[Add pyenv to PATH] action run
custom-ubuntu1604:       - execute the ruby block Add pyenv to PATH
custom-ubuntu1604:     * bash[Initialize user root pyenv] action run
custom-ubuntu1604:       - execute "bash"  "/tmp/chef-script20190909-2937-micg5t"
custom-ubuntu1604:     * directory[/root/.pyenv/plugins] action create (up to date)
custom-ubuntu1604:     * directory[/root/.pyenv/shims] action create (up to date)
custom-ubuntu1604:     * directory[/root/.pyenv/versions] action create (up to date)
custom-ubuntu1604:     * ruby_block[Add pyenv to PATH] action nothing (skipped due to action :nothing)
custom-ubuntu1604:     * bash[Initialize user root pyenv] action nothing (skipped due to action :nothing)
custom-ubuntu1604:
custom-ubuntu1604:   * pyenv_python[3.6.9] action install
custom-ubuntu1604:     * pyenv_script[pyenv install 3.6.9  (root)] action run
custom-ubuntu1604:       * bash[pyenv install 3.6.9  (root)] action run
custom-ubuntu1604:         - execute "bash" -e "/tmp/chef-script20190909-2937-1xwiuwe"
custom-ubuntu1604:
custom-ubuntu1604:
custom-ubuntu1604:   * pyenv_plugin[virtualenv] action install
custom-ubuntu1604:     * git[Install virtualenv plugin] action sync
custom-ubuntu1604:       - clone from https://github.com/pyenv/pyenv-virtualenv into /root/.pyenv/plugins/virtualenv
custom-ubuntu1604:       - checkout ref 294f64f76b6b7fbf1a22a4ebba7710faa75c21f7 branch master
custom-ubuntu1604:
custom-ubuntu1604:   * pyenv_script[pyenv virtualenv cookbook] action run
custom-ubuntu1604:     * bash[pyenv virtualenv cookbook] action run
custom-ubuntu1604:       - execute "bash" -e "/tmp/chef-script20190909-2937-1p4a701"
custom-ubuntu1604:
custom-ubuntu1604:   * pyenv_script[pyenv virtualenv node] action run
custom-ubuntu1604:     * bash[pyenv virtualenv node] action run
custom-ubuntu1604:       - execute "bash" -e "/tmp/chef-script20190909-2937-ah2eao"
custom-ubuntu1604:
custom-ubuntu1604:   * cookbook_file[/root/.pyenv/versions/3.6.9/envs/cookbook_virtualenv/requirements.txt] action create
custom-ubuntu1604:     - create new file /root/.pyenv/versions/3.6.9/envs/cookbook_virtualenv/requirements.txt
custom-ubuntu1604:     - update content in file /root/.pyenv/versions/3.6.9/envs/cookbook_virtualenv/requirements.txt from none to d0242c
custom-ubuntu1604:     --- /root/.pyenv/versions/3.6.9/envs/cookbook_virtualenv/requirements.txt    2019-09-09 15:59:47.271978029 +0000
custom-ubuntu1604:     +++ /root/.pyenv/versions/3.6.9/envs/cookbook_virtualenv/.chef-requirements20190909-2937-1uh03x3.txt 2019-09-09 15:59:47.271978029 +0000
custom-ubuntu1604:     @@ -1 +1,5 @@
custom-ubuntu1604:     +awscli
custom-ubuntu1604:     +boto3
custom-ubuntu1604:     +supervisor
custom-ubuntu1604:     +requests
custom-ubuntu1604:     - change mode from '' to '0755'
custom-ubuntu1604:     - change owner from '' to 'root'
custom-ubuntu1604:     - change group from '' to 'root'
custom-ubuntu1604:   * pyenv_pip[/root/.pyenv/versions/3.6.9/envs/cookbook_virtualenv/requirements.txt] action install
custom-ubuntu1604:     * pyenv_script[/root/.pyenv/versions/3.6.9/envs/cookbook_virtualenv/requirements.txt] action run
custom-ubuntu1604:       * bash[/root/.pyenv/versions/3.6.9/envs/cookbook_virtualenv/requirements.txt] action run
custom-ubuntu1604:         - execute "bash" -e "/tmp/chef-script20190909-2937-x24ckl"
custom-ubuntu1604:
custom-ubuntu1604:
custom-ubuntu1604: Recipe: aws-parallelcluster::base_install
custom-ubuntu1604:   * apt_package[vim] action install (up to date)
custom-ubuntu1604:   * apt_package[ksh] action install (up to date)
custom-ubuntu1604:   * apt_package[tcsh] action install (up to date)
custom-ubuntu1604:   * apt_package[zsh] action install (up to date)
custom-ubuntu1604:   * apt_package[libssl-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[ncurses-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[libpam-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[net-tools] action install (up to date)
custom-ubuntu1604:   * apt_package[libhwloc-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[dkms] action install (up to date)
custom-ubuntu1604:   * apt_package[tcl-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[automake] action install (up to date)
custom-ubuntu1604:   * apt_package[autoconf] action install (up to date)
custom-ubuntu1604:   * apt_package[python-parted] action install (up to date)
custom-ubuntu1604:   * apt_package[libtool] action install (up to date)
custom-ubuntu1604:   * apt_package[librrd-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[libapr1-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[libconfuse-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[apache2] action install (up to date)
custom-ubuntu1604:   * apt_package[libboost-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[libdb-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[tcsh] action install (up to date)
custom-ubuntu1604:   * apt_package[libssl-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[libncurses5-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[libpam0g-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[libxt-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[libmotif-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[libxmu-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[libxft-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[libhwloc-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[man-db] action install (up to date)
custom-ubuntu1604:   * apt_package[lvm2] action install (up to date)
custom-ubuntu1604:   * apt_package[libmpich-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[python] action install (up to date)
custom-ubuntu1604:   * apt_package[python-pip] action install
custom-ubuntu1604:     - install version 8.1.1-2ubuntu0.4 of package python-pip
custom-ubuntu1604:   * apt_package[r-base] action install (up to date)
custom-ubuntu1604:   * apt_package[libatlas-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[libblas-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[libfftw3-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[libffi-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[libssl-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[libxml2-dev] action install (up to date)
custom-ubuntu1604:   * apt_package[mdadm] action install (up to date)
custom-ubuntu1604:   * bash[install awscli] action run (skipped due to not_if)
custom-ubuntu1604: Recipe: openssh::default
custom-ubuntu1604:   * apt_package[openssh-client, openssh-server] action install (up to date)
custom-ubuntu1604:   * template[/etc/ssh/ssh_config] action create (up to date)
custom-ubuntu1604:   * template[sshd_ca_keys_file] action create (up to date)
custom-ubuntu1604:   * template[sshd_revoked_keys_file] action create (up to date)
custom-ubuntu1604:   * directory[/var/run/sshd] action create (up to date)
custom-ubuntu1604:   * template[/etc/ssh/sshd_config] action create (up to date)
custom-ubuntu1604:   * execute[sshd-config-check] action nothing (skipped due to action :nothing)
custom-ubuntu1604:   * service[ssh] action enable (up to date)
custom-ubuntu1604:   * service[ssh] action start (up to date)
custom-ubuntu1604: Recipe: aws-parallelcluster::base_install
custom-ubuntu1604:   * selinux_state_debian[SELinux Disabled] action disabled (skipped due to only_if)
custom-ubuntu1604:   * cookbook_file[AWS-ParallelCluster-License-README.txt] action create (up to date)
custom-ubuntu1604: Recipe: nfs::_common
custom-ubuntu1604:   * apt_package[nfs-common] action install (up to date)
custom-ubuntu1604:   * apt_package[rpcbind] action install (up to date)
custom-ubuntu1604:   * directory[/etc/default] action create (skipped due to only_if)
custom-ubuntu1604:   * template[/etc/default/nfs-common] action create
custom-ubuntu1604:     - update content in file /etc/default/nfs-common from 4ee4f0 to 5df65a
custom-ubuntu1604:     --- /etc/default/nfs-common  2018-11-16 15:01:38.297266085 +0000
custom-ubuntu1604:     +++ /etc/default/.chef-nfs-common20190909-2937-339cq6    2019-09-09 16:00:00.335496177 +0000
custom-ubuntu1604:     @@ -1,3 +1,3 @@
custom-ubuntu1604:     -# Generated by Chef for ip-172-30-0-80.ec2.internal# Local modifications will be overwritten.
custom-ubuntu1604:     +# Generated by Chef for ip-172-31-25-178.us-west-2.compute.internal# Local modifications will be overwritten.
custom-ubuntu1604:      STATDOPTS="--port 32765 --outgoing-port 32766"
custom-ubuntu1604:   * service[portmap] action restart
custom-ubuntu1604:     - restart service service[portmap]
custom-ubuntu1604:   * service[lock] action restart
custom-ubuntu1604:     - restart service service[lock]
custom-ubuntu1604:   * service[nfs-config.service] action restart
custom-ubuntu1604:     - restart service service[nfs-config.service]
custom-ubuntu1604:   * template[/etc/modprobe.d/lockd.conf] action create
custom-ubuntu1604:     - update content in file /etc/modprobe.d/lockd.conf from 34d82e to 7c52fd
custom-ubuntu1604:     --- /etc/modprobe.d/lockd.conf   2018-11-16 15:01:38.621045195 +0000
custom-ubuntu1604:     +++ /etc/modprobe.d/.chef-lockd20190909-2937-1ldxl3p.conf    2019-09-09 16:00:00.591487326 +0000
custom-ubuntu1604:     @@ -1,4 +1,4 @@
custom-ubuntu1604:     -# Generated by Chef for ip-172-30-0-80.ec2.internal
custom-ubuntu1604:     +# Generated by Chef for ip-172-31-25-178.us-west-2.compute.internal
custom-ubuntu1604:      # Local modifications will be overwritten.
custom-ubuntu1604:      options lockd nlm_udpport=32768 nlm_tcpport=32768
custom-ubuntu1604:   * service[portmap] action restart
custom-ubuntu1604:     - restart service service[portmap]
custom-ubuntu1604:   * service[lock] action restart
custom-ubuntu1604:     - restart service service[lock]
custom-ubuntu1604:   * service[nfs-config.service] action restart
custom-ubuntu1604:     - restart service service[nfs-config.service]
custom-ubuntu1604:   * service[portmap] action start (up to date)
custom-ubuntu1604:   * service[portmap] action enable (up to date)
custom-ubuntu1604:   * service[lock] action start (up to date)
custom-ubuntu1604:   * service[lock] action enable (up to date)
custom-ubuntu1604:   * service[nfs-config.service] action start (up to date)
custom-ubuntu1604:   * service[nfs-config.service] action enable (up to date)
custom-ubuntu1604: Recipe: aws-parallelcluster::base_install
custom-ubuntu1604:   * service[rpcbind] action start (skipped due to only_if)
custom-ubuntu1604:   * service[rpcbind] action enable (skipped due to only_if)
custom-ubuntu1604: Recipe: nfs::server
custom-ubuntu1604:   * apt_package[nfs-kernel-server] action install (up to date)
custom-ubuntu1604:   * template[/etc/default/nfs-kernel-server] action create
custom-ubuntu1604:     - update content in file /etc/default/nfs-kernel-server from b67944 to 637e0c
custom-ubuntu1604:     --- /etc/default/nfs-kernel-server   2018-11-16 15:01:42.490532125 +0000
custom-ubuntu1604:     +++ /etc/default/.chef-nfs-kernel-server20190909-2937-1404wgx    2019-09-09 16:00:01.223465557 +0000
custom-ubuntu1604:     @@ -1,4 +1,4 @@
custom-ubuntu1604:     -# Generated by Chef for ip-172-30-0-80.ec2.internal# Local modifications will be overwritten.
custom-ubuntu1604:     +# Generated by Chef for ip-172-31-25-178.us-west-2.compute.internal# Local modifications will be overwritten.
custom-ubuntu1604:      # Rendered Debian/Ubuntu template variant
custom-ubuntu1604:      RPCMOUNTDOPTS="-p 32767"
custom-ubuntu1604:      RPCNFSDCOUNT="8"
custom-ubuntu1604:   * service[nfs-kernel-server] action start
custom-ubuntu1604:     - start service service[nfs-kernel-server]
custom-ubuntu1604:   * service[nfs-kernel-server] action enable (up to date)
custom-ubuntu1604: Recipe: nfs::_idmap
custom-ubuntu1604:   * template[/etc/idmapd.conf] action create
custom-ubuntu1604:     - update content in file /etc/idmapd.conf from ca812b to 5c7563
custom-ubuntu1604:     --- /etc/idmapd.conf 2018-11-16 15:01:42.638440452 +0000
custom-ubuntu1604:     +++ /etc/.chef-idmapd20190909-2937-l2mnqv.conf   2019-09-09 16:00:01.343461451 +0000
custom-ubuntu1604:     @@ -5,7 +5,7 @@
custom-ubuntu1604:
custom-ubuntu1604:      # The following should be set to the local NFSv4 domain name
custom-ubuntu1604:      # The default is the host's DNS domain name.
custom-ubuntu1604:     -Domain = ec2.internal
custom-ubuntu1604:     +Domain = us-west-2.compute.internal
custom-ubuntu1604:
custom-ubuntu1604:      # The following is a comma-separated list of Kerberos realm
custom-ubuntu1604:      # names that should be considered to be equivalent to the
custom-ubuntu1604:   * service[idmap] action restart
custom-ubuntu1604:     - restart service service[idmap]
custom-ubuntu1604:   * service[idmap] action start (up to date)
custom-ubuntu1604:   * service[idmap] action enable (up to date)
custom-ubuntu1604: Recipe: aws-parallelcluster::base_install
custom-ubuntu1604:   * cookbook_file[configure-pat.sh] action create (up to date)
custom-ubuntu1604:   * cookbook_file[setup-ephemeral-drives.sh] action create
custom-ubuntu1604:     - update content in file /usr/local/sbin/setup-ephemeral-drives.sh from 8ef482 to 536fbc
custom-ubuntu1604:     --- /usr/local/sbin/setup-ephemeral-drives.sh    2018-11-16 15:01:42.890284360 +0000
custom-ubuntu1604:     +++ /usr/local/sbin/.chef-setup-ephemeral-drives20190909-2937-4359k9.sh  2019-09-09 16:00:01.547454472 +0000
custom-ubuntu1604:     @@ -21,7 +21,14 @@
custom-ubuntu1604:        exit 1
custom-ubuntu1604:      }
custom-ubuntu1604:
custom-ubuntu1604:     +function exec_command() {
custom-ubuntu1604:     +  _command_output=$($@ 2>&1)
custom-ubuntu1604:     +  _exit_code=$?
custom-ubuntu1604:
custom-ubuntu1604:     +  # Do not set RC=1 if error says that changes have been written but a reboot is required to inform the kernel
custom-ubuntu1604:     +  [[ $_exit_code -ne 0 && $(echo "${_command_output}" | grep -i "you should reboot now") ]] && RC=1
custom-ubuntu1604:     +}
custom-ubuntu1604:     +
custom-ubuntu1604:      # LVM stripe, format, mount ephemeral drives
custom-ubuntu1604:      function setup_ephemeral_drives () {
custom-ubuntu1604:        RC=0
custom-ubuntu1604:     @@ -48,11 +55,11 @@
custom-ubuntu1604:          for d in $DEVS; do
custom-ubuntu1604:            d=/dev/${d}
custom-ubuntu1604:            dd if=/dev/zero of=${d} bs=32k count=1 || RC=1
custom-ubuntu1604:     -      parted -s ${d} mklabel gpt || RC=1
custom-ubuntu1604:     -      parted -s ${d} || RC=1
custom-ubuntu1604:     -      parted -s -a optimal ${d} mkpart primary 1MB 100% || RC=1
custom-ubuntu1604:     -      partprobe
custom-ubuntu1604:     -      parted -s ${d} set 1 lvm on || RC=1
custom-ubuntu1604:     +      exec_command "parted -s ${d} mklabel gpt"
custom-ubuntu1604:     +      exec_command "parted -s ${d}"
custom-ubuntu1604:     +      exec_command "parted -s -a optimal ${d} mkpart primary 1MB 100%"
custom-ubuntu1604:     +      partprobe ${d}
custom-ubuntu1604:     +      exec_command "parted -s ${d} set 1 lvm on"
custom-ubuntu1604:            if [ $IS_NVME -eq 1 ]; then
custom-ubuntu1604:              PARTITIONS="${d}p1 $PARTITIONS"
custom-ubuntu1604:            else
custom-ubuntu1604: Recipe: aws-parallelcluster::_ec2_udev_rules
custom-ubuntu1604:   * template[ec2-volid.rules] action create
custom-ubuntu1604:     - update content in file /etc/udev/rules.d/52-ec2-volid.rules from cf2a20 to 9b1963
custom-ubuntu1604:     --- /etc/udev/rules.d/52-ec2-volid.rules 2018-11-16 15:01:42.898279405 +0000
custom-ubuntu1604:     +++ /etc/udev/rules.d/.chef-52-ec2-volid20190909-2937-13bzotq.rules  2019-09-09 16:00:01.551454335 +0000
custom-ubuntu1604:     @@ -1,5 +1,5 @@
custom-ubuntu1604:     -KERNEL=="xvd*", ENV{DEVTYPE}=="disk", PROGRAM="/sbin/ec2_dev_2_volid.py %k", SYMLINK+="disk/by-ebs-volumeid/%c"
custom-ubuntu1604:     -KERNEL=="xvd*", ENV{DEVTYPE}=="partition", PROGRAM="/sbin/ec2_dev_2_volid.py %k", SYMLINK+="disk/by-ebs-volumeid/%c-p%n"
custom-ubuntu1604:     -KERNEL=="nvme*", ENV{DEVTYPE}=="disk", PROGRAM="/sbin/ec2_dev_2_volid.py %k", SYMLINK+="disk/by-ebs-volumeid/%c"
custom-ubuntu1604:     -KERNEL=="nvme*", ENV{DEVTYPE}=="partition", PROGRAM="/sbin/ec2_dev_2_volid.py %k", SYMLINK+="disk/by-ebs-volumeid/%c-p%n"
custom-ubuntu1604:     +KERNEL=="xvd*", ENV{DEVTYPE}=="disk", PROGRAM="/root/.pyenv/versions/3.6.9/envs/cookbook_virtualenv/bin/python /sbin/ec2_dev_2_volid.py %k", SYMLINK+="disk/by-ebs-volumeid/%c"
custom-ubuntu1604:     +KERNEL=="xvd*", ENV{DEVTYPE}=="partition", PROGRAM="/root/.pyenv/versions/3.6.9/envs/cookbook_virtualenv/bin/python /sbin/ec2_dev_2_volid.py %k", SYMLINK+="disk/by-ebs-volumeid/%c-p%n"
custom-ubuntu1604:     +KERNEL=="nvme*", ENV{DEVTYPE}=="disk", PROGRAM="/root/.pyenv/versions/3.6.9/envs/cookbook_virtualenv/bin/python /sbin/ec2_dev_2_volid.py %k", SYMLINK+="disk/by-ebs-volumeid/%c"
custom-ubuntu1604:     +KERNEL=="nvme*", ENV{DEVTYPE}=="partition", PROGRAM="/root/.pyenv/versions/3.6.9/envs/cookbook_virtualenv/bin/python /sbin/ec2_dev_2_volid.py %k", SYMLINK+="disk/by-ebs-volumeid/%c-p%n"
custom-ubuntu1604:   * cookbook_file[parallelcluster-ebsnvme-id] action create (up to date)
custom-ubuntu1604:   * cookbook_file[ec2_dev_2_volid.py] action create
custom-ubuntu1604:     - update content in file /sbin/ec2_dev_2_volid.py from 3e8cc0 to 0d4626
custom-ubuntu1604:     --- /sbin/ec2_dev_2_volid.py 2018-11-16 15:01:42.922264539 +0000
custom-ubuntu1604:     +++ /sbin/.chef-ec2_dev_2_volid20190909-2937-fgsf7z.py   2019-09-09 16:00:01.563453924 +0000
custom-ubuntu1604:     @@ -1,12 +1,10 @@
custom-ubuntu1604:     -#!/usr/bin/env python
custom-ubuntu1604:     -
custom-ubuntu1604:     -import urllib2
custom-ubuntu1604:     +import requests
custom-ubuntu1604:      import sys
custom-ubuntu1604:      import os
custom-ubuntu1604:      import syslog
custom-ubuntu1604:      import time
custom-ubuntu1604:      import boto3
custom-ubuntu1604:     -import ConfigParser
custom-ubuntu1604:     +import configparser
custom-ubuntu1604:      from botocore.config import Config
custom-ubuntu1604:
custom-ubuntu1604:
custom-ubuntu1604:     @@ -31,20 +29,20 @@
custom-ubuntu1604:              dev = '/dev/' + dev
custom-ubuntu1604:
custom-ubuntu1604:          # Get instance ID
custom-ubuntu1604:     -    instanceId = urllib2.urlopen("http://169.254.169.254/latest/meta-data/instance-id").read()
custom-ubuntu1604:     +    instanceId = requests.get("http://169.254.169.254/latest/meta-data/instance-id").text
custom-ubuntu1604:
custom-ubuntu1604:          # Get region
custom-ubuntu1604:     -    region = urllib2.urlopen("http://169.254.169.254/latest/meta-data/placement/availability-zone").read()
custom-ubuntu1604:     +    region = requests.get("http://169.254.169.254/latest/meta-data/placement/availability-zone").text
custom-ubuntu1604:          region = region[:-1]
custom-ubuntu1604:
custom-ubuntu1604:          # Parse configuration file to read proxy settings
custom-ubuntu1604:     -    config = ConfigParser.RawConfigParser()
custom-ubuntu1604:     +    config = configparser.RawConfigParser()
custom-ubuntu1604:          config.read('/etc/boto.cfg')
custom-ubuntu1604:          proxy_config = Config()
custom-ubuntu1604:          if config.has_option('Boto', 'proxy') and config.has_option('Boto', 'proxy_port'):
custom-ubuntu1604:              proxy = config.get('Boto', 'proxy')
custom-ubuntu1604:              proxy_port = config.get('Boto', 'proxy_port')
custom-ubuntu1604:     -        proxy_config = Config(proxies={'https': "{}:{}".format(proxy, proxy_port)})
custom-ubuntu1604:     +        proxy_config = Config(proxies={'https': "{0}:{1}".format(proxy, proxy_port)})
custom-ubuntu1604:
custom-ubuntu1604:          # Connect to AWS using boto
custom-ubuntu1604:          ec2 = boto3.client('ec2', region_name=region, config=proxy_config)
custom-ubuntu1604:     @@ -53,7 +51,7 @@
custom-ubuntu1604:          devices = ec2.describe_instance_attribute(InstanceId=instanceId, Attribute='blockDeviceMapping').get('BlockDeviceMappings')
custom-ubuntu1604:          devmap = dict((d.get('DeviceName'), d) for d in devices)
custom-ubuntu1604:          x = 0
custom-ubuntu1604:     -    while not devmap.has_key(dev):
custom-ubuntu1604:     +    while dev not in devmap:
custom-ubuntu1604:              if x == 36:
custom-ubuntu1604:                  syslog.syslog("Dev %s did not appears in 180 seconds." % dev)
custom-ubuntu1604:                  sys.exit(1)
custom-ubuntu1604:   * cookbook_file[ec2blkdev-init] action create (up to date)
custom-ubuntu1604:   * cookbook_file[attachVolume.py] action create
custom-ubuntu1604:     - update content in file /usr/local/sbin/attachVolume.py from f7714d to 0895f8
custom-ubuntu1604:     --- /usr/local/sbin/attachVolume.py  2018-11-16 15:01:42.938254629 +0000
custom-ubuntu1604:     +++ /usr/local/sbin/.chef-attachVolume20190909-2937-1kh7dc1.py   2019-09-09 16:00:01.575453514 +0000
custom-ubuntu1604:     @@ -1,12 +1,10 @@
custom-ubuntu1604:     -#!/usr/bin/env python
custom-ubuntu1604:     -
custom-ubuntu1604:      import sys
custom-ubuntu1604:     -import parted
custom-ubuntu1604:     +import subprocess
custom-ubuntu1604:      import os
custom-ubuntu1604:     -import urllib2
custom-ubuntu1604:     +import requests
custom-ubuntu1604:      import boto3
custom-ubuntu1604:      import time
custom-ubuntu1604:     -import ConfigParser
custom-ubuntu1604:     +import configparser
custom-ubuntu1604:      from botocore.config import Config
custom-ubuntu1604:
custom-ubuntu1604:
custom-ubuntu1604:     @@ -15,7 +13,7 @@
custom-ubuntu1604:          # FIXME This approach could be broken in some OS variants, see
custom-ubuntu1604:          # https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html
custom-ubuntu1604:          if '/nvme' in dev:
custom-ubuntu1604:     -        return '/dev/' + os.popen('sudo /usr/local/sbin/parallelcluster-ebsnvme-id -b ' + dev).read().strip()
custom-ubuntu1604:     +        return '/dev/' + os.popen('sudo /usr/local/sbin/parallelcluster-ebsnvme-id -u -b ' + dev).read().strip()
custom-ubuntu1604:          elif '/hd' in dev:
custom-ubuntu1604:              return dev.replace('hd', 'sd')
custom-ubuntu1604:          elif '/xvd' in dev:
custom-ubuntu1604:     @@ -23,24 +21,36 @@
custom-ubuntu1604:          else:
custom-ubuntu1604:              return dev
custom-ubuntu1604:
custom-ubuntu1604:     +def get_all_devices():
custom-ubuntu1604:     +    # lsblk -d -n
custom-ubuntu1604:     +    # xvda 202:0    0  17G  0 disk
custom-ubuntu1604:     +    # xvdb 202:16   0  20G  0 disk /shared
custom-ubuntu1604:     +    command = ["/bin/lsblk", "-d", "-n"]
custom-ubuntu1604:
custom-ubuntu1604:     +    try:
custom-ubuntu1604:     +        output = subprocess.check_output(command, stderr=subprocess.STDOUT, universal_newlines=True).split("\n")
custom-ubuntu1604:     +        return ["/dev/{}".format(line.split()[0]) for line in output if len(line.split()) > 0]
custom-ubuntu1604:     +    except subprocess.CalledProcessError as e:
custom-ubuntu1604:     +        print("Failed to get devices with lsblk -d -n")
custom-ubuntu1604:     +        raise e
custom-ubuntu1604:     +
custom-ubuntu1604:      def main():
custom-ubuntu1604:          # Get EBS volume Id
custom-ubuntu1604:          try:
custom-ubuntu1604:              volumeId = str(sys.argv[1])
custom-ubuntu1604:          except IndexError:
custom-ubuntu1604:     -        print "Provide an EBS volume ID to attach i.e. vol-cc789ea5"
custom-ubuntu1604:     +        print("Provide an EBS volume ID to attach i.e. vol-cc789ea5")
custom-ubuntu1604:              sys.exit(1)
custom-ubuntu1604:
custom-ubuntu1604:          # Get instance ID
custom-ubuntu1604:     -    instanceId = urllib2.urlopen("http://169.254.169.254/latest/meta-data/instance-id").read()
custom-ubuntu1604:     +    instanceId = requests.get("http://169.254.169.254/latest/meta-data/instance-id").text
custom-ubuntu1604:
custom-ubuntu1604:          # Get region
custom-ubuntu1604:     -    region = urllib2.urlopen("http://169.254.169.254/latest/meta-data/placement/availability-zone").read()
custom-ubuntu1604:     +    region = requests.get("http://169.254.169.254/latest/meta-data/placement/availability-zone").text
custom-ubuntu1604:          region = region[:-1]
custom-ubuntu1604:
custom-ubuntu1604:          # Generate a list of system paths minus the root path
custom-ubuntu1604:     -    paths = [convert_dev(device.path) for device in parted.getAllDevices()]
custom-ubuntu1604:     +    paths = [convert_dev(device) for device in get_all_devices()]
custom-ubuntu1604:
custom-ubuntu1604:          # List of possible block devices
custom-ubuntu1604:          blockDevices = ['/dev/sdb', '/dev/sdc', '/dev/sdd', '/dev/sde', '/dev/sdf', '/dev/sdg', '/dev/sdh',
custom-ubuntu1604:     @@ -52,13 +62,13 @@
custom-ubuntu1604:          availableDevices = [a for a in blockDevices if a not in paths]
custom-ubuntu1604:
custom-ubuntu1604:          # Parse configuration file to read proxy settings
custom-ubuntu1604:     -    config = ConfigParser.RawConfigParser()
custom-ubuntu1604:     +    config = configparser.RawConfigParser()
custom-ubuntu1604:          config.read('/etc/boto.cfg')
custom-ubuntu1604:          proxy_config = Config()
custom-ubuntu1604:          if config.has_option('Boto', 'proxy') and config.has_option('Boto', 'proxy_port'):
custom-ubuntu1604:              proxy = config.get('Boto', 'proxy')
custom-ubuntu1604:              proxy_port = config.get('Boto', 'proxy_port')
custom-ubuntu1604:     -        proxy_config = Config(proxies={'https': "{}:{}".format(proxy, proxy_port)})
custom-ubuntu1604:     +        proxy_config = Config(proxies={'https': "{0}:{1}".format(proxy, proxy_port)})
custom-ubuntu1604:
custom-ubuntu1604:          # Connect to AWS using boto
custom-ubuntu1604:          ec2 = boto3.client('ec2', region_name=region, config=proxy_config)
custom-ubuntu1604:     @@ -72,12 +82,12 @@
custom-ubuntu1604:          x = 0
custom-ubuntu1604:          while state != "attached":
custom-ubuntu1604:              if x == 36:
custom-ubuntu1604:     -            print "Volume %s failed to mount in 180 seconds." % volumeId
custom-ubuntu1604:     +            print("Volume %s failed to mount in 180 seconds." % volumeId)
custom-ubuntu1604:                  exit(1)
custom-ubuntu1604:              if state in ["busy" or "detached"]:
custom-ubuntu1604:     -            print "Volume %s in bad state %s" % (volumeId, state)
custom-ubuntu1604:     +            print("Volume %s in bad state %s" % (volumeId, state))
custom-ubuntu1604:                  exit(1)
custom-ubuntu1604:     -        print "Volume %s in state %s ... waiting to be 'attached'" % (volumeId, state)
custom-ubuntu1604:     +        print("Volume %s in state %s ... waiting to be 'attached'" % (volumeId, state))
custom-ubuntu1604:              time.sleep(5)
custom-ubuntu1604:              x += 1
custom-ubuntu1604:              try:
custom-ubuntu1604:   * service[ec2blkdev] action enable (up to date)
custom-ubuntu1604:   * service[ec2blkdev] action start (up to date)
custom-ubuntu1604: Recipe: aws-parallelcluster::base_install
custom-ubuntu1604:   * remote_file[/usr/bin/ec2-metadata] action create (up to date)
custom-ubuntu1604:   * pyenv_pip[aws-parallelcluster-node] action install
custom-ubuntu1604:     * pyenv_script[aws-parallelcluster-node] action run
custom-ubuntu1604:       * bash[aws-parallelcluster-node] action run
custom-ubuntu1604:         - execute "bash" -e "/tmp/chef-script20190909-2937-15xs7t3"
custom-ubuntu1604:
custom-ubuntu1604:
custom-ubuntu1604:   * cookbook_file[supervisord.conf] action create (up to date)
custom-ubuntu1604:   * template[supervisord-init] action create
custom-ubuntu1604:     - update content in file /etc/init.d/supervisord from 543a51 to beec0b
custom-ubuntu1604:     --- /etc/init.d/supervisord  2018-11-16 15:02:03.851985138 +0000
custom-ubuntu1604:     +++ /etc/init.d/.chef-supervisord20190909-2937-ojhn6v    2019-09-09 16:00:10.871148239 +0000
custom-ubuntu1604:     @@ -28,8 +28,8 @@
custom-ubuntu1604:      . /lib/lsb/init-functions
custom-ubuntu1604:
custom-ubuntu1604:      PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
custom-ubuntu1604:     -DAEMON=/usr/local/bin/supervisord
custom-ubuntu1604:     -SUPERVISORCTL=/usr/local/bin/supervisorctl
custom-ubuntu1604:     +DAEMON="/root/.pyenv/versions/3.6.9/envs/cookbook_virtualenv/bin/supervisord"
custom-ubuntu1604:     +SUPERVISORCTL="/root/.pyenv/versions/3.6.9/envs/cookbook_virtualenv/bin/supervisorctl"
custom-ubuntu1604:      NAME=supervisord
custom-ubuntu1604:      DESC=supervisor
custom-ubuntu1604:
custom-ubuntu1604:   * cookbook_file[ami_cleanup.sh] action create
custom-ubuntu1604:     - update content in file /usr/local/sbin/ami_cleanup.sh from ae6cdd to b64d46
custom-ubuntu1604:     --- /usr/local/sbin/ami_cleanup.sh   2018-11-16 15:02:03.863980394 +0000
custom-ubuntu1604:     +++ /usr/local/sbin/.chef-ami_cleanup20190909-2937-1g9v5zi.sh    2019-09-09 16:00:10.883147860 +0000
custom-ubuntu1604:     @@ -5,6 +5,14 @@
custom-ubuntu1604:      rm -f /var/lib/cloud/instance
custom-ubuntu1604:      rm -rf /etc/ssh/ssh_host_*
custom-ubuntu1604:      rm -f /etc/udev/rules.d/70-persistent-net.rules
custom-ubuntu1604:     +grep -l "Created by cloud-init on instance boot automatically" /etc/sysconfig/network-scripts/ifcfg-* | xargs rm -f
custom-ubuntu1604:     +
custom-ubuntu1604:     +# https://bugs.centos.org/view.php?id=13836#c33128
custom-ubuntu1604:     +source /etc/os-release
custom-ubuntu1604:     +if [ "${ID}${VERSION_ID}" == "centos7" ]; then
custom-ubuntu1604:     +    rm -f /etc/sysconfig/network-scripts/ifcfg-eth0
custom-ubuntu1604:     +fi
custom-ubuntu1604:     +
custom-ubuntu1604:      find /var/log -type f -exec /bin/rm -v {} \;
custom-ubuntu1604:      touch /var/log/lastlog
custom-ubuntu1604: Recipe: aws-parallelcluster::_efa_install
custom-ubuntu1604:   * remote_file[/opt/parallelcluster/sources/aws-efa-installer-latest.tar.gz] action create
custom-ubuntu1604:     - create new file /opt/parallelcluster/sources/aws-efa-installer-latest.tar.gz
custom-ubuntu1604:     - update content in file /opt/parallelcluster/sources/aws-efa-installer-latest.tar.gz from none to c59122
custom-ubuntu1604:     (file sizes exceed 10000000 bytes, diff output suppressed)
custom-ubuntu1604:     - change mode from '' to '0644'
custom-ubuntu1604:   * apt_package[libopenmpi-dev] action remove
custom-ubuntu1604:     - remove package libopenmpi-dev
custom-ubuntu1604:   * bash[install efa] action run
custom-ubuntu1604:
custom-ubuntu1604:     ================================================================================
custom-ubuntu1604:     Error executing action `run` on resource 'bash[install efa]'
custom-ubuntu1604:     ================================================================================
custom-ubuntu1604:
custom-ubuntu1604:     Mixlib::ShellOut::ShellCommandFailed
custom-ubuntu1604:     ------------------------------------
custom-ubuntu1604:     Expected process to exit with [0], but received '1'
custom-ubuntu1604:     ---- Begin output of "bash"  "/tmp/chef-script20190909-2937-3rauqg" ----
custom-ubuntu1604:     STDOUT: = Starting Amazon Elastic Fabric Adapter Installation Script =
custom-ubuntu1604:     = EFA Installer Version: 1.5.1 =
custom-ubuntu1604:
custom-ubuntu1604:     == Installing EFA dependencies ==
custom-ubuntu1604:     Get:1 file:/var/cuda-repo-10-1-local-10.1.243-418.87.00  InRelease
custom-ubuntu1604:     Ign:1 file:/var/cuda-repo-10-1-local-10.1.243-418.87.00  InRelease
custom-ubuntu1604:     Get:2 file:/var/cuda-repo-10-1-local-10.1.243-418.87.00  Release [574 B]
custom-ubuntu1604:     Get:2 file:/var/cuda-repo-10-1-local-10.1.243-418.87.00  Release [574 B]
custom-ubuntu1604:     Hit:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial InRelease
custom-ubuntu1604:     Hit:4 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates InRelease
custom-ubuntu1604:     Hit:5 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-backports InRelease
custom-ubuntu1604:     Hit:7 http://security.ubuntu.com/ubuntu xenial-security InRelease
custom-ubuntu1604:     Reading package lists...
custom-ubuntu1604:     Building dependency tree...
custom-ubuntu1604:     Reading state information...
custom-ubuntu1604:     185 packages can be upgraded. Run 'apt list --upgradable' to see them.
custom-ubuntu1604:     Reading package lists...
custom-ubuntu1604:     Building dependency tree...
custom-ubuntu1604:     Reading state information...
custom-ubuntu1604:     tcl is already the newest version (8.6.0+9).
custom-ubuntu1604:     tcl set to manually installed.
custom-ubuntu1604:     libnl-3-200 is already the newest version (3.2.27-1ubuntu0.16.04.1).
custom-ubuntu1604:     libnl-3-200 set to manually installed.
custom-ubuntu1604:     linux-headers-4.4.0-1070-aws is already the newest version (4.4.0-1070.80).
custom-ubuntu1604:     The following packages were automatically installed and are no longer required:
custom-ubuntu1604:       libibverbs-dev libibverbs1 libopenmpi1.10 openmpi-bin openmpi-common
custom-ubuntu1604:     Use 'sudo apt autoremove' to remove them.
custom-ubuntu1604:     The following additional packages will be installed:
custom-ubuntu1604:       libpci3
custom-ubuntu1604:     The following NEW packages will be installed:
custom-ubuntu1604:       environment-modules libnl-3-dev libnl-route-3-200 libnl-route-3-dev
custom-ubuntu1604:     The following packages will be upgraded:
custom-ubuntu1604:       dkms libpci3 pciutils
custom-ubuntu1604:     3 upgraded, 4 newly installed, 0 to remove and 182 not upgraded.
custom-ubuntu1604:     Need to get 764 kB of archives.
custom-ubuntu1604:     After this operation, 2,287 kB of additional disk space will be used.
custom-ubuntu1604:     Get:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 pciutils amd64 1:3.3.1-1.1ubuntu1.3 [254 kB]
custom-ubuntu1604:     Get:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpci3 amd64 1:3.3.1-1.1ubuntu1.3 [24.3 kB]
custom-ubuntu1604:     Get:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 dkms all 2.2.0.3-2ubuntu11.8 [66.4 kB]
custom-ubuntu1604:     Get:4 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial/universe amd64 environment-modules amd64 3.2.10-9 [77.0 kB]
custom-ubuntu1604:     Get:5 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libnl-3-dev amd64 3.2.27-1ubuntu0.16.04.1 [85.8 kB]
custom-ubuntu1604:     Get:6 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libnl-route-3-200 amd64 3.2.27-1ubuntu0.16.04.1 [124 kB]
custom-ubuntu1604:     Get:7 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libnl-route-3-dev amd64 3.2.27-1ubuntu0.16.04.1 [133 kB]
custom-ubuntu1604:     Fetched 764 kB in 0s (1,369 kB/s)
custom-ubuntu1604: (Reading database ... 151003 files and directories currently installed.)
custom-ubuntu1604:     Preparing to unpack .../pciutils_1%3a3.3.1-1.1ubuntu1.3_amd64.deb ...
custom-ubuntu1604:     Unpacking pciutils (1:3.3.1-1.1ubuntu1.3) over (1:3.3.1-1.1ubuntu1.2) ...
custom-ubuntu1604:     Preparing to unpack .../libpci3_1%3a3.3.1-1.1ubuntu1.3_amd64.deb ...
custom-ubuntu1604:     Unpacking libpci3:amd64 (1:3.3.1-1.1ubuntu1.3) over (1:3.3.1-1.1ubuntu1.2) ...
custom-ubuntu1604:     Preparing to unpack .../dkms_2.2.0.3-2ubuntu11.8_all.deb ...
custom-ubuntu1604:     Unpacking dkms (2.2.0.3-2ubuntu11.8) over (2.2.0.3-2ubuntu11.5) ...
custom-ubuntu1604:     Selecting previously unselected package environment-modules.
custom-ubuntu1604:     Preparing to unpack .../environment-modules_3.2.10-9_amd64.deb ...
custom-ubuntu1604:     Unpacking environment-modules (3.2.10-9) ...
custom-ubuntu1604:     Selecting previously unselected package libnl-3-dev:amd64.
custom-ubuntu1604:     Preparing to unpack .../libnl-3-dev_3.2.27-1ubuntu0.16.04.1_amd64.deb ...
custom-ubuntu1604:     Unpacking libnl-3-dev:amd64 (3.2.27-1ubuntu0.16.04.1) ...
custom-ubuntu1604:     Selecting previously unselected package libnl-route-3-200:amd64.
custom-ubuntu1604:     Preparing to unpack .../libnl-route-3-200_3.2.27-1ubuntu0.16.04.1_amd64.deb ...
custom-ubuntu1604:     Unpacking libnl-route-3-200:amd64 (3.2.27-1ubuntu0.16.04.1) ...
custom-ubuntu1604:     Selecting previously unselected package libnl-route-3-dev:amd64.
custom-ubuntu1604:     Preparing to unpack .../libnl-route-3-dev_3.2.27-1ubuntu0.16.04.1_amd64.deb ...
custom-ubuntu1604:     Unpacking libnl-route-3-dev:amd64 (3.2.27-1ubuntu0.16.04.1) ...
custom-ubuntu1604:     Processing triggers for man-db (2.7.5-1) ...
custom-ubuntu1604:     Processing triggers for libc-bin (2.23-0ubuntu10) ...
custom-ubuntu1604:     Setting up libpci3:amd64 (1:3.3.1-1.1ubuntu1.3) ...
custom-ubuntu1604:     Setting up pciutils (1:3.3.1-1.1ubuntu1.3) ...
custom-ubuntu1604:     Setting up dkms (2.2.0.3-2ubuntu11.8) ...
custom-ubuntu1604:     Installing new version of config file /etc/kernel/header_postinst.d/dkms ...
custom-ubuntu1604:     Installing new version of config file /etc/kernel/postinst.d/dkms ...
custom-ubuntu1604:     Setting up environment-modules (3.2.10-9) ...
custom-ubuntu1604:     Setting up libnl-3-dev:amd64 (3.2.27-1ubuntu0.16.04.1) ...
custom-ubuntu1604:     Setting up libnl-route-3-200:amd64 (3.2.27-1ubuntu0.16.04.1) ...
custom-ubuntu1604:     Setting up libnl-route-3-dev:amd64 (3.2.27-1ubuntu0.16.04.1) ...
custom-ubuntu1604:     Processing triggers for libc-bin (2.23-0ubuntu10) ...
custom-ubuntu1604:     == Installing EFA packages ==
custom-ubuntu1604:     Installing ./DEBS/UBUNTU/efa_1.3.0-1.amzn1_amd64.deb
custom-ubuntu1604:     Reading package lists...
custom-ubuntu1604:     Building dependency tree...
custom-ubuntu1604:     Reading state information...
custom-ubuntu1604:     The following packages were automatically installed and are no longer required:
custom-ubuntu1604:       libopenmpi1.10 openmpi-bin openmpi-common
custom-ubuntu1604:     Use 'sudo apt autoremove' to remove them.
custom-ubuntu1604:     The following NEW packages will be installed:
custom-ubuntu1604:       efa ibacm ibverbs-providers ibverbs-utils infiniband-diags libfabric-bin
custom-ubuntu1604:       libfabric-dev libfabric1 libibmad-dev libibmad5 libibnetdisc-dev
custom-ubuntu1604:       libibnetdisc5 libibumad-dev libibumad3 librdmacm-dev librdmacm1 openmpi
custom-ubuntu1604:       rdma-core rdmacm-utils
custom-ubuntu1604:     The following packages will be upgraded:
custom-ubuntu1604:       libibverbs-dev libibverbs1
custom-ubuntu1604:     2 upgraded, 19 newly installed, 0 to remove and 182 not upgraded.
custom-ubuntu1604:     Need to get 0 B/5,500 kB of archives.
custom-ubuntu1604:     After this operation, 22.4 MB of additional disk space will be used.
custom-ubuntu1604:     Get:1 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/rdma-core_25.0-1_amd64.deb rdma-core amd64 25.0-1 [61.4 kB]
custom-ubuntu1604:     Get:2 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libibumad3_25.0-1_amd64.deb libibumad3 amd64 25.0-1 [24.9 kB]
custom-ubuntu1604:     Get:3 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/ibverbs-providers_25.0-1_amd64.deb ibverbs-providers amd64 25.0-1 [219 kB]
custom-ubuntu1604:     Get:4 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libibverbs-dev_25.0-1_amd64.deb libibverbs-dev amd64 25.0-1 [428 kB]
custom-ubuntu1604:     Get:5 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libibverbs1_25.0-1_amd64.deb libibverbs1 amd64 25.0-1 [52.6 kB]
custom-ubuntu1604:     Get:6 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/ibacm_25.0-1_amd64.deb ibacm amd64 25.0-1 [79.1 kB]
custom-ubuntu1604:     Get:7 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/ibverbs-utils_25.0-1_amd64.deb ibverbs-utils amd64 25.0-1 [52.7 kB]
custom-ubuntu1604:     Get:8 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libibmad5_25.0-1_amd64.deb libibmad5 amd64 25.0-1 [38.9 kB]
custom-ubuntu1604:     Get:9 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libibnetdisc5_25.0-1_amd64.deb libibnetdisc5 amd64 25.0-1 [31.3 kB]
custom-ubuntu1604:     Get:10 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/infiniband-diags_25.0-1_amd64.deb infiniband-diags amd64 25.0-1 [216 kB]
custom-ubuntu1604:     Get:11 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libibmad-dev_25.0-1_amd64.deb libibmad-dev amd64 25.0-1 [47.9 kB]
custom-ubuntu1604:     Get:12 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libibnetdisc-dev_25.0-1_amd64.deb libibnetdisc-dev amd64 25.0-1 [39.9 kB]
custom-ubuntu1604:     Get:13 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libibumad-dev_25.0-1_amd64.deb libibumad-dev amd64 25.0-1 [53.2 kB]
custom-ubuntu1604:     Get:14 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/librdmacm1_25.0-1_amd64.deb librdmacm1 amd64 25.0-1 [60.3 kB]
custom-ubuntu1604:     Get:15 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/librdmacm-dev_25.0-1_amd64.deb librdmacm-dev amd64 25.0-1 [114 kB]
custom-ubuntu1604:     Get:16 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libfabric1_1.8.0amzn1.0_amd64.deb libfabric1 amd64 1.8.0amzn1.0 [315 kB]
custom-ubuntu1604:     Get:17 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/openmpi_3.1.4-2_amd64.deb openmpi amd64 3.1.4-2 [3,086 kB]
custom-ubuntu1604:     Get:18 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/rdmacm-utils_25.0-1_amd64.deb rdmacm-utils amd64 25.0-1 [68.6 kB]
custom-ubuntu1604:     Get:19 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/efa_1.3.0-1.amzn1_amd64.deb efa amd64 1.3.0-1.amzn1 [38.8 kB]
custom-ubuntu1604:     Get:20 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libfabric-bin_1.8.0amzn1.0_amd64.deb libfabric-bin amd64 1.8.0amzn1.0 [26.7 kB]
custom-ubuntu1604:     Get:21 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libfabric-dev_1.8.0amzn1.0_amd64.deb libfabric-dev amd64 1.8.0amzn1.0 [446 kB]
custom-ubuntu1604:     Selecting previously unselected package rdma-core.
custom-ubuntu1604: (Reading database ... 151185 files and directories currently installed.)
custom-ubuntu1604:     Preparing to unpack .../rdma-core_25.0-1_amd64.deb ...
custom-ubuntu1604:     Unpacking rdma-core (25.0-1) ...
custom-ubuntu1604:     Selecting previously unselected package libibumad3:amd64.
custom-ubuntu1604:     Preparing to unpack .../libibumad3_25.0-1_amd64.deb ...
custom-ubuntu1604:     Unpacking libibumad3:amd64 (25.0-1) ...
custom-ubuntu1604:     Selecting previously unselected package ibverbs-providers:amd64.
custom-ubuntu1604:     Preparing to unpack .../ibverbs-providers_25.0-1_amd64.deb ...
custom-ubuntu1604:     Unpacking ibverbs-providers:amd64 (25.0-1) ...
custom-ubuntu1604:     Preparing to unpack .../libibverbs-dev_25.0-1_amd64.deb ...
custom-ubuntu1604:     Unpacking libibverbs-dev:amd64 (25.0-1) over (1.1.8-1.1ubuntu2) ...
custom-ubuntu1604:     Preparing to unpack .../libibverbs1_25.0-1_amd64.deb ...
custom-ubuntu1604:     Unpacking libibverbs1:amd64 (25.0-1) over (1.1.8-1.1ubuntu2) ...
custom-ubuntu1604:     Selecting previously unselected package ibacm.
custom-ubuntu1604:     Preparing to unpack .../UBUNTU/ibacm_25.0-1_amd64.deb ...
custom-ubuntu1604:     Unpacking ibacm (25.0-1) ...
custom-ubuntu1604:     Selecting previously unselected package ibverbs-utils.
custom-ubuntu1604:     Preparing to unpack .../ibverbs-utils_25.0-1_amd64.deb ...
custom-ubuntu1604:     Unpacking ibverbs-utils (25.0-1) ...
custom-ubuntu1604:     Selecting previously unselected package libibmad5.
custom-ubuntu1604:     Preparing to unpack .../libibmad5_25.0-1_amd64.deb ...
custom-ubuntu1604:     Unpacking libibmad5 (25.0-1) ...
custom-ubuntu1604:     Selecting previously unselected package libibnetdisc5.
custom-ubuntu1604:     Preparing to unpack .../libibnetdisc5_25.0-1_amd64.deb ...
custom-ubuntu1604:     Unpacking libibnetdisc5 (25.0-1) ...
custom-ubuntu1604:     Selecting previously unselected package infiniband-diags.
custom-ubuntu1604:     Preparing to unpack .../infiniband-diags_25.0-1_amd64.deb ...
custom-ubuntu1604:     Unpacking infiniband-diags (25.0-1) ...
custom-ubuntu1604:     Selecting previously unselected package libibmad-dev.
custom-ubuntu1604:     Preparing to unpack .../libibmad-dev_25.0-1_amd64.deb ...
custom-ubuntu1604:     Unpacking libibmad-dev (25.0-1) ...
custom-ubuntu1604:     Selecting previously unselected package libibnetdisc-dev.
custom-ubuntu1604:     Preparing to unpack .../libibnetdisc-dev_25.0-1_amd64.deb ...
custom-ubuntu1604:     Unpacking libibnetdisc-dev (25.0-1) ...
custom-ubuntu1604:     Selecting previously unselected package libibumad-dev:amd64.
custom-ubuntu1604:     Preparing to unpack .../libibumad-dev_25.0-1_amd64.deb ...
custom-ubuntu1604:     Unpacking libibumad-dev:amd64 (25.0-1) ...
custom-ubuntu1604:     Selecting previously unselected package librdmacm1:amd64.
custom-ubuntu1604:     Preparing to unpack .../librdmacm1_25.0-1_amd64.deb ...
custom-ubuntu1604:     Unpacking librdmacm1:amd64 (25.0-1) ...
custom-ubuntu1604:     Selecting previously unselected package librdmacm-dev:amd64.
custom-ubuntu1604:     Preparing to unpack .../librdmacm-dev_25.0-1_amd64.deb ...
custom-ubuntu1604:     Unpacking librdmacm-dev:amd64 (25.0-1) ...
custom-ubuntu1604:     Selecting previously unselected package libfabric1.
custom-ubuntu1604:     Preparing to unpack .../libfabric1_1.8.0amzn1.0_amd64.deb ...
custom-ubuntu1604:     Unpacking libfabric1 (1.8.0amzn1.0) ...
custom-ubuntu1604:     Selecting previously unselected package openmpi.
custom-ubuntu1604:     Preparing to unpack .../openmpi_3.1.4-2_amd64.deb ...
custom-ubuntu1604:     Unpacking openmpi (3.1.4-2) ...
custom-ubuntu1604:     Selecting previously unselected package rdmacm-utils.
custom-ubuntu1604:     Preparing to unpack .../rdmacm-utils_25.0-1_amd64.deb ...
custom-ubuntu1604:     Unpacking rdmacm-utils (25.0-1) ...
custom-ubuntu1604:     Selecting previously unselected package efa.
custom-ubuntu1604:     Preparing to unpack .../efa_1.3.0-1.amzn1_amd64.deb ...
custom-ubuntu1604:     Unpacking efa (1.3.0-1.amzn1) ...
custom-ubuntu1604:     Selecting previously unselected package libfabric-bin.
custom-ubuntu1604:     Preparing to unpack .../libfabric-bin_1.8.0amzn1.0_amd64.deb ...
custom-ubuntu1604:     Unpacking libfabric-bin (1.8.0amzn1.0) ...
custom-ubuntu1604:     Selecting previously unselected package libfabric-dev.
custom-ubuntu1604:     Preparing to unpack .../libfabric-dev_1.8.0amzn1.0_amd64.deb ...
custom-ubuntu1604:     Unpacking libfabric-dev (1.8.0amzn1.0) ...
custom-ubuntu1604:     Processing triggers for man-db (2.7.5-1) ...
custom-ubuntu1604:     Processing triggers for systemd (229-4ubuntu21.8) ...
custom-ubuntu1604:     Processing triggers for ureadahead (0.100.0-19) ...
custom-ubuntu1604:     Processing triggers for libc-bin (2.23-0ubuntu10) ...
custom-ubuntu1604:     Setting up rdma-core (25.0-1) ...
custom-ubuntu1604:     rdma-hw.target is a disabled or a static unit, not starting it.
custom-ubuntu1604:     rdma-ndd.service is a disabled or a static unit, not starting it.
custom-ubuntu1604:     Setting up libibumad3:amd64 (25.0-1) ...
custom-ubuntu1604:     Setting up libibverbs1:amd64 (25.0-1) ...
custom-ubuntu1604:     Setting up ibverbs-providers:amd64 (25.0-1) ...
custom-ubuntu1604:     Setting up libibverbs-dev:amd64 (25.0-1) ...
custom-ubuntu1604:     Setting up ibacm (25.0-1) ...
custom-ubuntu1604:     Job for ibacm.service failed because the control process exited with error code. See "systemctl status ibacm.service" and "journalctl -xe" for details.
custom-ubuntu1604:     ibacm.service couldn't start.
custom-ubuntu1604:     Setting up ibverbs-utils (25.0-1) ...
custom-ubuntu1604:     Setting up libibmad5 (25.0-1) ...
custom-ubuntu1604:     Setting up libibnetdisc5 (25.0-1) ...
custom-ubuntu1604:     Setting up infiniband-diags (25.0-1) ...
custom-ubuntu1604:     Setting up libibmad-dev (25.0-1) ...
custom-ubuntu1604:     Setting up libibnetdisc-dev (25.0-1) ...
custom-ubuntu1604:     Setting up libibumad-dev:amd64 (25.0-1) ...
custom-ubuntu1604:     Setting up librdmacm1:amd64 (25.0-1) ...
custom-ubuntu1604:     Setting up librdmacm-dev:amd64 (25.0-1) ...
custom-ubuntu1604:     Setting up libfabric1 (1.8.0amzn1.0) ...
custom-ubuntu1604:     Setting up openmpi (3.1.4-2) ...
custom-ubuntu1604:     Setting up rdmacm-utils (25.0-1) ...
custom-ubuntu1604:     Setting up efa (1.3.0-1.amzn1) ...
custom-ubuntu1604:
custom-ubuntu1604:     Creating symlink /var/lib/dkms/efa/1.3.0/source ->
custom-ubuntu1604:                      /usr/src/efa-1.3.0
custom-ubuntu1604:
custom-ubuntu1604:     DKMS: add completed.
custom-ubuntu1604:
custom-ubuntu1604:     Kernel preparation unnecessary for this kernel.  Skipping...
custom-ubuntu1604:
custom-ubuntu1604:     Building module:
custom-ubuntu1604:     cleaning build area....(bad exit status: 2)
custom-ubuntu1604:     make KERNELRELEASE=4.4.0-1070-aws KERNEL_VERSION=4.4.0-1070-aws.......
custom-ubuntu1604:     cleaning build area....(bad exit status: 2)
custom-ubuntu1604:
custom-ubuntu1604:     DKMS: build completed.
custom-ubuntu1604:
custom-ubuntu1604:     efa:
custom-ubuntu1604:     Running module version sanity check.
custom-ubuntu1604:      - Original module
custom-ubuntu1604:        - No original module exists within this kernel
custom-ubuntu1604:      - Installation
custom-ubuntu1604:        - Installing to /lib/modules/4.4.0-1070-aws/updates/dkms/
custom-ubuntu1604:
custom-ubuntu1604:     depmod......
custom-ubuntu1604:
custom-ubuntu1604:     DKMS: install completed.
custom-ubuntu1604:     Setting up libfabric-bin (1.8.0amzn1.0) ...
custom-ubuntu1604:     Setting up libfabric-dev (1.8.0amzn1.0) ...
custom-ubuntu1604:     Processing triggers for systemd (229-4ubuntu21.8) ...
custom-ubuntu1604:     Processing triggers for ureadahead (0.100.0-19) ...
custom-ubuntu1604:     Processing triggers for libc-bin (2.23-0ubuntu10) ...
custom-ubuntu1604:     == Writing EFA profile.d configuration ==
custom-ubuntu1604:     == Writing /etc/ld.so.conf.d/efa.conf ==
custom-ubuntu1604:     == Kernel version checking ==
custom-ubuntu1604:     Your Ubuntu AMI launches with a kernel which does not include the ib_uverbs kernel module, which is necessary for EFA support.
custom-ubuntu1604:     Canonical has published a new kernel in the Ubuntu 16.04 repositories which includes the ib_ubverbs kernel module.
custom-ubuntu1604:
custom-ubuntu1604:     Please run the following commands to upgrade the kernel. Note that the system will be rebooted after kernel upgrade!!
custom-ubuntu1604:     $ sudo DEBIAN_FRONTEND=noninteractive apt-get -y --with-new-pkgs -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" upgrade
custom-ubuntu1604:     $ sudo reboot
custom-ubuntu1604:     After reboot, you can run ./efa_test.sh to locally test EFA connectivity with fi_pingpong test.
custom-ubuntu1604:     STDERR: WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
custom-ubuntu1604:
custom-ubuntu1604:
custom-ubuntu1604:     WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
custom-ubuntu1604:     ---- End output of "bash"  "/tmp/chef-script20190909-2937-3rauqg" ----
custom-ubuntu1604:     Ran "bash"  "/tmp/chef-script20190909-2937-3rauqg" returned 1
custom-ubuntu1604:
custom-ubuntu1604:     Resource Declaration:
custom-ubuntu1604:     ---------------------
custom-ubuntu1604:     # In /tmp/packer-chef-solo/local-mode-cache/cache/cookbooks/aws-parallelcluster/recipes/_efa_install.rb
custom-ubuntu1604:
custom-ubuntu1604:      42: bash "install efa" do
custom-ubuntu1604:      43:   cwd node['cfncluster']['sources_dir']
custom-ubuntu1604:      44:   code <<-EFAINSTALL
custom-ubuntu1604:      45:     set -e
custom-ubuntu1604:      46:     tar -xzf #{efa_tarball}
custom-ubuntu1604:      47:     cd aws-efa-installer
custom-ubuntu1604:      48:     ./efa_installer.sh -y --skip-limit-conf
custom-ubuntu1604:      49:   EFAINSTALL
custom-ubuntu1604:      50:   not_if { ::Dir.exist?('/opt/amazon/efa') }
custom-ubuntu1604:      51: end
custom-ubuntu1604:
custom-ubuntu1604:     Compiled Resource:
custom-ubuntu1604:     ------------------
custom-ubuntu1604:     # Declared in /tmp/packer-chef-solo/local-mode-cache/cache/cookbooks/aws-parallelcluster/recipes/_efa_install.rb:42:in `from_file'
custom-ubuntu1604:
custom-ubuntu1604:     bash("install efa") do
custom-ubuntu1604:       action [:run]
custom-ubuntu1604:       default_guard_interpreter :default
custom-ubuntu1604:       command nil
custom-ubuntu1604:       backup 5
custom-ubuntu1604:       interpreter "bash"
custom-ubuntu1604:       declared_type :bash
custom-ubuntu1604:       cookbook_name "aws-parallelcluster"
custom-ubuntu1604:       recipe_name "_efa_install"
custom-ubuntu1604:       cwd "/opt/parallelcluster/sources"
custom-ubuntu1604:       code "    set -e\n    tar -xzf /opt/parallelcluster/sources/aws-efa-installer-latest.tar.gz\n    cd aws-efa-installer\n    ./efa_installer.sh -y --skip-limit-conf\n"
custom-ubuntu1604:       domain nil
custom-ubuntu1604:       user nil
custom-ubuntu1604:       not_if { #code block }
custom-ubuntu1604:     end
custom-ubuntu1604:
custom-ubuntu1604:     System Info:
custom-ubuntu1604:     ------------
custom-ubuntu1604:     chef_version=14.2.0
custom-ubuntu1604:     platform=ubuntu
custom-ubuntu1604:     platform_version=16.04
custom-ubuntu1604:     ruby=ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux]
custom-ubuntu1604:     program_name=/usr/bin/chef-client
custom-ubuntu1604:     executable=/opt/chef/bin/chef-client
custom-ubuntu1604:
custom-ubuntu1604: Recipe: nfs::server
custom-ubuntu1604:   * service[nfs-kernel-server] action restart
custom-ubuntu1604:     - restart service service[nfs-kernel-server]
custom-ubuntu1604:
custom-ubuntu1604: Running handlers:
custom-ubuntu1604: [2019-09-09T16:01:04+00:00] ERROR: Running exception handlers
custom-ubuntu1604: Running handlers complete
custom-ubuntu1604: [2019-09-09T16:01:04+00:00] ERROR: Exception handlers complete
custom-ubuntu1604: Chef Client failed. 44 resources updated in 03 minutes 21 seconds
custom-ubuntu1604: [2019-09-09T16:01:04+00:00] FATAL: Stacktrace dumped to /tmp/packer-chef-solo/local-mode-cache/cache/chef-stacktrace.out
custom-ubuntu1604: [2019-09-09T16:01:04+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
custom-ubuntu1604: [2019-09-09T16:01:04+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: bash[install efa] (aws-parallelcluster::_efa_install line 42) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
custom-ubuntu1604: ---- Begin output of "bash"  "/tmp/chef-script20190909-2937-3rauqg" ----
custom-ubuntu1604: STDOUT: = Starting Amazon Elastic Fabric Adapter Installation Script =
custom-ubuntu1604: = EFA Installer Version: 1.5.1 =
custom-ubuntu1604:
custom-ubuntu1604: == Installing EFA dependencies ==
custom-ubuntu1604: Get:1 file:/var/cuda-repo-10-1-local-10.1.243-418.87.00  InRelease
custom-ubuntu1604: Ign:1 file:/var/cuda-repo-10-1-local-10.1.243-418.87.00  InRelease
custom-ubuntu1604: Get:2 file:/var/cuda-repo-10-1-local-10.1.243-418.87.00  Release [574 B]
custom-ubuntu1604: Get:2 file:/var/cuda-repo-10-1-local-10.1.243-418.87.00  Release [574 B]
custom-ubuntu1604: Hit:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial InRelease
custom-ubuntu1604: Hit:4 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates InRelease
custom-ubuntu1604: Hit:5 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-backports InRelease
custom-ubuntu1604: Hit:7 http://security.ubuntu.com/ubuntu xenial-security InRelease
custom-ubuntu1604: Reading package lists...
custom-ubuntu1604: Building dependency tree...
custom-ubuntu1604: Reading state information...
custom-ubuntu1604: 185 packages can be upgraded. Run 'apt list --upgradable' to see them.
custom-ubuntu1604: Reading package lists...
custom-ubuntu1604: Building dependency tree...
custom-ubuntu1604: Reading state information...
custom-ubuntu1604: tcl is already the newest version (8.6.0+9).
custom-ubuntu1604: tcl set to manually installed.
custom-ubuntu1604: libnl-3-200 is already the newest version (3.2.27-1ubuntu0.16.04.1).
custom-ubuntu1604: libnl-3-200 set to manually installed.
custom-ubuntu1604: linux-headers-4.4.0-1070-aws is already the newest version (4.4.0-1070.80).
custom-ubuntu1604: The following packages were automatically installed and are no longer required:
custom-ubuntu1604:   libibverbs-dev libibverbs1 libopenmpi1.10 openmpi-bin openmpi-common
custom-ubuntu1604: Use 'sudo apt autoremove' to remove them.
custom-ubuntu1604: The following additional packages will be installed:
custom-ubuntu1604:   libpci3
custom-ubuntu1604: The following NEW packages will be installed:
custom-ubuntu1604:   environment-modules libnl-3-dev libnl-route-3-200 libnl-route-3-dev
custom-ubuntu1604: The following packages will be upgraded:
custom-ubuntu1604:   dkms libpci3 pciutils
custom-ubuntu1604: 3 upgraded, 4 newly installed, 0 to remove and 182 not upgraded.
custom-ubuntu1604: Need to get 764 kB of archives.
custom-ubuntu1604: After this operation, 2,287 kB of additional disk space will be used.
custom-ubuntu1604: Get:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 pciutils amd64 1:3.3.1-1.1ubuntu1.3 [254 kB]
custom-ubuntu1604: Get:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpci3 amd64 1:3.3.1-1.1ubuntu1.3 [24.3 kB]
custom-ubuntu1604: Get:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 dkms all 2.2.0.3-2ubuntu11.8 [66.4 kB]
custom-ubuntu1604: Get:4 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial/universe amd64 environment-modules amd64 3.2.10-9 [77.0 kB]
custom-ubuntu1604: Get:5 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libnl-3-dev amd64 3.2.27-1ubuntu0.16.04.1 [85.8 kB]
custom-ubuntu1604: Get:6 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libnl-route-3-200 amd64 3.2.27-1ubuntu0.16.04.1 [124 kB]
custom-ubuntu1604: Get:7 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libnl-route-3-dev amd64 3.2.27-1ubuntu0.16.04.1 [133 kB]
custom-ubuntu1604: Fetched 764 kB in 0s (1,369 kB/s)
custom-ubuntu1604: (Reading database ... 151003 files and directories currently installed.)
custom-ubuntu1604: Preparing to unpack .../pciutils_1%3a3.3.1-1.1ubuntu1.3_amd64.deb ...
custom-ubuntu1604: Unpacking pciutils (1:3.3.1-1.1ubuntu1.3) over (1:3.3.1-1.1ubuntu1.2) ...
custom-ubuntu1604: Preparing to unpack .../libpci3_1%3a3.3.1-1.1ubuntu1.3_amd64.deb ...
custom-ubuntu1604: Unpacking libpci3:amd64 (1:3.3.1-1.1ubuntu1.3) over (1:3.3.1-1.1ubuntu1.2) ...
custom-ubuntu1604: Preparing to unpack .../dkms_2.2.0.3-2ubuntu11.8_all.deb ...
custom-ubuntu1604: Unpacking dkms (2.2.0.3-2ubuntu11.8) over (2.2.0.3-2ubuntu11.5) ...
custom-ubuntu1604: Selecting previously unselected package environment-modules.
custom-ubuntu1604: Preparing to unpack .../environment-modules_3.2.10-9_amd64.deb ...
custom-ubuntu1604: Unpacking environment-modules (3.2.10-9) ...
custom-ubuntu1604: Selecting previously unselected package libnl-3-dev:amd64.
custom-ubuntu1604: Preparing to unpack .../libnl-3-dev_3.2.27-1ubuntu0.16.04.1_amd64.deb ...
custom-ubuntu1604: Unpacking libnl-3-dev:amd64 (3.2.27-1ubuntu0.16.04.1) ...
custom-ubuntu1604: Selecting previously unselected package libnl-route-3-200:amd64.
custom-ubuntu1604: Preparing to unpack .../libnl-route-3-200_3.2.27-1ubuntu0.16.04.1_amd64.deb ...
custom-ubuntu1604: Unpacking libnl-route-3-200:amd64 (3.2.27-1ubuntu0.16.04.1) ...
custom-ubuntu1604: Selecting previously unselected package libnl-route-3-dev:amd64.
custom-ubuntu1604: Preparing to unpack .../libnl-route-3-dev_3.2.27-1ubuntu0.16.04.1_amd64.deb ...
custom-ubuntu1604: Unpacking libnl-route-3-dev:amd64 (3.2.27-1ubuntu0.16.04.1) ...
custom-ubuntu1604: Processing triggers for man-db (2.7.5-1) ...
custom-ubuntu1604: Processing triggers for libc-bin (2.23-0ubuntu10) ...
custom-ubuntu1604: Setting up libpci3:amd64 (1:3.3.1-1.1ubuntu1.3) ...
custom-ubuntu1604: Setting up pciutils (1:3.3.1-1.1ubuntu1.3) ...
custom-ubuntu1604: Setting up dkms (2.2.0.3-2ubuntu11.8) ...
custom-ubuntu1604: Installing new version of config file /etc/kernel/header_postinst.d/dkms ...
custom-ubuntu1604: Installing new version of config file /etc/kernel/postinst.d/dkms ...
custom-ubuntu1604: Setting up environment-modules (3.2.10-9) ...
custom-ubuntu1604: Setting up libnl-3-dev:amd64 (3.2.27-1ubuntu0.16.04.1) ...
custom-ubuntu1604: Setting up libnl-route-3-200:amd64 (3.2.27-1ubuntu0.16.04.1) ...
custom-ubuntu1604: Setting up libnl-route-3-dev:amd64 (3.2.27-1ubuntu0.16.04.1) ...
custom-ubuntu1604: Processing triggers for libc-bin (2.23-0ubuntu10) ...
custom-ubuntu1604: == Installing EFA packages ==
custom-ubuntu1604: Installing ./DEBS/UBUNTU/efa_1.3.0-1.amzn1_amd64.deb
custom-ubuntu1604: Reading package lists...
custom-ubuntu1604: Building dependency tree...
custom-ubuntu1604: Reading state information...
custom-ubuntu1604: The following packages were automatically installed and are no longer required:
custom-ubuntu1604:   libopenmpi1.10 openmpi-bin openmpi-common
custom-ubuntu1604: Use 'sudo apt autoremove' to remove them.
custom-ubuntu1604: The following NEW packages will be installed:
custom-ubuntu1604:   efa ibacm ibverbs-providers ibverbs-utils infiniband-diags libfabric-bin
custom-ubuntu1604:   libfabric-dev libfabric1 libibmad-dev libibmad5 libibnetdisc-dev
custom-ubuntu1604:   libibnetdisc5 libibumad-dev libibumad3 librdmacm-dev librdmacm1 openmpi
custom-ubuntu1604:   rdma-core rdmacm-utils
custom-ubuntu1604: The following packages will be upgraded:
custom-ubuntu1604:   libibverbs-dev libibverbs1
custom-ubuntu1604: 2 upgraded, 19 newly installed, 0 to remove and 182 not upgraded.
custom-ubuntu1604: Need to get 0 B/5,500 kB of archives.
custom-ubuntu1604: After this operation, 22.4 MB of additional disk space will be used.
custom-ubuntu1604: Get:1 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/rdma-core_25.0-1_amd64.deb rdma-core amd64 25.0-1 [61.4 kB]
custom-ubuntu1604: Get:2 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libibumad3_25.0-1_amd64.deb libibumad3 amd64 25.0-1 [24.9 kB]
custom-ubuntu1604: Get:3 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/ibverbs-providers_25.0-1_amd64.deb ibverbs-providers amd64 25.0-1 [219 kB]
custom-ubuntu1604: Get:4 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libibverbs-dev_25.0-1_amd64.deb libibverbs-dev amd64 25.0-1 [428 kB]
custom-ubuntu1604: Get:5 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libibverbs1_25.0-1_amd64.deb libibverbs1 amd64 25.0-1 [52.6 kB]
custom-ubuntu1604: Get:6 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/ibacm_25.0-1_amd64.deb ibacm amd64 25.0-1 [79.1 kB]
custom-ubuntu1604: Get:7 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/ibverbs-utils_25.0-1_amd64.deb ibverbs-utils amd64 25.0-1 [52.7 kB]
custom-ubuntu1604: Get:8 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libibmad5_25.0-1_amd64.deb libibmad5 amd64 25.0-1 [38.9 kB]
custom-ubuntu1604: Get:9 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libibnetdisc5_25.0-1_amd64.deb libibnetdisc5 amd64 25.0-1 [31.3 kB]
custom-ubuntu1604: Get:10 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/infiniband-diags_25.0-1_amd64.deb infiniband-diags amd64 25.0-1 [216 kB]
custom-ubuntu1604: Get:11 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libibmad-dev_25.0-1_amd64.deb libibmad-dev amd64 25.0-1 [47.9 kB]
custom-ubuntu1604: Get:12 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libibnetdisc-dev_25.0-1_amd64.deb libibnetdisc-dev amd64 25.0-1 [39.9 kB]
custom-ubuntu1604: Get:13 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libibumad-dev_25.0-1_amd64.deb libibumad-dev amd64 25.0-1 [53.2 kB]
custom-ubuntu1604: Get:14 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/librdmacm1_25.0-1_amd64.deb librdmacm1 amd64 25.0-1 [60.3 kB]
custom-ubuntu1604: Get:15 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/librdmacm-dev_25.0-1_amd64.deb librdmacm-dev amd64 25.0-1 [114 kB]
custom-ubuntu1604: Get:16 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libfabric1_1.8.0amzn1.0_amd64.deb libfabric1 amd64 1.8.0amzn1.0 [315 kB]
custom-ubuntu1604: Get:17 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/openmpi_3.1.4-2_amd64.deb openmpi amd64 3.1.4-2 [3,086 kB]
custom-ubuntu1604: Get:18 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/rdmacm-utils_25.0-1_amd64.deb rdmacm-utils amd64 25.0-1 [68.6 kB]
custom-ubuntu1604: Get:19 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/efa_1.3.0-1.amzn1_amd64.deb efa amd64 1.3.0-1.amzn1 [38.8 kB]
custom-ubuntu1604: Get:20 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libfabric-bin_1.8.0amzn1.0_amd64.deb libfabric-bin amd64 1.8.0amzn1.0 [26.7 kB]
custom-ubuntu1604: Get:21 /opt/parallelcluster/sources/aws-efa-installer/DEBS/UBUNTU/libfabric-dev_1.8.0amzn1.0_amd64.deb libfabric-dev amd64 1.8.0amzn1.0 [446 kB]
custom-ubuntu1604: Selecting previously unselected package rdma-core.
custom-ubuntu1604: (Reading database ... 151185 files and directories currently installed.)
custom-ubuntu1604: Preparing to unpack .../rdma-core_25.0-1_amd64.deb ...
custom-ubuntu1604: Unpacking rdma-core (25.0-1) ...
custom-ubuntu1604: Selecting previously unselected package libibumad3:amd64.
custom-ubuntu1604: Preparing to unpack .../libibumad3_25.0-1_amd64.deb ...
custom-ubuntu1604: Unpacking libibumad3:amd64 (25.0-1) ...
custom-ubuntu1604: Selecting previously unselected package ibverbs-providers:amd64.
custom-ubuntu1604: Preparing to unpack .../ibverbs-providers_25.0-1_amd64.deb ...
custom-ubuntu1604: Unpacking ibverbs-providers:amd64 (25.0-1) ...
custom-ubuntu1604: Preparing to unpack .../libibverbs-dev_25.0-1_amd64.deb ...
custom-ubuntu1604: Unpacking libibverbs-dev:amd64 (25.0-1) over (1.1.8-1.1ubuntu2) ...
custom-ubuntu1604: Preparing to unpack .../libibverbs1_25.0-1_amd64.deb ...
custom-ubuntu1604: Unpacking libibverbs1:amd64 (25.0-1) over (1.1.8-1.1ubuntu2) ...
custom-ubuntu1604: Selecting previously unselected package ibacm.
custom-ubuntu1604: Preparing to unpack .../UBUNTU/ibacm_25.0-1_amd64.deb ...
custom-ubuntu1604: Unpacking ibacm (25.0-1) ...
custom-ubuntu1604: Selecting previously unselected package ibverbs-utils.
custom-ubuntu1604: Preparing to unpack .../ibverbs-utils_25.0-1_amd64.deb ...
custom-ubuntu1604: Unpacking ibverbs-utils (25.0-1) ...
custom-ubuntu1604: Selecting previously unselected package libibmad5.
custom-ubuntu1604: Preparing to unpack .../libibmad5_25.0-1_amd64.deb ...
custom-ubuntu1604: Unpacking libibmad5 (25.0-1) ...
custom-ubuntu1604: Selecting previously unselected package libibnetdisc5.
custom-ubuntu1604: Preparing to unpack .../libibnetdisc5_25.0-1_amd64.deb ...
custom-ubuntu1604: Unpacking libibnetdisc5 (25.0-1) ...
custom-ubuntu1604: Selecting previously unselected package infiniband-diags.
custom-ubuntu1604: Preparing to unpack .../infiniband-diags_25.0-1_amd64.deb ...
custom-ubuntu1604: Unpacking infiniband-diags (25.0-1) ...
custom-ubuntu1604: Selecting previously unselected package libibmad-dev.
custom-ubuntu1604: Preparing to unpack .../libibmad-dev_25.0-1_amd64.deb ...
custom-ubuntu1604: Unpacking libibmad-dev (25.0-1) ...
custom-ubuntu1604: Selecting previously unselected package libibnetdisc-dev.
custom-ubuntu1604: Preparing to unpack .../libibnetdisc-dev_25.0-1_amd64.deb ...
custom-ubuntu1604: Unpacking libibnetdisc-dev (25.0-1) ...
custom-ubuntu1604: Selecting previously unselected package libibumad-dev:amd64.
custom-ubuntu1604: Preparing to unpack .../libibumad-dev_25.0-1_amd64.deb ...
custom-ubuntu1604: Unpacking libibumad-dev:amd64 (25.0-1) ...
custom-ubuntu1604: Selecting previously unselected package librdmacm1:amd64.
custom-ubuntu1604: Preparing to unpack .../librdmacm1_25.0-1_amd64.deb ...
custom-ubuntu1604: Unpacking librdmacm1:amd64 (25.0-1) ...
custom-ubuntu1604: Selecting previously unselected package librdmacm-dev:amd64.
custom-ubuntu1604: Preparing to unpack .../librdmacm-dev_25.0-1_amd64.deb ...
custom-ubuntu1604: Unpacking librdmacm-dev:amd64 (25.0-1) ...
custom-ubuntu1604: Selecting previously unselected package libfabric1.
custom-ubuntu1604: Preparing to unpack .../libfabric1_1.8.0amzn1.0_amd64.deb ...
custom-ubuntu1604: Unpacking libfabric1 (1.8.0amzn1.0) ...
custom-ubuntu1604: Selecting previously unselected package openmpi.
custom-ubuntu1604: Preparing to unpack .../openmpi_3.1.4-2_amd64.deb ...
custom-ubuntu1604: Unpacking openmpi (3.1.4-2) ...
custom-ubuntu1604: Selecting previously unselected package rdmacm-utils.
custom-ubuntu1604: Preparing to unpack .../rdmacm-utils_25.0-1_amd64.deb ...
custom-ubuntu1604: Unpacking rdmacm-utils (25.0-1) ...
custom-ubuntu1604: Selecting previously unselected package efa.
custom-ubuntu1604: Preparing to unpack .../efa_1.3.0-1.amzn1_amd64.deb ...
custom-ubuntu1604: Unpacking efa (1.3.0-1.amzn1) ...
custom-ubuntu1604: Selecting previously unselected package libfabric-bin.
custom-ubuntu1604: Preparing to unpack .../libfabric-bin_1.8.0amzn1.0_amd64.deb ...
custom-ubuntu1604: Unpacking libfabric-bin (1.8.0amzn1.0) ...
custom-ubuntu1604: Selecting previously unselected package libfabric-dev.
custom-ubuntu1604: Preparing to unpack .../libfabric-dev_1.8.0amzn1.0_amd64.deb ...
custom-ubuntu1604: Unpacking libfabric-dev (1.8.0amzn1.0) ...
custom-ubuntu1604: Processing triggers for man-db (2.7.5-1) ...
custom-ubuntu1604: Processing triggers for systemd (229-4ubuntu21.8) ...
custom-ubuntu1604: Processing triggers for ureadahead (0.100.0-19) ...
custom-ubuntu1604: Processing triggers for libc-bin (2.23-0ubuntu10) ...
custom-ubuntu1604: Setting up rdma-core (25.0-1) ...
custom-ubuntu1604: rdma-hw.target is a disabled or a static unit, not starting it.
custom-ubuntu1604: rdma-ndd.service is a disabled or a static unit, not starting it.
custom-ubuntu1604: Setting up libibumad3:amd64 (25.0-1) ...
custom-ubuntu1604: Setting up libibverbs1:amd64 (25.0-1) ...
custom-ubuntu1604: Setting up ibverbs-providers:amd64 (25.0-1) ...
custom-ubuntu1604: Setting up libibverbs-dev:amd64 (25.0-1) ...
custom-ubuntu1604: Setting up ibacm (25.0-1) ...
custom-ubuntu1604: Job for ibacm.service failed because the control process exited with error code. See "systemctl status ibacm.service" and "journalctl -xe" for details.
custom-ubuntu1604: ibacm.service couldn't start.
custom-ubuntu1604: Setting up ibverbs-utils (25.0-1) ...
custom-ubuntu1604: Setting up libibmad5 (25.0-1) ...
custom-ubuntu1604: Setting up libibnetdisc5 (25.0-1) ...
custom-ubuntu1604: Setting up infiniband-diags (25.0-1) ...
custom-ubuntu1604: Setting up libibmad-dev (25.0-1) ...
custom-ubuntu1604: Setting up libibnetdisc-dev (25.0-1) ...
custom-ubuntu1604: Setting up libibumad-dev:amd64 (25.0-1) ...
custom-ubuntu1604: Setting up librdmacm1:amd64 (25.0-1) ...
custom-ubuntu1604: Setting up librdmacm-dev:amd64 (25.0-1) ...
custom-ubuntu1604: Setting up libfabric1 (1.8.0amzn1.0) ...
custom-ubuntu1604: Setting up openmpi (3.1.4-2) ...
custom-ubuntu1604: Setting up rdmacm-utils (25.0-1) ...
custom-ubuntu1604: Setting up efa (1.3.0-1.amzn1) ...
custom-ubuntu1604:
custom-ubuntu1604: Creating symlink /var/lib/dkms/efa/1.3.0/source ->
custom-ubuntu1604:                  /usr/src/efa-1.3.0
custom-ubuntu1604:
custom-ubuntu1604: DKMS: add completed.
custom-ubuntu1604:
custom-ubuntu1604: Kernel preparation unnecessary for this kernel.  Skipping...
custom-ubuntu1604:
custom-ubuntu1604: Building module:
custom-ubuntu1604: cleaning build area....(bad exit status: 2)
custom-ubuntu1604: make KERNELRELEASE=4.4.0-1070-aws KERNEL_VERSION=4.4.0-1070-aws.......
custom-ubuntu1604: cleaning build area....(bad exit status: 2)
custom-ubuntu1604:
custom-ubuntu1604: DKMS: build completed.
custom-ubuntu1604:
custom-ubuntu1604: efa:
custom-ubuntu1604: Running module version sanity check.
custom-ubuntu1604:  - Original module
custom-ubuntu1604:    - No original module exists within this kernel
custom-ubuntu1604:  - Installation
custom-ubuntu1604:    - Installing to /lib/modules/4.4.0-1070-aws/updates/dkms/
custom-ubuntu1604:
custom-ubuntu1604: depmod......
custom-ubuntu1604:
custom-ubuntu1604: DKMS: install completed.
custom-ubuntu1604: Setting up libfabric-bin (1.8.0amzn1.0) ...
custom-ubuntu1604: Setting up libfabric-dev (1.8.0amzn1.0) ...
custom-ubuntu1604: Processing triggers for systemd (229-4ubuntu21.8) ...
custom-ubuntu1604: Processing triggers for ureadahead (0.100.0-19) ...
custom-ubuntu1604: Processing triggers for libc-bin (2.23-0ubuntu10) ...
custom-ubuntu1604: == Writing EFA profile.d configuration ==
custom-ubuntu1604: == Writing /etc/ld.so.conf.d/efa.conf ==
custom-ubuntu1604: == Kernel version checking ==
custom-ubuntu1604: Your Ubuntu AMI launches with a kernel which does not include the ib_uverbs kernel module, which is necessary for EFA support.
custom-ubuntu1604: Canonical has published a new kernel in the Ubuntu 16.04 repositories which includes the ib_ubverbs kernel module.
custom-ubuntu1604:
custom-ubuntu1604: Please run the following commands to upgrade the kernel. Note that the system will be rebooted after kernel upgrade!!
custom-ubuntu1604: $ sudo DEBIAN_FRONTEND=noninteractive apt-get -y --with-new-pkgs -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" upgrade
custom-ubuntu1604: $ sudo reboot
custom-ubuntu1604: After reboot, you can run ./efa_test.sh to locally test EFA connectivity with fi_pingpong test.
custom-ubuntu1604: STDERR: WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
custom-ubuntu1604:
custom-ubuntu1604:
custom-ubuntu1604: WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
custom-ubuntu1604: ---- End output of "bash"  "/tmp/chef-script20190909-2937-3rauqg" ----
custom-ubuntu1604: Ran "bash"  "/tmp/chef-script20190909-2937-3rauqg" returned 1
==> custom-ubuntu1604: Terminating the source AWS instance...
==> custom-ubuntu1604: Cleaning up any extra volumes...
==> custom-ubuntu1604: No volumes to clean up, skipping
==> custom-ubuntu1604: Deleting temporary security group...
==> custom-ubuntu1604: Deleting temporary keypair...
Build 'custom-ubuntu1604' errored: Error executing Chef: Non-zero exit status: 1

==> Some builds didn't complete successfully and had errors:
--> custom-ubuntu1604: Error executing Chef: Non-zero exit status: 1

==> Builds finished but no artifacts were created.
lukeseawalker commented 5 years ago

Hi @ssbotelh, the source AMI you are using is already a baked ParallelCluster AMI, I can see from the logs

custom-ubuntu1604:   * directory[/etc/parallelcluster] action create (up to date)
custom-ubuntu1604:   * directory[/opt/parallelcluster] action create (up to date)

and then it fails installing EFA

custom-ubuntu1604:     Error executing action `run` on resource 'bash[install efa]'

because

custom-ubuntu1604: Your Ubuntu AMI launches with a kernel which does not include the ib_uverbs kernel module, which is necessary for EFA support.
custom-ubuntu1604: Canonical has published a new kernel in the Ubuntu 16.04 repositories which includes the ib_ubverbs kernel module.
custom-ubuntu1604:
custom-ubuntu1604: Please run the following commands to upgrade the kernel. Note that the system will be rebooted after kernel upgrade!!
custom-ubuntu1604: $ sudo DEBIAN_FRONTEND=noninteractive apt-get -y --with-new-pkgs -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" upgrade
custom-ubuntu1604: $ sudo reboot
custom-ubuntu1604: After reboot, you can run ./efa_test

Before going further I would make sure you are starting from a NON ParallelCluster AMI, with an updated kernel version.

L

ssbotelh commented 5 years ago

Hi @lukeseawalker, Thanks again. Although I generated the base AMI without using pcluster, it seems like somewhere in its history pcluster was indeed used, and that apparently contaminates the whole lineage! Thanks for looking into it, and sorry for the false alarm. I'll go ahead and close the ticket.