net-ssh broken with AWS codebuild?

ChrisPetr0 commented 4 years ago

I know technically this may not be an issue with ami-spec. Essentially, I can run ami-spec fine locally with the exact same command I am running via AWS codebuild with Ruby 2.7 (and tried 2.6). I thought it was an IAM permissions issue, but using the --debug flag proves that is not the case (the EC2 instance stays running with --debug).

I'm getting this stack trace:

/root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/net-ssh-5.2.0/lib/net/ssh/transport/session.rb:92:in `rescue in initialize': Net::SSH::ConnectionTimeout (Net::SSH::ConnectionTimeout)
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/net-ssh-5.2.0/lib/net/ssh/transport/session.rb:59:in `initialize'
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/net-ssh-5.2.0/lib/net/ssh.rb:246:in `new'
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/net-ssh-5.2.0/lib/net/ssh.rb:246:in `start'
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/ami_spec-1.6.0/lib/ami_spec/wait_for_ssh.rb:11:in `wait'
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/ami_spec-1.6.0/lib/ami_spec.rb:91:in `block in run'
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/ami_spec-1.6.0/lib/ami_spec.rb:89:in `each'
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/ami_spec-1.6.0/lib/ami_spec.rb:89:in `run'
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/ami_spec-1.6.0/lib/ami_spec.rb:191:in `invoke'
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/ami_spec-1.6.0/bin/ami_spec:5:in `<top (required)>'
    from /root/.rbenv/versions/2.7.1/bin/ami_spec:23:in `load'
    from /root/.rbenv/versions/2.7.1/bin/ami_spec:23:in `<main>'
/root/.rbenv/versions/2.7.1/lib/ruby/2.7.0/socket.rb:61:in `connect_internal': Connection timed out - user specified timeout (Errno::ETIMEDOUT)
    from /root/.rbenv/versions/2.7.1/lib/ruby/2.7.0/socket.rb:137:in `connect'
    from /root/.rbenv/versions/2.7.1/lib/ruby/2.7.0/socket.rb:642:in `block in tcp'
    from /root/.rbenv/versions/2.7.1/lib/ruby/2.7.0/socket.rb:227:in `each'
    from /root/.rbenv/versions/2.7.1/lib/ruby/2.7.0/socket.rb:227:in `foreach'
    from /root/.rbenv/versions/2.7.1/lib/ruby/2.7.0/socket.rb:632:in `tcp'
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/net-ssh-5.2.0/lib/net/ssh/transport/session.rb:73:in `initialize'
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/net-ssh-5.2.0/lib/net/ssh.rb:246:in `new'
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/net-ssh-5.2.0/lib/net/ssh.rb:246:in `start'
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/ami_spec-1.6.0/lib/ami_spec/wait_for_ssh.rb:11:in `wait'
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/ami_spec-1.6.0/lib/ami_spec.rb:91:in `block in run'
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/ami_spec-1.6.0/lib/ami_spec.rb:89:in `each'
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/ami_spec-1.6.0/lib/ami_spec.rb:89:in `run'
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/ami_spec-1.6.0/lib/ami_spec.rb:191:in `invoke'
    from /root/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/ami_spec-1.6.0/bin/ami_spec:5:in `<top (required)>'
    from /root/.rbenv/versions/2.7.1/bin/ami_spec:23:in `load'
    from /root/.rbenv/versions/2.7.1/bin/ami_spec:23:in `<main>'

I wanted to try with a newer version of net-ssh gem but the spec is locked to version 5 and I see version 6 now available over at net-ssh.

It's definitely not security group related, and like I said, I can run the ami_spec command all day local from my Mac with great success.

I'm wondering if you, or anyone may have any idea what's going on here?

Thanks,

Chris

patrobinson commented 4 years ago

I dare say this is a network configuration error. Can you please supply your route table, network ACLs and security group configs?

ChrisPetr0 commented 4 years ago

Hi, thank you for the reply. Yes, it's really bizarre. I basically took all the measures to make sure it wasn't network.

NACLs are default config (no restriction)

I set the security group to allow 22 inbound from 0.0.0.0/0 (temporarily to debug, I know AWS publishes codebuild originating CIDR blocks)

Route Table is just the default for a typical 2 Public (local and IGW) 2 Private subnet (local and NAT) VPC.

In the ami_spec command, I have everything hardcoded right now (so there are no differences running locally from Mac versus via Codebuild from CodePipeline )

Something like this: (${SG_ID} is set ahead of time)

ami_spec --ami=ami-<truncated> --debug --role=ami --subnet-id=subnet-<truncated> --specs=spec --ssh-user=ubuntu --aws-region=us-east-1 --aws-security-groups=${SG_ID} --ssh-retries=5 --aws-public-ip

So verbatim, I can copy the command from my buildspec.yml file, run it just fine from my local Mac with IAM role (Admin), and then watch codebuild fail, running the same exact command (with an IAM role admin as well, to debug).

I have the whole pipeline coded up in Cloudformation, so this wasn't manually built. I'm just wondering if there is something about the managed service CodeBuild and the Ruby / net-ssh config but I see it's using rbenv so probably not. In addition, I tried using bundler with Gemfile and installing the gems natively to no avail.

What's interesting, is, it seems like ami_spec is failing pretty much right away and doing the API call to terminate the instance, but the SSH error doesn't populate on the Codebuild console for awhile. This is even with retries set to 100, it looks like ami_spec sends the API call terminate instance early. That's why I set it to --debug to see if the EC2 instance was getting terminated by some other means (no KMS permission for EBS, etc), and it's not.

Thanks,

Chris

patrobinson commented 4 years ago

Is CodeBuild running in the private or public subnet?

ChrisPetr0 commented 4 years ago

Hey, thanks for the insight. I didn't quite read all the documentation about codebuild and notice the VpcConfig parameter. I've played with it a bit, and still haven't had success. However, I think I am missing something. If you leave this issue open, I will keep this updated on what the problem was in case others need it.

ChrisPetr0 commented 4 years ago

Hey Patrick,

I've read the documentation and have setup CodeBuild with proper VPC, Subnets, and Security Group configuration. I've posted an example below where I'm running packer and ami_spec with the same env vars defined in the Cloudformation template, under the same build_spec.yml file. Packer is able to stand up EC2 instance in private subnet, use same security group, ssh into instance, and run simple chef config. I then use the ami created by packer, with the same private subnet and security group id, and get net-ssh ssh failures with ami_spec gem.

Here is the CodeBuild output with both packer ami build, and ami_spec call. It looks like it's failing immediately based on the times in the log.

2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:02 Waiting for agent ping
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:04 Waiting for DOWNLOAD_SOURCE
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:05 Phase is DOWNLOAD_SOURCE
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:05 CODEBUILD_SRC_DIR=/codebuild/output/src693153883/src
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:05 YAML location is /codebuild/output/src693153883/src/buildspec_sanity_check.yml
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:05 No commands found for phase name: install
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:05 Processing environment variables
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:05 Selecting 'ruby' runtime version '2.6' based on manual selections...
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:05 Running command echo "Installing Ruby version 2.6 ..."
    2020-10-09T14:10:10.039-04:00   Installing Ruby version 2.6 ...
    2020-10-09T14:10:10.039-04:00   
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:05 Running command rbenv global $RUBY_26_VERSION
    2020-10-09T14:10:10.039-04:00   
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:06 Moving to directory /codebuild/output/src693153883/src
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:06 Registering with agent
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:06 Phases found in YAML: 3
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:06 INSTALL: 0 commands
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:06 PRE_BUILD: 5 commands
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:06 BUILD: 8 commands
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:06 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:06 Phase context status code: Message:
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:06 Entering phase INSTALL
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:06 Phase complete: INSTALL State: SUCCEEDED
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:06 Phase context status code: Message:
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:06 Entering phase PRE_BUILD
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:06 Running command echo "Installing Packer"
    2020-10-09T14:10:10.039-04:00   Installing Packer
    2020-10-09T14:10:10.039-04:00   
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:06 Running command curl -o packer.zip https://releases.hashicorp.com/packer/1.6.1/packer_1.6.1_linux_amd64.zip && unzip packer.zip
    2020-10-09T14:10:10.039-04:00   % Total % Received % Xferd Average Speed Time Time Time Current
    2020-10-09T14:10:10.039-04:00   Dload Upload Total Spent Left Speed
    2020-10-09T14:10:10.039-04:00   0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 29.6M 100 29.6M 0 0 199M 0 --:--:-- --:--:-- --:--:-- 200M
    2020-10-09T14:10:10.039-04:00   Archive: packer.zip
    2020-10-09T14:10:10.039-04:00   inflating: packer
    2020-10-09T14:10:10.039-04:00   
    2020-10-09T14:10:10.039-04:00   [Container] 2020/10/09 18:10:08 Running command gem install bundler
    2020-10-09T14:10:16.084-04:00   Successfully installed bundler-2.1.4
    2020-10-09T14:10:16.084-04:00   Parsing documentation for bundler-2.1.4
    2020-10-09T14:10:16.084-04:00   Installing ri documentation for bundler-2.1.4
    2020-10-09T14:10:16.084-04:00   Done installing documentation for bundler after 3 seconds
    2020-10-09T14:10:16.084-04:00   1 gem installed
    2020-10-09T14:10:16.084-04:00   
    2020-10-09T14:10:16.084-04:00   [Container] 2020/10/09 18:10:14 Running command bundle install
    2020-10-09T14:10:16.084-04:00   Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
    2020-10-09T14:10:16.084-04:00   installing your bundle as root will break this application for all non-root
    2020-10-09T14:10:16.084-04:00   users on this machine.
    2020-10-09T14:10:16.084-04:00   Warning: the running version of Bundler (1.17.2) is older than the version that created the lockfile (1.17.3). We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.
    2020-10-09T14:10:20.115-04:00   Fetching gem metadata from https://rubygems.org/........
    2020-10-09T14:10:20.115-04:00   Fetching public_suffix 4.0.6
    2020-10-09T14:10:20.115-04:00   Installing public_suffix 4.0.6
    2020-10-09T14:10:20.115-04:00   Fetching addressable 2.7.0
    2020-10-09T14:10:20.115-04:00   Installing addressable 2.7.0
    2020-10-09T14:10:20.115-04:00   Fetching aws-eventstream 1.1.0
    2020-10-09T14:10:20.115-04:00   Installing aws-eventstream 1.1.0
    2020-10-09T14:10:20.115-04:00   Fetching aws-partitions 1.380.0
    2020-10-09T14:10:20.115-04:00   Installing aws-partitions 1.380.0
    2020-10-09T14:10:20.115-04:00   Fetching aws-sigv4 1.2.2
    2020-10-09T14:10:20.115-04:00   Installing aws-sigv4 1.2.2
    2020-10-09T14:10:20.115-04:00   Fetching jmespath 1.4.0
    2020-10-09T14:10:20.115-04:00   Installing jmespath 1.4.0
    2020-10-09T14:10:20.115-04:00   Fetching aws-sdk-core 3.109.1
    2020-10-09T14:10:20.115-04:00   Installing aws-sdk-core 3.109.1
    2020-10-09T14:10:20.115-04:00   Fetching aws-sdk-ec2 1.199.0
    2020-10-09T14:10:20.115-04:00   Installing aws-sdk-ec2 1.199.0
    2020-10-09T14:10:20.115-04:00   Fetching hashie 4.1.0
    2020-10-09T14:10:20.115-04:00   Installing hashie 4.1.0
    2020-10-09T14:10:20.115-04:00   Fetching net-ssh 5.2.0
    2020-10-09T14:10:20.115-04:00   Installing net-ssh 5.2.0
    2020-10-09T14:10:20.115-04:00   Fetching optimist 3.0.1
    2020-10-09T14:10:20.115-04:00   Installing optimist 3.0.1
    2020-10-09T14:10:20.115-04:00   Fetching multi_json 1.15.0
    2020-10-09T14:10:20.115-04:00   Installing multi_json 1.15.0
    2020-10-09T14:10:20.115-04:00   Fetching rspec-support 3.9.3
    2020-10-09T14:10:20.115-04:00   Installing rspec-support 3.9.3
    2020-10-09T14:10:20.115-04:00   Fetching rspec-core 3.9.3
    2020-10-09T14:10:20.115-04:00   Installing rspec-core 3.9.3
    2020-10-09T14:10:20.115-04:00   Fetching diff-lcs 1.3
    2020-10-09T14:10:20.115-04:00   Installing diff-lcs 1.3
    2020-10-09T14:10:20.115-04:00   Fetching rspec-expectations 3.9.2
    2020-10-09T14:10:20.115-04:00   Installing rspec-expectations 3.9.2
    2020-10-09T14:10:20.115-04:00   Fetching rspec-mocks 3.9.1
    2020-10-09T14:10:20.115-04:00   Installing rspec-mocks 3.9.1
    2020-10-09T14:10:20.115-04:00   Fetching rspec 3.9.0
    2020-10-09T14:10:20.115-04:00   Installing rspec 3.9.0
    2020-10-09T14:10:20.115-04:00   Fetching rspec-its 1.3.0
    2020-10-09T14:10:20.115-04:00   Installing rspec-its 1.3.0
    2020-10-09T14:10:20.115-04:00   Fetching net-scp 3.0.0
    2020-10-09T14:10:20.115-04:00   Installing net-scp 3.0.0
    2020-10-09T14:10:20.115-04:00   Fetching net-telnet 0.1.1
    2020-10-09T14:10:20.115-04:00   Installing net-telnet 0.1.1
    2020-10-09T14:10:20.115-04:00   Fetching sfl 2.3
    2020-10-09T14:10:20.115-04:00   Installing sfl 2.3
    2020-10-09T14:10:20.115-04:00   Fetching specinfra 2.82.19
    2020-10-09T14:10:20.115-04:00   Installing specinfra 2.82.19
    2020-10-09T14:10:22.156-04:00   Fetching serverspec 2.41.5
    2020-10-09T14:10:22.156-04:00   Installing serverspec 2.41.5
    2020-10-09T14:10:22.156-04:00   Fetching ami_spec 1.6.0
    2020-10-09T14:10:22.156-04:00   Installing ami_spec 1.6.0
    2020-10-09T14:10:22.156-04:00   Fetching aws-sdk-cloudformation 1.44.0
    2020-10-09T14:10:22.156-04:00   Installing aws-sdk-cloudformation 1.44.0
    2020-10-09T14:10:22.156-04:00   Fetching bcrypt_pbkdf 1.1.0.rc1
    2020-10-09T14:10:22.156-04:00   Installing bcrypt_pbkdf 1.1.0.rc1 with native extensions
    2020-10-09T14:10:26.174-04:00   Using bundler 1.17.2
    2020-10-09T14:10:26.174-04:00   Fetching chef-utils 16.5.77
    2020-10-09T14:10:26.174-04:00   Installing chef-utils 16.5.77
    2020-10-09T14:10:26.174-04:00   Fetching fuzzyurl 0.9.0
    2020-10-09T14:10:26.174-04:00   Installing fuzzyurl 0.9.0
    2020-10-09T14:10:26.174-04:00   Fetching tomlrb 1.3.0
    2020-10-09T14:10:26.174-04:00   Installing tomlrb 1.3.0
    2020-10-09T14:10:26.174-04:00   Fetching mixlib-config 3.0.9
    2020-10-09T14:10:26.174-04:00   Installing mixlib-config 3.0.9
    2020-10-09T14:10:26.174-04:00   Fetching mixlib-shellout 3.1.6
    2020-10-09T14:10:26.174-04:00   Installing mixlib-shellout 3.1.6
    2020-10-09T14:10:26.174-04:00   Fetching chef-config 16.5.77
    2020-10-09T14:10:26.174-04:00   Installing chef-config 16.5.77
    2020-10-09T14:10:26.174-04:00   Fetching chef-vault 4.0.12
    2020-10-09T14:10:26.174-04:00   Installing chef-vault 4.0.12
    2020-10-09T14:10:26.174-04:00   Fetching libyajl2 1.2.0
    2020-10-09T14:10:26.174-04:00   Installing libyajl2 1.2.0 with native extensions
    2020-10-09T14:10:26.174-04:00   Fetching ffi-yajl 2.3.4
    2020-10-09T14:10:26.174-04:00   Installing ffi-yajl 2.3.4 with native extensions
    2020-10-09T14:10:28.196-04:00   Fetching mixlib-log 3.0.9
    2020-10-09T14:10:28.196-04:00   Installing mixlib-log 3.0.9
    2020-10-09T14:10:28.196-04:00   Fetching rack 2.2.3
    2020-10-09T14:10:28.196-04:00   Installing rack 2.2.3
    2020-10-09T14:10:28.196-04:00   Fetching uuidtools 2.1.5
    2020-10-09T14:10:28.196-04:00   Installing uuidtools 2.1.5
    2020-10-09T14:10:28.196-04:00   Fetching chef-zero 15.0.3
    2020-10-09T14:10:28.196-04:00   Installing chef-zero 15.0.3
    2020-10-09T14:10:28.196-04:00   Fetching ed25519 1.2.4
    2020-10-09T14:10:28.196-04:00   Installing ed25519 1.2.4 with native extensions
    2020-10-09T14:10:30.218-04:00   Fetching erubis 2.7.0
    2020-10-09T14:10:30.218-04:00   Installing erubis 2.7.0
    2020-10-09T14:10:30.218-04:00   Fetching ffi 1.13.1
    2020-10-09T14:10:30.218-04:00   Installing ffi 1.13.1 with native extensions
    2020-10-09T14:10:36.237-04:00   Fetching ffi-libarchive 1.0.4
    2020-10-09T14:10:36.237-04:00   Installing ffi-libarchive 1.0.4
    2020-10-09T14:10:36.237-04:00   Fetching highline 2.0.3
    2020-10-09T14:10:36.237-04:00   Installing highline 2.0.3
    2020-10-09T14:10:36.237-04:00   Fetching iniparse 1.5.0
    2020-10-09T14:10:36.237-04:00   Installing iniparse 1.5.0
    2020-10-09T14:10:36.237-04:00   Fetching tty-color 0.5.2
    2020-10-09T14:10:36.237-04:00   Installing tty-color 0.5.2
    2020-10-09T14:10:36.237-04:00   Fetching pastel 0.8.0
    2020-10-09T14:10:36.237-04:00   Installing pastel 0.8.0
    2020-10-09T14:10:36.237-04:00   Fetching strings-ansi 0.2.0
    2020-10-09T14:10:36.237-04:00   Installing strings-ansi 0.2.0
    2020-10-09T14:10:36.237-04:00   Fetching unicode-display_width 1.7.0
    2020-10-09T14:10:36.237-04:00   Installing unicode-display_width 1.7.0
    2020-10-09T14:10:36.237-04:00   Fetching unicode_utils 1.4.0
    2020-10-09T14:10:36.237-04:00   Installing unicode_utils 1.4.0
    2020-10-09T14:10:36.237-04:00   Fetching strings 0.2.0
    2020-10-09T14:10:36.237-04:00   Installing strings 0.2.0
    2020-10-09T14:10:36.237-04:00   Fetching tty-cursor 0.7.1
    2020-10-09T14:10:36.237-04:00   Installing tty-cursor 0.7.1
    2020-10-09T14:10:36.237-04:00   Fetching tty-box 0.6.0
    2020-10-09T14:10:36.237-04:00   Installing tty-box 0.6.0
    2020-10-09T14:10:36.237-04:00   Fetching tty-screen 0.8.1
    2020-10-09T14:10:36.237-04:00   Installing tty-screen 0.8.1
    2020-10-09T14:10:36.237-04:00   Fetching wisper 2.0.1
    2020-10-09T14:10:36.237-04:00   Installing wisper 2.0.1
    2020-10-09T14:10:36.237-04:00   Fetching tty-reader 0.8.0
    2020-10-09T14:10:36.237-04:00   Installing tty-reader 0.8.0
    2020-10-09T14:10:36.237-04:00   Fetching tty-prompt 0.22.0
    2020-10-09T14:10:36.237-04:00   Installing tty-prompt 0.22.0
    2020-10-09T14:10:36.237-04:00   Fetching license-acceptance 2.1.2
    2020-10-09T14:10:36.237-04:00   Installing license-acceptance 2.1.2
    2020-10-09T14:10:36.237-04:00   Fetching mixlib-archive 1.0.7
    2020-10-09T14:10:36.237-04:00   Installing mixlib-archive 1.0.7
    2020-10-09T14:10:36.237-04:00   Fetching mixlib-authentication 3.0.7
    2020-10-09T14:10:36.237-04:00   Installing mixlib-authentication 3.0.7
    2020-10-09T14:10:36.237-04:00   Fetching mixlib-cli 2.1.8
    2020-10-09T14:10:36.237-04:00   Installing mixlib-cli 2.1.8
    2020-10-09T14:10:36.237-04:00   Fetching net-sftp 3.0.0
    2020-10-09T14:10:36.237-04:00   Installing net-sftp 3.0.0
    2020-10-09T14:10:36.237-04:00   Fetching net-ssh-gateway 2.0.0
    2020-10-09T14:10:36.237-04:00   Installing net-ssh-gateway 2.0.0
    2020-10-09T14:10:36.237-04:00   Fetching net-ssh-multi 1.2.1
    2020-10-09T14:10:36.237-04:00   Installing net-ssh-multi 1.2.1
    2020-10-09T14:10:36.237-04:00   Fetching ipaddress 0.8.3
    2020-10-09T14:10:36.237-04:00   Installing ipaddress 0.8.3
    2020-10-09T14:10:36.237-04:00   Fetching plist 3.5.0
    2020-10-09T14:10:36.237-04:00   Installing plist 3.5.0
    2020-10-09T14:10:36.237-04:00   Fetching wmi-lite 1.0.5
    2020-10-09T14:10:36.237-04:00   Installing wmi-lite 1.0.5
    2020-10-09T14:10:36.237-04:00   Fetching ohai 16.5.6
    2020-10-09T14:10:36.237-04:00   Installing ohai 16.5.6
    2020-10-09T14:10:38.261-04:00   Fetching proxifier 1.0.3
    2020-10-09T14:10:38.261-04:00   Installing proxifier 1.0.3
    2020-10-09T14:10:38.261-04:00   Fetching syslog-logger 1.6.8
    2020-10-09T14:10:38.261-04:00   Installing syslog-logger 1.6.8
    2020-10-09T14:10:38.261-04:00   Fetching json 2.3.1
    2020-10-09T14:10:38.261-04:00   Installing json 2.3.1 with native extensions
    2020-10-09T14:10:38.261-04:00   Fetching train-core 3.3.24
    2020-10-09T14:10:38.261-04:00   Installing train-core 3.3.24
    2020-10-09T14:10:38.261-04:00   Fetching builder 3.2.4
    2020-10-09T14:10:38.261-04:00   Installing builder 3.2.4
    2020-10-09T14:10:40.352-04:00   Fetching erubi 1.9.0
    2020-10-09T14:10:40.352-04:00   Installing erubi 1.9.0
    2020-10-09T14:10:40.352-04:00   Fetching gssapi 1.3.0
    2020-10-09T14:10:40.352-04:00   Installing gssapi 1.3.0
    2020-10-09T14:10:40.352-04:00   Fetching gyoku 1.3.1
    2020-10-09T14:10:40.352-04:00   Installing gyoku 1.3.1
    2020-10-09T14:10:40.352-04:00   Fetching httpclient 2.8.3
    2020-10-09T14:10:40.352-04:00   Installing httpclient 2.8.3
    2020-10-09T14:10:40.352-04:00   Fetching little-plugger 1.1.4
    2020-10-09T14:10:40.352-04:00   Installing little-plugger 1.1.4
    2020-10-09T14:10:40.352-04:00   Fetching logging 2.3.0
    2020-10-09T14:10:40.352-04:00   Installing logging 2.3.0
    2020-10-09T14:10:40.352-04:00   Fetching nori 2.6.0
    2020-10-09T14:10:40.352-04:00   Installing nori 2.6.0
    2020-10-09T14:10:40.352-04:00   Fetching rubyntlm 0.6.2
    2020-10-09T14:10:40.352-04:00   Installing rubyntlm 0.6.2
    2020-10-09T14:10:40.352-04:00   Fetching winrm 2.3.4
    2020-10-09T14:10:40.352-04:00   Installing winrm 2.3.4
    2020-10-09T14:10:40.352-04:00   Fetching rubyzip 2.3.0
    2020-10-09T14:10:40.352-04:00   Installing rubyzip 2.3.0
    2020-10-09T14:10:40.352-04:00   Fetching winrm-fs 1.3.5
    2020-10-09T14:10:40.352-04:00   Installing winrm-fs 1.3.5
    2020-10-09T14:10:40.352-04:00   Fetching winrm-elevated 1.2.2
    2020-10-09T14:10:40.352-04:00   Installing winrm-elevated 1.2.2
    2020-10-09T14:10:40.352-04:00   Fetching train-winrm 0.2.11
    2020-10-09T14:10:40.352-04:00   Installing train-winrm 0.2.11
    2020-10-09T14:10:40.352-04:00   Fetching tty-table 0.12.0
    2020-10-09T14:10:40.352-04:00   Installing tty-table 0.12.0
    2020-10-09T14:10:40.352-04:00   Fetching chef 16.5.77
    2020-10-09T14:10:40.352-04:00   Installing chef 16.5.77
    2020-10-09T14:10:42.386-04:00   Fetching cleanroom 1.0.0
    2020-10-09T14:10:42.386-04:00   Installing cleanroom 1.0.0
    2020-10-09T14:10:42.386-04:00   Fetching concurrent-ruby 1.1.7
    2020-10-09T14:10:42.386-04:00   Installing concurrent-ruby 1.1.7
    2020-10-09T14:10:42.386-04:00   Fetching minitar 0.9
    2020-10-09T14:10:42.386-04:00   Installing minitar 0.9
    2020-10-09T14:10:42.386-04:00   Fetching multipart-post 2.1.1
    2020-10-09T14:10:42.386-04:00   Installing multipart-post 2.1.1
    2020-10-09T14:10:42.386-04:00   Fetching faraday 1.0.1
    2020-10-09T14:10:42.386-04:00   Installing faraday 1.0.1
    2020-10-09T14:10:42.386-04:00   Fetching sawyer 0.8.2
    2020-10-09T14:10:42.386-04:00   Installing sawyer 0.8.2
    2020-10-09T14:10:42.386-04:00   Fetching octokit 4.18.0
    2020-10-09T14:10:42.386-04:00   Installing octokit 4.18.0
    2020-10-09T14:10:42.386-04:00   Fetching retryable 3.0.5
    2020-10-09T14:10:42.386-04:00   Installing retryable 3.0.5
    2020-10-09T14:10:42.386-04:00   Fetching molinillo 0.6.6
    2020-10-09T14:10:42.386-04:00   Installing molinillo 0.6.6
    2020-10-09T14:10:42.386-04:00   Fetching semverse 3.0.0
    2020-10-09T14:10:42.386-04:00   Installing semverse 3.0.0
    2020-10-09T14:10:42.386-04:00   Fetching solve 4.0.4
    2020-10-09T14:10:42.386-04:00   Installing solve 4.0.4
    2020-10-09T14:10:42.386-04:00   Fetching thor 1.0.1
    2020-10-09T14:10:42.386-04:00   Installing thor 1.0.1
    2020-10-09T14:10:42.386-04:00   Fetching berkshelf 7.1.0
    2020-10-09T14:10:42.386-04:00   Installing berkshelf 7.1.0
    2020-10-09T14:10:42.386-04:00   Fetching yajl-ruby 1.4.1
    2020-10-09T14:10:42.386-04:00   Installing yajl-ruby 1.4.1 with native extensions
    2020-10-09T14:10:44.409-04:00   Fetching json2yaml 0.0.3
    2020-10-09T14:10:44.409-04:00   Installing json2yaml 0.0.3
    2020-10-09T14:10:44.409-04:00   Bundle complete! 4 Gemfile dependencies, 105 gems now installed.
    2020-10-09T14:10:44.409-04:00   Use `bundle info [gemname]` to see where a bundled gem is installed.
    2020-10-09T14:10:44.409-04:00   Post-install message from minitar:
    2020-10-09T14:10:44.409-04:00   The `minitar` executable is no longer bundled with `minitar`. If you are
    2020-10-09T14:10:44.409-04:00   expecting this executable, make sure you also install `minitar-cli`.
    2020-10-09T14:10:44.409-04:00   
    2020-10-09T14:10:44.409-04:00   [Container] 2020/10/09 18:10:43 Running command bundle exec yaml2json pipeline/ubuntu_16_ami/packer/packer.yml > pipeline/ubuntu_16_ami/packer/packer.json
    2020-10-09T14:10:44.409-04:00   
    2020-10-09T14:10:44.409-04:00   [Container] 2020/10/09 18:10:43 Phase complete: PRE_BUILD State: SUCCEEDED
    2020-10-09T14:10:44.409-04:00   [Container] 2020/10/09 18:10:43 Phase context status code: Message:
    2020-10-09T14:10:44.409-04:00   [Container] 2020/10/09 18:10:43 Entering phase BUILD
    2020-10-09T14:10:44.409-04:00   [Container] 2020/10/09 18:10:43 Running command DATE=$(date +%d-%b-%Y"-"%H_%M_%S%Z)
    2020-10-09T14:10:44.409-04:00   
    2020-10-09T14:10:44.409-04:00   [Container] 2020/10/09 18:10:43 Running command ./packer build \
    2020-10-09T14:10:44.409-04:00   -var ami_name=ubuntu16_ami_${DATE} \
    2020-10-09T14:10:44.409-04:00   -var sg_1=${PACKER_SEC_GRP_ID} \
    2020-10-09T14:10:44.409-04:00   -var source_ami=${PACKER_SOURCE_AMI} \
    2020-10-09T14:10:44.409-04:00   -var subnet_id=${PACKER_SUBNET_ID} \
    2020-10-09T14:10:44.409-04:00   -var vpc_id=${VPC_ID} \
    2020-10-09T14:10:44.409-04:00   -var instance_profile=CLIENT_REMOVEDPackerInstanceProfile \
    2020-10-09T14:10:44.409-04:00   -color=false pipeline/ubuntu_16_ami/packer/packer.json | tee build.log
    2020-10-09T14:10:44.409-04:00   
    2020-10-09T14:10:44.409-04:00   ==> amazon-ebs: Prevalidating any provided VPC information
    2020-10-09T14:10:44.409-04:00   ==> amazon-ebs: Prevalidating AMI Name: ubuntu16_ami_09-Oct-2020-18_10_43UTC
    2020-10-09T14:10:46.430-04:00   amazon-ebs: Found Image ID: ami-0f82752aa17ff8f5d
    2020-10-09T14:10:46.430-04:00   ==> amazon-ebs: Creating temporary keypair: packer_5f80a7a4-dad5-c561-828e-4c24ed18b355
    2020-10-09T14:10:46.430-04:00   ==> amazon-ebs: Launching a source AWS instance...
    2020-10-09T14:10:46.430-04:00   ==> amazon-ebs: Adding tags to source instance
    2020-10-09T14:10:46.430-04:00   amazon-ebs: Adding tag: "ECS:Environment": ""
    2020-10-09T14:10:46.430-04:00   amazon-ebs: Adding tag: "Name": "Packer Builder"
    2020-10-09T14:10:48.451-04:00   amazon-ebs: Instance ID: i-0497be1e67c330233
    2020-10-09T14:10:48.451-04:00   ==> amazon-ebs: Waiting for instance (i-0497be1e67c330233) to become ready...
    2020-10-09T14:11:02.476-04:00   ==> amazon-ebs: Using ssh communicator to connect: 10.127.XXX.XX
    2020-10-09T14:11:02.476-04:00   ==> amazon-ebs: Waiting for SSH to become available...
    2020-10-09T14:11:22.513-04:00   ==> amazon-ebs: Connected to SSH!
    2020-10-09T14:11:22.513-04:00   ==> amazon-ebs: Provisioning with chef-solo
    2020-10-09T14:11:22.513-04:00   amazon-ebs: Installing Chef...
    2020-10-09T14:11:24.556-04:00   ==> amazon-ebs: % Total % Received % Xferd Average Speed Time Time Time Current
    2020-10-09T14:11:24.556-04:00   ==> amazon-ebs: Dload Upload Total Spent Left Speed
    2020-10-09T14:11:24.556-04:00   ==> amazon-ebs: 100 23547 100 23547 0 0 81559 0 --:--:-- --:--:-- --:--:-- 81760
    2020-10-09T14:11:24.556-04:00   amazon-ebs: ubuntu 16.04 x86_64
    2020-10-09T14:11:24.556-04:00   amazon-ebs: Getting information for chef stable for ubuntu...
    2020-10-09T14:11:24.556-04:00   amazon-ebs: downloading https://omnitruck.chef.io/stable/chef/metadata?v=&p=ubuntu&pv=16.04&m=x86_64
    2020-10-09T14:11:24.556-04:00   amazon-ebs: to file /tmp/install.sh.1392/metadata.txt
    2020-10-09T14:11:24.556-04:00   amazon-ebs: trying wget...
    2020-10-09T14:11:24.556-04:00   amazon-ebs: sha1 029fe3b3091ca6e5d0d117e4589c4bd1c8ff4ad6
    2020-10-09T14:11:24.556-04:00   amazon-ebs: sha256 b841bd480f7fd24a93317446d5e86e6114fc8635767a28f6c76dc4a740a7ac40
    2020-10-09T14:11:24.556-04:00   amazon-ebs: url https://packages.chef.io/files/stable/chef/16.5.77/ubuntu/16.04/chef_16.5.77-1_amd64.deb
    2020-10-09T14:11:24.556-04:00   amazon-ebs: version 16.5.77
    2020-10-09T14:11:24.556-04:00   amazon-ebs: downloaded metadata file looks valid...
    2020-10-09T14:11:24.556-04:00   amazon-ebs: downloading https://packages.chef.io/files/stable/chef/16.5.77/ubuntu/16.04/chef_16.5.77-1_amd64.deb
    2020-10-09T14:11:24.556-04:00   amazon-ebs: to file /tmp/install.sh.1392/chef_16.5.77-1_amd64.deb
    2020-10-09T14:11:24.556-04:00   amazon-ebs: trying wget...
    2020-10-09T14:11:24.556-04:00   amazon-ebs: Comparing checksum with sha256sum...
    2020-10-09T14:11:24.556-04:00   amazon-ebs:
    2020-10-09T14:11:24.556-04:00   amazon-ebs: WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
    2020-10-09T14:11:24.556-04:00   amazon-ebs:
    2020-10-09T14:11:24.556-04:00   amazon-ebs: You are installing a package without a version pin. If you are installing
    2020-10-09T14:11:24.556-04:00   amazon-ebs: on production servers via an automated process this is DANGEROUS and you will
    2020-10-09T14:11:24.556-04:00   amazon-ebs: be upgraded without warning on new releases, even to new major releases.
    2020-10-09T14:11:24.556-04:00   amazon-ebs: Letting the version float is only appropriate in desktop, test, development or
    2020-10-09T14:11:24.556-04:00   amazon-ebs: CI/CD environments.
    2020-10-09T14:11:24.556-04:00   amazon-ebs:
    2020-10-09T14:11:24.556-04:00   amazon-ebs: WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
    2020-10-09T14:11:24.556-04:00   amazon-ebs:
    2020-10-09T14:11:24.556-04:00   amazon-ebs: Installing chef
    2020-10-09T14:11:24.556-04:00   amazon-ebs: installing with dpkg...
    2020-10-09T14:11:24.556-04:00   amazon-ebs: Selecting previously unselected package chef.
    2020-10-09T14:11:24.556-04:00   amazon-ebs: (Reading database ... 51447 files and directories currently installed.)
    2020-10-09T14:11:24.556-04:00   amazon-ebs: Preparing to unpack .../chef_16.5.77-1_amd64.deb ...
    2020-10-09T14:11:24.556-04:00   amazon-ebs: Unpacking chef (16.5.77-1) ...
    2020-10-09T14:11:28.595-04:00   amazon-ebs: Setting up chef (16.5.77-1) ...
    2020-10-09T14:11:28.595-04:00   amazon-ebs: Thank you for installing Chef Infra Client! For help getting started visit https://learn.chef.io
    2020-10-09T14:11:28.595-04:00   amazon-ebs: Creating directory: /tmp/packer-chef-solo
    2020-10-09T14:11:28.595-04:00   amazon-ebs: Creating directory: /tmp/packer-chef-solo/cookbooks-0
    2020-10-09T14:11:28.595-04:00   amazon-ebs: Creating configuration file 'solo.rb'
    2020-10-09T14:11:28.595-04:00   amazon-ebs: Creating JSON attribute file
    2020-10-09T14:11:28.595-04:00   amazon-ebs: Executing Chef: sudo chef-solo --no-color -c /tmp/packer-chef-solo/solo.rb -j /tmp/packer-chef-solo/node.json
    2020-10-09T14:11:30.621-04:00   amazon-ebs: Starting Chef Infra Client, version 16.5.77
    2020-10-09T14:11:30.621-04:00   amazon-ebs: Patents: https://www.chef.io/patents
    2020-10-09T14:11:32.667-04:00   amazon-ebs: resolving cookbooks for run list: ["ubuntu_16_ami_CLIENT_REMOVED::install_base_packages"]
    2020-10-09T14:11:32.667-04:00   amazon-ebs: Synchronizing Cookbooks:
    2020-10-09T14:11:32.667-04:00   amazon-ebs: - ubuntu_16_ami_CLIENT_REMOVED (0.0.0)
    2020-10-09T14:11:32.667-04:00   amazon-ebs: Installing Cookbook Gems:
    2020-10-09T14:11:32.667-04:00   amazon-ebs: Compiling Cookbooks...
    2020-10-09T14:11:32.667-04:00   amazon-ebs: Converging 2 resources
    2020-10-09T14:11:32.667-04:00   amazon-ebs: Recipe: ubuntu_16_ami_CLIENT_REMOVED::install_base_packages
    2020-10-09T14:11:40.698-04:00   amazon-ebs: * execute[apt update cmd] action run
    2020-10-09T14:11:40.698-04:00   amazon-ebs: - execute sudo apt-get update -y
    2020-10-09T14:11:42.735-04:00   amazon-ebs: * apt_package[software-properties-common] action install (up to date)
    2020-10-09T14:11:42.735-04:00   amazon-ebs:
    2020-10-09T14:11:42.735-04:00   amazon-ebs: Running handlers:
    2020-10-09T14:11:42.735-04:00   amazon-ebs: Running handlers complete
    2020-10-09T14:11:42.735-04:00   amazon-ebs: Chef Infra Client finished, 1/2 resources updated in 10 seconds
    2020-10-09T14:11:42.735-04:00   ==> amazon-ebs: Stopping the source instance...
    2020-10-09T14:11:42.735-04:00   amazon-ebs: Stopping instance
    2020-10-09T14:11:42.735-04:00   ==> amazon-ebs: Waiting for the instance to stop...
    2020-10-09T14:12:12.767-04:00   ==> amazon-ebs: Creating AMI ubuntu16_ami_09-Oct-2020-18_10_43UTC from instance i-0497be1e67c330233
    2020-10-09T14:12:12.767-04:00   amazon-ebs: AMI: ami-0f14dfb09d25955c7
    2020-10-09T14:12:12.767-04:00   ==> amazon-ebs: Waiting for AMI to become ready...
    2020-10-09T14:21:46.861-04:00   ==> amazon-ebs: Adding tags to AMI (ami-0f14dfb09d25955c7)...
    2020-10-09T14:21:46.861-04:00   ==> amazon-ebs: Tagging snapshot: snap-0a4e1d58b3ff9eea4
    2020-10-09T14:21:46.861-04:00   ==> amazon-ebs: Creating AMI tags
    2020-10-09T14:21:46.861-04:00   amazon-ebs: Adding tag: "Name": "ubuntu16_ami_09-Oct-2020-18_10_43UTC"
    2020-10-09T14:21:46.861-04:00   amazon-ebs: Adding tag: "source_ami": "ami-0f82752aa17ff8f5d"
    2020-10-09T14:21:46.861-04:00   ==> amazon-ebs: Creating snapshot tags
    2020-10-09T14:21:46.861-04:00   ==> amazon-ebs: Terminating the source AWS instance...
    2020-10-09T14:22:00.910-04:00   ==> amazon-ebs: Cleaning up any extra volumes...
    2020-10-09T14:22:00.910-04:00   ==> amazon-ebs: No volumes to clean up, skipping
    2020-10-09T14:22:00.910-04:00   ==> amazon-ebs: Deleting temporary keypair...
    2020-10-09T14:22:00.910-04:00   Build 'amazon-ebs' finished.
    2020-10-09T14:22:00.910-04:00   
    2020-10-09T14:22:00.910-04:00   ==> Builds finished. The artifacts of successful builds are:
    2020-10-09T14:22:00.910-04:00   --> amazon-ebs: AMIs were created:
    2020-10-09T14:22:00.910-04:00   us-east-1: ami-0f14dfb09d25955c7
    2020-10-09T14:22:00.910-04:00   
    2020-10-09T14:22:00.910-04:00   
    2020-10-09T14:22:00.910-04:00   [Container] 2020/10/09 18:22:00 Running command egrep "${AWS_REGION}\:\sami\-" build.log | cut -d' ' -f2 > ami_id.txt
    2020-10-09T14:22:00.910-04:00   
    2020-10-09T14:22:00.910-04:00   [Container] 2020/10/09 18:22:00 Running command test -s ami_id.txt || exit 1
    2020-10-09T14:22:00.910-04:00   
    2020-10-09T14:22:00.910-04:00   [Container] 2020/10/09 18:22:00 Running command AMI_ID=`cat ami_id.txt`
    2020-10-09T14:22:00.910-04:00   
    2020-10-09T14:22:00.910-04:00   [Container] 2020/10/09 18:22:00 Running command aws ssm put-parameter --name latest_ubuntu16_ami --value ${AMI_ID} --type String --overwrite
    2020-10-09T14:22:06.966-04:00   {
    2020-10-09T14:22:06.966-04:00   "Version": 3,
    2020-10-09T14:22:06.966-04:00   "Tier": "Standard"
    2020-10-09T14:22:06.966-04:00   }
    2020-10-09T14:22:06.966-04:00   
    2020-10-09T14:22:06.966-04:00   [Container] 2020/10/09 18:22:06 Running command echo "build completed on `date`"
    2020-10-09T14:22:06.966-04:00   build completed on Fri Oct 9 18:22:06 UTC 2020
    2020-10-09T14:22:06.966-04:00   
    2020-10-09T14:22:06.966-04:00   [Container] 2020/10/09 18:22:06 Running command bundle exec ami_spec \
    2020-10-09T14:22:06.966-04:00   --ami ${AMI_ID} \
    2020-10-09T14:22:06.966-04:00   --role ami \
    2020-10-09T14:22:06.966-04:00   --subnet-id ${PACKER_SUBNET_ID} \
    2020-10-09T14:22:06.966-04:00   --aws-security-groups ${PACKER_SEC_GRP_ID} \
    2020-10-09T14:22:06.966-04:00   --specs spec \
    2020-10-09T14:22:06.966-04:00   --ssh-user ubuntu \
    2020-10-09T14:22:06.966-04:00   --aws-region us-east-1 \
    2020-10-09T14:22:06.966-04:00   --ssh-retries 50 \
    2020-10-09T14:22:06.966-04:00   --aws-instance-type m5.large
    2020-10-09T14:22:06.966-04:00   
    2020-10-09T14:27:07.027-04:00   Creating temporary AWS key pair: ami-spec-87443c2b-26d1-42fc-a531-b1430e824b2e
    2020-10-09T14:27:07.027-04:00   Deleting temporary AWS key pair: ami-spec-87443c2b-26d1-42fc-a531-b1430e824b2e
    2020-10-09T14:27:07.027-04:00   bundler: failed to load command: ami_spec (/root/.rbenv/versions/2.6.6/bin/ami_spec)
    2020-10-09T14:27:07.027-04:00   Net::SSH::ConnectionTimeout: Net::SSH::ConnectionTimeout
    2020-10-09T14:27:07.027-04:00   /root/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/net-ssh-5.2.0/lib/net/ssh/transport/session.rb:92:in `rescue in initialize'
    2020-10-09T14:27:07.027-04:00   /root/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/net-ssh-5.2.0/lib/net/ssh/transport/session.rb:59:in `initialize'
    2020-10-09T14:27:07.027-04:00   /root/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/net-ssh-5.2.0/lib/net/ssh.rb:246:in `new'
    2020-10-09T14:27:07.027-04:00   /root/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/net-ssh-5.2.0/lib/net/ssh.rb:246:in `start'
    2020-10-09T14:27:07.027-04:00   /root/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/ami_spec-1.6.0/lib/ami_spec/wait_for_ssh.rb:11:in `wait'
    2020-10-09T14:27:07.027-04:00   /root/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/ami_spec-1.6.0/lib/ami_spec.rb:91:in `block in run'
    2020-10-09T14:27:07.027-04:00   /root/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/ami_spec-1.6.0/lib/ami_spec.rb:89:in `each'
    2020-10-09T14:27:07.027-04:00   /root/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/ami_spec-1.6.0/lib/ami_spec.rb:89:in `run'
    2020-10-09T14:27:07.027-04:00   /root/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/ami_spec-1.6.0/lib/ami_spec.rb:191:in `invoke'
    2020-10-09T14:27:07.027-04:00   /root/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/ami_spec-1.6.0/bin/ami_spec:5:in `<top (required)>'
    2020-10-09T14:27:07.027-04:00   /root/.rbenv/versions/2.6.6/bin/ami_spec:23:in `load'
    2020-10-09T14:27:07.027-04:00   /root/.rbenv/versions/2.6.6/bin/ami_spec:23:in `<top (required)>'
    2020-10-09T14:27:07.027-04:00   
    2020-10-09T14:27:07.027-04:00   [Container] 2020/10/09 18:27:06 Command did not exit successfully bundle exec ami_spec \
    2020-10-09T14:27:07.027-04:00   --ami ${AMI_ID} \
    2020-10-09T14:27:07.027-04:00   --role ami \
    2020-10-09T14:27:07.027-04:00   --subnet-id ${PACKER_SUBNET_ID} \
    2020-10-09T14:27:07.027-04:00   --aws-security-groups ${PACKER_SEC_GRP_ID} \
    2020-10-09T14:27:07.027-04:00   --specs spec \
    2020-10-09T14:27:07.027-04:00   --ssh-user ubuntu \
    2020-10-09T14:27:07.027-04:00   --aws-region us-east-1 \
    2020-10-09T14:27:07.027-04:00   --ssh-retries 50 \
    2020-10-09T14:27:07.027-04:00   --aws-instance-type m5.large
    2020-10-09T14:27:07.027-04:00   exit status 1
    2020-10-09T14:27:07.027-04:00   [Container] 2020/10/09 18:27:07 Phase complete: BUILD State: FAILED
    2020-10-09T14:27:07.027-04:00   [Container] 2020/10/09 18:27:07 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: bundle exec ami_spec \
    2020-10-09T14:27:07.027-04:00   --ami ${AMI_ID} \
    2020-10-09T14:27:07.027-04:00   --role ami \
    2020-10-09T14:27:07.027-04:00   --subnet-id ${PACKER_SUBNET_ID} \
    2020-10-09T14:27:07.027-04:00   --aws-security-groups ${PACKER_SEC_GRP_ID} \
    2020-10-09T14:27:07.027-04:00   --specs spec \
    2020-10-09T14:27:07.027-04:00   --ssh-user ubuntu \
    2020-10-09T14:27:07.027-04:00   --aws-region us-east-1 \
    2020-10-09T14:27:07.027-04:00   --ssh-retries 50 \
    2020-10-09T14:27:07.027-04:00   --aws-instance-type m5.large
    2020-10-09T14:27:07.027-04:00   . Reason: exit status 1
    2020-10-09T14:27:07.027-04:00   [Container] 2020/10/09 18:27:07 Entering phase POST_BUILD
    2020-10-09T14:27:07.027-04:00   [Container] 2020/10/09 18:27:07 Phase complete: POST_BUILD State: SUCCEEDED
    2020-10-09T14:27:07.027-04:00   [Container] 2020/10/09 18:27:07 Phase context status code: Message:
    2020-10-09T14:27:07.027-04:00   [Container] 2020/10/09 18:27:07 Expanding base directory path: .
    2020-10-09T14:27:07.027-04:00   [Container] 2020/10/09 18:27:07 Assembling file list
    2020-10-09T14:27:07.027-04:00   [Container] 2020/10/09 18:27:07 Expanding .
    2020-10-09T14:27:07.027-04:00   [Container] 2020/10/09 18:27:07 Expanding file paths for base directory .
    2020-10-09T14:27:07.027-04:00   [Container] 2020/10/09 18:27:07 Assembling file list
    2020-10-09T14:27:07.027-04:00   [Container] 2020/10/09 18:27:07 Expanding **/*
    2020-10-09T14:27:07.027-04:00   [Container] 2020/10/09 18:27:07 Found 22 file(s)

ChrisPetr0 commented 4 years ago

Here is buildspec, driving the build above:

# https://github.com/awslabs/ami-builder-packer/blob/master/buildspec.yml
# example used for reference

version: 0.2

phases:
  install:
    runtime-versions:
      ruby: 2.6
  pre_build:
    commands:
      - echo "Installing Packer"
      - curl -o packer.zip https://releases.hashicorp.com/packer/1.6.1/packer_1.6.1_linux_amd64.zip && unzip packer.zip
      - gem install bundler
      - bundle install
      - bundle exec yaml2json pipeline/ubuntu_16_ami/packer/packer.yml > pipeline/ubuntu_16_ami/packer/packer.json

  build:
    commands:
      - DATE=$(date +%d-%b-%Y"-"%H_%M_%S%Z)
      # Packer Build Command
      - |
        ./packer build \
        -var ami_name=ubuntu16_ami_${DATE} \
        -var sg_1=${PACKER_SEC_GRP_ID} \
        -var source_ami=${PACKER_SOURCE_AMI} \
        -var subnet_id=${PACKER_SUBNET_ID} \
        -var vpc_id=${VPC_ID} \
        -var instance_profile=PackerInstanceProfile \
        -color=false pipeline/ubuntu_16_ami/packer/packer.json | tee build.log

      - egrep "${AWS_REGION}\:\sami\-" build.log | cut -d' ' -f2 > ami_id.txt
      # Packer doesn't return non-zero status; we must do that if Packer build failed
      - test -s ami_id.txt || exit 1
      - AMI_ID=`cat ami_id.txt`
      - aws ssm put-parameter --name latest_ubuntu16_ami --value ${AMI_ID} --type String --overwrite
      - echo "build completed on `date`"

      - |
        bundle exec ami_spec \
        --ami ${AMI_ID} \
        --role ami \
        --subnet-id ${PACKER_SUBNET_ID} \
        --aws-security-groups ${PACKER_SEC_GRP_ID} \
        --specs spec \
        --ssh-user ubuntu \
        --aws-region us-east-1 \
        --ssh-retries 50 \
        --aws-instance-type m5.large

artifacts:
  files:
    - '**/*'
  name: BuildZip

ChrisPetr0 commented 4 years ago

Here are the CFN template snippets:

CodeBuildProjectBuildAmi:
  Type: AWS::CodeBuild::Project
  Properties:
      Name: !Sub '${ServiceName}_build'
      Artifacts:
          Type: CODEPIPELINE
      Environment: 
          Type: LINUX_CONTAINER
          ComputeType: BUILD_GENERAL1_MEDIUM
          Image: !Sub 'aws/codebuild/${CodeBuildEnvironment}'
          PrivilegedMode: true
          EnvironmentVariables:
            - Name: VPC_ID
              Value: !ImportValue NONPROD-vpc-id
            - Name: PACKER_SUBNET_ID
              Value: !ImportValue NONPROD-private1-id
            - Name: PACKER_SEC_GRP_ID
              Value: !ImportValue NONPROD-base-codebuild-sec-grp
            - Name: PACKER_INSTANCE_PROFILE
              Value: !Ref Ubuntu16PackerInstanceRole
            - Name: PACKER_SOURCE_AMI
              Value: !Ref Ubuntu16AmiId
      ServiceRole: !GetAtt Ubuntu16CodeBuildServiceRole.Arn
      Source: 
          Type: CODEPIPELINE
      VpcConfig:
        SecurityGroupIds: 
          - !ImportValue NONPROD-base-codebuild-sec-grp
        Subnets: 
          - !ImportValue NONPROD-private1-id
          - !ImportValue NONPROD-private2-id
          - !ImportValue NONPROD-private3-id
        VpcId: !ImportValue NONPROD-vpc-id

ChrisPetr0 commented 4 years ago

So basically, packer can SSH, ami_spec is failing, seemingly right away, with exact same codebuild setup (in same stage). I can provide you with the whole CFN template and associated packer config if you would like to see that.

I'm really at a loss here and I definitely appreciate you looking into.

Btw - README.MD has small typo here --allow_any_temporary_security_group ( _ instead of -)

patrobinson commented 4 years ago

Can you please provide the configuration of the following security groups? NONPROD-base-codebuild-sec-grp NONPROD-base-codebuild-sec-grp

ChrisPetr0 commented 4 years ago

Sure, it's the same security group, Protocol	Type	Port	IP CIDR	Desc
SSH	TCP	22	34.228.4.208/28	us-east-1 published codebulid range
SSH	TCP	22	10.127.XX.0/24	22 inbound vpc whole CIDR
HTTPS	TCP	443	34.228.4.208/28	us-east-1 published codebulid range
HTTPS	TCP	443	10.127.XX.0/24	443 inbound vpc whole CIDR

Outbound is default / wide open

I'm going to open this SG up to 0.0.0.0/0 on port 22 right now and test.

Thanks

Chris

ChrisPetr0 commented 4 years ago

Hello, I just wanted to confirm, just now I ran the same build as Friday with an additional SG Rule Inbound for port 22, 0.0.0.0/0 and still got the net-ssh error (and packer success). Any thoughts appreciated, I'm pretty sure configuration should work now especially if packer can ssh without issue across same settings and SG, and now especially with 22 open from anywhere.

Let me know what else I can provide. Thanks,

Chris

patrobinson commented 4 years ago

The problem this is being raised deep inside net-ssh code https://github.com/net-ssh/net-ssh/blob/v5.2.0/lib/net/ssh/transport/session.rb#L92

This only gets raised when the very explicit exception Errno::ETIMEDOUT gets raised. In this scenario this only get thrown when the connect system call returns a ETIMEDOUT error, as we're not setting a connection timeout ourselves.

So I can't imagine a scenario where this is a problem with our code or even net-ssh.

ChrisPetr0 commented 4 years ago

Thanks Patrick. Yeah, I think my journey of trying to figure this out may be ending until I have more time. I appreciate the thoughts and input. I might try to get another colleague to take a look and will close for now.

envato / ami-spec

net-ssh broken with AWS codebuild? #54