search

test-kitchen / kitchen-ec2

A Test Kitchen Driver for Amazon EC2
Other
222 stars 203 forks source link

EC2 instance closes connection while transferring files #267

Closed fiunchinho closed 6 years ago

fiunchinho commented 7 years ago

Using this driver everything goes fine until it tries to transfer files to the server

.
..
...
...
Transferring files to <ansible-amazon>
/Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.0.0/gems/net-ssh-3.2.0/lib/net/ssh/transport/packet_stream.rb:89:in `next_packet': connection closed by remote host (Net::SSH::Disconnect)
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.0.0/gems/net-ssh-3.2.0/lib/net/ssh/transport/session.rb:193:in `block in poll_message'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.0.0/gems/net-ssh-3.2.0/lib/net/ssh/transport/session.rb:188:in `loop'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.0.0/gems/net-ssh-3.2.0/lib/net/ssh/transport/session.rb:188:in `poll_message'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.0.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:474:in `dispatch_incoming_packets'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.0.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:225:in `preprocess'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.0.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:206:in `process'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.0.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:170:in `block in loop'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.0.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:170:in `loop'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.0.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:170:in `loop'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.0.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:119:in `close'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.0.0/gems/test-kitchen-1.10.2/lib/kitchen/transport/ssh.rb:115:in `close'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.0.0/gems/test-kitchen-1.10.2/lib/kitchen/transport/ssh.rb:97:in `cleanup!'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.0.0/gems/test-kitchen-1.10.2/lib/kitchen/instance.rb:274:in `cleanup!'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.0.0/gems/test-kitchen-1.10.2/lib/kitchen/command.rb:209:in `run_action_in_thread'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.0.0/gems/test-kitchen-1.10.2/lib/kitchen/command.rb:173:in `block (2 levels) in run_action'

If I ssh into the instance and look the /var/log/secure log

Aug  1 16:40:24 ip-192-168-104-147 sshd[2403]: pam_unix(sshd:session): session opened for user ec2-user by (uid=0)
Aug  1 16:40:25 ip-192-168-104-147 sudo: ec2-user : TTY=pts/0 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/bin/mkdir -p /etc/chef/ohai/hints
Aug  1 16:40:25 ip-192-168-104-147 sudo: ec2-user : TTY=pts/0 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/bin/touch /etc/chef/ohai/hints/ec2.json
Aug  1 16:40:26 ip-192-168-104-147 sudo: ec2-user : TTY=pts/0 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/usr/bin/yum-config-manager --enable epel/x86_64
Aug  1 16:40:26 ip-192-168-104-147 sudo: ec2-user : TTY=pts/0 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/usr/bin/yum -y install ansible git
Aug  1 16:40:41 ip-192-168-104-147 sudo: ec2-user : TTY=pts/0 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/usr/sbin/alternatives --set python /usr/bin/python2.6
Aug  1 16:40:41 ip-192-168-104-147 sudo: ec2-user : TTY=pts/0 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/usr/bin/yum clean all
Aug  1 16:40:41 ip-192-168-104-147 sudo: ec2-user : TTY=pts/0 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/usr/bin/yum install yum-python26 -y
Aug  1 16:40:47 ip-192-168-104-147 sudo: ec2-user : TTY=pts/0 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/bin/rm /usr/bin/ruby
Aug  1 16:40:47 ip-192-168-104-147 sudo: ec2-user : TTY=pts/0 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/bin/ln -s /usr/bin/ruby2.0 /usr/bin/ruby
Aug  1 16:40:47 ip-192-168-104-147 sudo: ec2-user : TTY=pts/0 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/bin/rm -rf /tmp/kitchen/modules /tmp/kitchen/roles /tmp/kitchen/group_vars /tmp/kitchen/host_vars
Aug  1 16:40:48 ip-192-168-104-147 sshd[2403]: error: buffer_get_ret: trying to get more bytes 68 than in buffer 63
Aug  1 16:40:48 ip-192-168-104-147 sshd[2403]: error: buffer_get_string_ret: buffer_get failed
Aug  1 16:40:48 ip-192-168-104-147 sshd[2403]: fatal: buffer_get_string: buffer error
Aug  1 16:40:48 ip-192-168-104-147 sshd[2403]: pam_unix(sshd:session): session closed for user ec2-user
Aug  1 16:40:48 ip-192-168-104-147 sshd[2642]: fatal: mm_request_send: write: Broken pipe
Aug  1 16:40:48 ip-192-168-104-147 sshd[2642]: fatal: mm_request_send: write: Broken pipe
Aug  1 16:40:48 ip-192-168-104-147 sshd[2641]: fatal: mm_request_send: write: Broken pipe
Aug  1 16:40:48 ip-192-168-104-147 sshd[2641]: fatal: mm_request_send: write: Broken pipe
Aug  1 16:40:48 ip-192-168-104-147 sshd[2640]: fatal: mm_request_send: write: Broken pipe
Aug  1 16:40:48 ip-192-168-104-147 sshd[2640]: fatal: mm_request_send: write: Broken pipe
Aug  1 16:40:48 ip-192-168-104-147 sshd[2639]: fatal: mm_request_send: write: Broken pipe
Aug  1 16:40:48 ip-192-168-104-147 sshd[2639]: fatal: mm_request_send: write: Broken pipe
Aug  1 16:40:48 ip-192-168-104-147 sshd[2638]: fatal: mm_request_send: write: Broken pipe
Aug  1 16:40:48 ip-192-168-104-147 sshd[2638]: fatal: mm_request_send: write: Broken pipe
Aug  1 16:40:48 ip-192-168-104-147 sshd[2455]: fatal: mm_request_send: write: Broken pipe
Aug  1 16:40:48 ip-192-168-104-147 sshd[2455]: fatal: mm_request_send: write: Broken pipe
Aug  1 16:43:08 ip-192-168-104-147 sshd[2661]: reverse mapping checking getaddrinfo for static-XX-XX-XXX-XX.ipcom.comunitel.net [xx.xx.x.x] failed - POSSIBLE BREAK-IN ATTEMPT!
cheeseplus commented 7 years ago

Without diving into it too much is that Ruby 2.0.0 is EOL and probably is broken in all sorts of ways as most people haven't supported it for a while now.

fiunchinho commented 7 years ago

It's the latest Ruby version in Amazon Linux

cheeseplus commented 7 years ago

Ruby considers it EOL https://www.ruby-lang.org/en/downloads/ and test-kitchen dropped support in it's testing matrix a while ago.

fiunchinho commented 7 years ago

so is there any way to run commands before this driver gets executed so I can update ruby?

cheeseplus commented 7 years ago

The ruby 2.0.0 I'm pointing out from the logs is on the machine running kitchen, not the kitchen instance so you this question shouldn't play into it.

fiunchinho commented 7 years ago

Ok, I've updated ruby locally, but the problem remains

Transferring files to <ansible-amazon>
/Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/transport/packet_stream.rb:89:in `next_packet': connection closed by remote host (Net::SSH::Disconnect)
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/transport/session.rb:193:in `block in poll_message'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/transport/session.rb:188:in `loop'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/transport/session.rb:188:in `poll_message'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:474:in `dispatch_incoming_packets'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:225:in `preprocess'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:206:in `process'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:170:in `block in loop'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:170:in `loop'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:170:in `loop'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:119:in `close'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/test-kitchen-1.10.2/lib/kitchen/transport/ssh.rb:115:in `close'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/test-kitchen-1.10.2/lib/kitchen/transport/ssh.rb:97:in `cleanup!'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/test-kitchen-1.10.2/lib/kitchen/instance.rb:274:in `cleanup!'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/test-kitchen-1.10.2/lib/kitchen/command.rb:209:in `run_action_in_thread'
    from /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/test-kitchen-1.10.2/lib/kitchen/command.rb:173:in `block (2 levels) in run_action'

any ideas?

fiunchinho commented 7 years ago

Running converge shows a similar error

$ kitchen converge ansible-amazon
-----> Starting Kitchen (v1.10.2)
-----> Converging <ansible-amazon>...
       Preparing files for transfer
       Preparing playbook
       Preparing inventory
       Preparing modules
       nothing to do for modules
       Preparing roles
       Preparing ansible.cfg file
       Empty ansible.cfg generated
       Preparing group_vars
       Preparing additional_copy_path
       Copy recursive additional path: playbook.yml
       Preparing host_vars
       nothing to do for host_vars
       Preparing hosts file
       Preparing spec
       Preparing library plugins
       nothing to do for library plugins
       Preparing callback plugins
       nothing to do for callback plugins
       Preparing filter_plugins
       nothing to do for filter_plugins
       Preparing lookup_plugins
       nothing to do for lookup_plugins
       Finished Preparing files for transfer
       Installing ansible on amazon
       Transferring files to <ansible-amazon>
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Failed to complete #converge action: [SCP did not finish successfully (255): ] on ansible-amazon
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
fiunchinho commented 7 years ago

Running on debug mode

D      [SSH] opening connection to ec2-user@ec2-xx-xx-xx-xx.eu-west-1.compute.amazonaws.com<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>22, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :keys_only=>true, :keys=>["/Users/jose.armesto/.ssh/server.pem"], :auth_methods=>["publickey"]}>
D      sudo -E rm -rf /tmp/kitchen/modules /tmp/kitchen/roles /tmp/kitchen/group_vars /tmp/kitchen/host_vars; mkdir -p /tmp/kitchen
D      [SSH] ec2-user@ec2-xx-xx-xx-xxx.eu-west-1.compute.amazonaws.com<{:user_known_hosts_file=>"/dev/null", :paranoid=>false, :port=>22, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :keys_only=>true, :keys=>["/Users/jose.armesto/.ssh/server.pem"], :auth_methods=>["publickey"], :user=>"ec2-user"}> (sudo -E rm -rf /tmp/kitchen/modules /tmp/kitchen/roles /tmp/kitchen/group_vars /tmp/kitchen/host_vars; mkdir -p /tmp/kitchen)
       Transferring files to <ansible-amazon>
D      TIMING: scp async upload (Kitchen::Transport::Ssh)
D      Cleaning up local sandbox in /var/folders/cm/svhz59210kng9dbl1sllhyqc0000gp/T/ansible-amazon-sandbox-20160802-2611-15ceqr2
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Failed to complete #converge action: [SCP did not finish successfully (255): ] on ansible-amazon
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

D      ------Exception-------
D      Class: Kitchen::ActionFailed
D      Message: 1 actions failed.
>>>>>>     Failed to complete #converge action: [SCP did not finish successfully (255): ] on ansible-amazon
D      ----------------------
D      ------Backtrace-------
D      /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/test-kitchen-1.10.2/lib/kitchen/command.rb:187:in `report_errors'
D      /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/test-kitchen-1.10.2/lib/kitchen/command.rb:178:in `run_action'
D      /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/test-kitchen-1.10.2/lib/kitchen/command/action.rb:39:in `block in call'
D      /Users/jose.armesto/.rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/benchmark.rb:293:in `measure'
D      /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/test-kitchen-1.10.2/lib/kitchen/command/action.rb:37:in `call'
D      /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/test-kitchen-1.10.2/lib/kitchen/cli.rb:56:in `perform'
D      /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/test-kitchen-1.10.2/lib/kitchen/cli.rb:184:in `block (2 levels) in <class:CLI>'
D      /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
D      /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
D      /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/test-kitchen-1.10.2/lib/kitchen/cli.rb:325:in `invoke_task'
D      /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
D      /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
D      /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/test-kitchen-1.10.2/bin/kitchen:13:in `block in <top (required)>'
D      /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/test-kitchen-1.10.2/lib/kitchen/errors.rb:174:in `with_friendly_errors'
D      /Users/jose.armesto/dev/test-kitchen-cookbook/vendor/ruby/2.3.0/gems/test-kitchen-1.10.2/bin/kitchen:13:in `<top (required)>'
D      ./bin/kitchen:17:in `load'
D      ./bin/kitchen:17:in `<main>'
D      ----End Backtrace-----
ti-mo commented 7 years ago

Facing the exact same issue, worked around it by using kitchen-sync (SFTP) as a transport mechanism.

cheeseplus commented 7 years ago

We need a repro case or this isn't going to get much traction - we've only seen this with the ansible and puppet provisioners I believe so narrowing it down is going to be critical to solving it so kitchen.yml configs that can repro this repeatedly are needed.

krazykid commented 7 years ago

I've been having this problem too. Sometimes it'll work, other times it won't. Here are the various outputs and files:

kitchen.log:

I, [2016-10-03T13:58:21.003503 #11843]  INFO -- Kitchen: -----> Starting Kitchen (v1.13.2)
I, [2016-10-03T13:58:23.884107 #11843]  INFO -- Kitchen: -----> Converging <default-amazon-linux>...
E, [2016-10-03T13:58:29.198800 #11843] ERROR -- Kitchen: ------Exception-------
E, [2016-10-03T13:58:29.198835 #11843] ERROR -- Kitchen: Class: Kitchen::ActionFailed
E, [2016-10-03T13:58:29.198910 #11843] ERROR -- Kitchen: Message: 1 actions failed.
>>>>>>     Failed to complete #converge action: [SCP did not finish successfully (255): ] on default-amazon-linux
E, [2016-10-03T13:58:29.198938 #11843] ERROR -- Kitchen: ----------------------
E, [2016-10-03T13:58:29.198958 #11843] ERROR -- Kitchen: ------Backtrace-------
E, [2016-10-03T13:58:29.198977 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/command.rb:187:in `report_errors'
E, [2016-10-03T13:58:29.198996 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/command.rb:178:in `run_action'
E, [2016-10-03T13:58:29.199036 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/command/action.rb:39:in `block in call'
E, [2016-10-03T13:58:29.199054 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/2.3.0/benchmark.rb:293:in `measure'
E, [2016-10-03T13:58:29.199072 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/command/action.rb:37:in `call'
E, [2016-10-03T13:58:29.199090 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/cli.rb:56:in `perform'
E, [2016-10-03T13:58:29.199108 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/cli.rb:186:in `block (2 levels) in <class:CLI>'
E, [2016-10-03T13:58:29.199126 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
E, [2016-10-03T13:58:29.199144 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
E, [2016-10-03T13:58:29.199162 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/cli.rb:334:in `invoke_task'
E, [2016-10-03T13:58:29.199180 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
E, [2016-10-03T13:58:29.199198 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
E, [2016-10-03T13:58:29.199216 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/bin/kitchen:13:in `block in <top (required)>'
E, [2016-10-03T13:58:29.199234 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/errors.rb:174:in `with_friendly_errors'
E, [2016-10-03T13:58:29.199252 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/bin/kitchen:13:in `<top (required)>'
E, [2016-10-03T13:58:29.199270 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/bin/kitchen:22:in `load'
E, [2016-10-03T13:58:29.199287 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/bin/kitchen:22:in `<main>'
E, [2016-10-03T13:58:29.199305 #11843] ERROR -- Kitchen: ----End Backtrace-----
E, [2016-10-03T13:58:29.199322 #11843] ERROR -- Kitchen: -Composite Exception--
E, [2016-10-03T13:58:29.199339 #11843] ERROR -- Kitchen: Class: Kitchen::ActionFailed
E, [2016-10-03T13:58:29.199356 #11843] ERROR -- Kitchen: Message: Failed to complete #converge action: [SCP did not finish successfully (255): ] on default-amazon-linux
E, [2016-10-03T13:58:29.199374 #11843] ERROR -- Kitchen: ----------------------
E, [2016-10-03T13:58:29.199391 #11843] ERROR -- Kitchen: ------Backtrace-------
E, [2016-10-03T13:58:29.199408 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-scp-1.2.1/lib/net/scp.rb:365:in `block (3 levels) in start_command'
E, [2016-10-03T13:58:29.199425 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/channel.rb:607:in `do_close'
E, [2016-10-03T13:58:29.199453 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:614:in `channel_close'
E, [2016-10-03T13:58:29.199486 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:479:in `dispatch_incoming_packets'
E, [2016-10-03T13:58:29.199506 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:225:in `preprocess'
E, [2016-10-03T13:58:29.199524 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:206:in `process'
E, [2016-10-03T13:58:29.199542 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:170:in `block in loop'
E, [2016-10-03T13:58:29.199559 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:170:in `loop'
E, [2016-10-03T13:58:29.199577 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:170:in `loop'
E, [2016-10-03T13:58:29.199595 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/channel.rb:269:in `wait'
E, [2016-10-03T13:58:29.199612 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/transport/ssh.rb:174:in `each'
E, [2016-10-03T13:58:29.199629 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/transport/ssh.rb:174:in `block in upload'
E, [2016-10-03T13:58:29.199647 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/2.3.0/benchmark.rb:293:in `measure'
E, [2016-10-03T13:58:29.199665 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/transport/ssh.rb:164:in `upload'
E, [2016-10-03T13:58:29.199682 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/provisioner/base.rb:75:in `block in call'
E, [2016-10-03T13:58:29.199700 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/transport/base.rb:94:in `initialize'
E, [2016-10-03T13:58:29.199717 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/transport/ssh.rb:441:in `new'
E, [2016-10-03T13:58:29.199735 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/transport/ssh.rb:441:in `create_new_connection'
E, [2016-10-03T13:58:29.199752 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/transport/ssh.rb:93:in `connection'
E, [2016-10-03T13:58:29.199770 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/provisioner/base.rb:71:in `call'
E, [2016-10-03T13:58:29.199788 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:380:in `block in converge_action'
E, [2016-10-03T13:58:29.199805 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:520:in `synchronize_or_call'
E, [2016-10-03T13:58:29.199836 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:485:in `block in action'
E, [2016-10-03T13:58:29.199855 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/2.3.0/benchmark.rb:293:in `measure'
E, [2016-10-03T13:58:29.199873 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:484:in `action'
E, [2016-10-03T13:58:29.199890 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:376:in `converge_action'
E, [2016-10-03T13:58:29.199914 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:355:in `block in transition_to'
E, [2016-10-03T13:58:29.199934 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:354:in `each'
E, [2016-10-03T13:58:29.199951 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:354:in `transition_to'
E, [2016-10-03T13:58:29.199969 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:138:in `converge'
E, [2016-10-03T13:58:29.199987 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/command.rb:201:in `public_send'
E, [2016-10-03T13:58:29.200005 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/command.rb:201:in `run_action_in_thread'
E, [2016-10-03T13:58:29.200022 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/command.rb:173:in `block (2 levels) in run_action'
E, [2016-10-03T13:58:29.200040 #11843] ERROR -- Kitchen: ----End Backtrace-----
E, [2016-10-03T13:58:29.200057 #11843] ERROR -- Kitchen: ---Nested Exception---
E, [2016-10-03T13:58:29.200073 #11843] ERROR -- Kitchen: Class: Kitchen::ActionFailed
E, [2016-10-03T13:58:29.200090 #11843] ERROR -- Kitchen: Message: Failed to complete #converge action: [SCP did not finish successfully (255): ]
E, [2016-10-03T13:58:29.200107 #11843] ERROR -- Kitchen: ----------------------
E, [2016-10-03T13:58:29.200144 #11843] ERROR -- Kitchen: ------Backtrace-------
E, [2016-10-03T13:58:29.200162 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-scp-1.2.1/lib/net/scp.rb:365:in `block (3 levels) in start_command'
E, [2016-10-03T13:58:29.200180 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/channel.rb:607:in `do_close'
E, [2016-10-03T13:58:29.200198 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:614:in `channel_close'
E, [2016-10-03T13:58:29.200216 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:479:in `dispatch_incoming_packets'
E, [2016-10-03T13:58:29.200233 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:225:in `preprocess'
E, [2016-10-03T13:58:29.200251 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:206:in `process'
E, [2016-10-03T13:58:29.200268 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:170:in `block in loop'
E, [2016-10-03T13:58:29.200286 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:170:in `loop'
E, [2016-10-03T13:58:29.200303 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:170:in `loop'
E, [2016-10-03T13:58:29.200321 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/channel.rb:269:in `wait'
E, [2016-10-03T13:58:29.200338 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/transport/ssh.rb:174:in `each'
E, [2016-10-03T13:58:29.200356 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/transport/ssh.rb:174:in `block in upload'
E, [2016-10-03T13:58:29.200373 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/2.3.0/benchmark.rb:293:in `measure'
E, [2016-10-03T13:58:29.200390 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/transport/ssh.rb:164:in `upload'
E, [2016-10-03T13:58:29.200414 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/provisioner/base.rb:75:in `block in call'
E, [2016-10-03T13:58:29.200433 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/transport/base.rb:94:in `initialize'
E, [2016-10-03T13:58:29.200450 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/transport/ssh.rb:441:in `new'
E, [2016-10-03T13:58:29.200468 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/transport/ssh.rb:441:in `create_new_connection'
E, [2016-10-03T13:58:29.200485 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/transport/ssh.rb:93:in `connection'
E, [2016-10-03T13:58:29.200503 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/provisioner/base.rb:71:in `call'
E, [2016-10-03T13:58:29.200520 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:380:in `block in converge_action'
E, [2016-10-03T13:58:29.200537 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:520:in `synchronize_or_call'
E, [2016-10-03T13:58:29.200555 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:485:in `block in action'
E, [2016-10-03T13:58:29.200588 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/2.3.0/benchmark.rb:293:in `measure'
E, [2016-10-03T13:58:29.200606 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:484:in `action'
E, [2016-10-03T13:58:29.200624 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:376:in `converge_action'
E, [2016-10-03T13:58:29.200641 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:355:in `block in transition_to'
E, [2016-10-03T13:58:29.200658 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:354:in `each'
E, [2016-10-03T13:58:29.200676 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:354:in `transition_to'
E, [2016-10-03T13:58:29.200693 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:138:in `converge'
E, [2016-10-03T13:58:29.200711 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/command.rb:201:in `public_send'
E, [2016-10-03T13:58:29.200728 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/command.rb:201:in `run_action_in_thread'
E, [2016-10-03T13:58:29.200745 #11843] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/command.rb:173:in `block (2 levels) in run_action'
E, [2016-10-03T13:58:29.200763 #11843] ERROR -- Kitchen: ----End Backtrace-----

.kitchen.yml:

#<% require 'kitchen-sync' %>

provisioner:
  name: chef_zero
  data_path: test/fixtures
  require_chef_omnibus: true
  http_proxy: <%= ENV['bamboo_PROVISIONER_HTTP_PROXY'] %>
  https_proxy: <%= ENV['bamboo_PROVISIONER_HTTPS_PROXY'] %>

driver:
  name: ec2
  aws_ssh_key_id: <%= ENV['bamboo_AWS_KEYPAIR_NAME'] %>
  security_group_ids: <%= ENV['bamboo_AWS_SEC_GROUP_IDS'].split(',') %>
  region: <%= ENV['bamboo_AWS_REGION'] %>
  availability_zone: <%= ENV['bamboo_AWS_AZ'] %>
  require_chef_omnibus: true
  subnet_id: <%= ENV['bamboo_AWS_SUBNET_ID'] %>
  instance_type: <%= ENV['bamboo_AWS_INSTANCE_TYPE'] %> # TODO: Make sure env is set in Bamboo because it's new
  shared_credentials_profile: <%= ENV['bamboo_AWS_PROFILE'] %>
  associate_public_ip: true
  ssl_verify_peer: false
  http_proxy: <%= ENV['bamboo_DRIVER_HTTP_PROXY'] %>
  https_proxy: <%= ENV['bamboo_DRIVER_HTTPS_PROXY'] %>

  # TODO: the settings below need to be researcehd and set appropriately
  # iam_profile_name: chef_client 
  # interface: dns

transport:
  ssh_key: <%= ENV['bamboo_EC2_SSH_KEY_PATH'] %>
  username: ec2-user

platforms:
  - name: amazon-linux
    driver:
      image_id: <%= ENV['bamboo_AWS_IMAGE_ID'] %>
      block_device_mappings:
        - device_name: /dev/xvda
          ebs:
            volume_type: gp2
            volume_size: 24
            delete_on_termination: true

suites:
  - name: default
    run_list:
      - recipe[mitre-baseline::default]
    attributes:
      mitre-bamboonessus: true

verifier:
  http_proxy: <%= ENV['bamboo_VERIFIER_HTTP_PROXY'] %>
  https_proxy: <%= ENV['bamboo_VERIFIER_HTTPS_PROXY'] %>

Client log file - /var/log/secure

Oct  3 17:58:25 ip-172-33-1-191 sshd[2701]: Accepted publickey for ec2-user from 192.160.51.86 port 55424 ssh2: RSA b1:0a:73:0f:a1:f5:6b:df:3c:9f:30:75:43:40:f2:af
Oct  3 17:58:25 ip-172-33-1-191 sshd[2701]: pam_unix(sshd:session): session opened for user ec2-user by (uid=0)
Oct  3 17:58:25 ip-172-33-1-191 sudo: ec2-user : TTY=pts/1 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/bin/sh -c #012#012chef_omnibus_root="/opt/chef"#012chef_omnibus_url="https://omnitruck.chef.io/install.sh"#012install_flags=""#012pretty_version="install only if missing"#012sudo_sh="sudo -E sh"#012version="true"#012#012tmp_stderr="/tmp/stderr";#012#012# capture_tmp_stderr SOURCE#012capture_tmp_stderr() {#012  # spool up $tmp_stderr from all the commands we called#012  if test -f "$tmp_stderr"; then#012    output="`cat $tmp_stderr`";#012    stderr_results="${stderr_results}\nSTDERR from $1:\n\n${output}\n";#012    rm $tmp_stderr;#012  fi#012}#012#012# do_curl URL FILENAME#012do_curl() {#012  echo "Trying curl...";#012  curl -sL -D "$tmp_stderr" "$1" > "$2";#012  ec=$?;#012  # check for 404#012  grep "404 Not Found" "$tmp_stderr" 2>&1 >/dev/null;#012  if test $? -eq 0; then#012    http_404_error "$1";#012  fi#012#012  # check for bad return status or empty output#012  if test $ec -ne 0 || test ! -s "$2"; then#012    capture_tmp_stderr "curl";#012    return 1;#012  else#012    echo "Download
Oct  3 17:58:25 ip-172-33-1-191 sudo: ec2-user : (command continued) complete.";#012    return 0;#012  fi#012}#012#012# do_download URL FILENAME#012do_download() {#012  echo "Downloading ${1} to file ${2}";#012#012  exists wget;#012  if test $? -eq 0; then#012    do_wget "$1" "$2" && return 0;#012  fi#012#012  exists curl;#012  if test $? -eq 0; then#012    do_curl "$1" "$2" && return 0;#012  fi#012#012  exists fetch;#012  if test $? -eq 0; then#012    do_fetch "$1" "$2" && return 0;#012  fi#012#012  exists python;#012  if test $? -eq 0; then#012    do_python "$1" "$2" && return 0;#012  fi#012#012  exists perl;#012  if test $? -eq 0; then#012    do_perl "$1" "$2" && return 0;#012  fi#012#012  unable_to_download "$1" "$2";#012}#012#012# do_fetch URL FILENAME#012do_fetch() {#012  echo "Trying fetch...";#012  fetch -o "$2" "$1" 2>"$tmp_stderr";#012  ec=$?;#012  # check for 404#012  grep "Not Found" "$tmp_stderr" 2>&1 >/dev/null;#012  if test $? -eq 0; then#012    http_404_error "$1";#012  fi#012#012  # check for bad return status or empty output#012  if test $ec -ne 0 || test ! -s "$2"; then#012    capture_tmp_stderr "fetch";#012   
Oct  3 17:58:25 ip-172-33-1-191 sudo: ec2-user : (command continued) return 1;#012  else#012    echo "Download complete.";#012    return 0;#012  fi#012}#012#012# do_perl URL FILENAME#012do_perl() {#012  echo "Trying perl...";#012  perl -e "use LWP::Simple; getprint(\$ARGV[0]);" "$1" > "$2" 2>"$tmp_stderr";#012  ec=$?;#012  # check for 404#012  grep "404 Not Found" "$tmp_stderr" 2>&1 >/dev/null;#012  if test $? -eq 0; then#012    http_404_error "$1";#012  fi#012#012  # check for bad return status or empty output#012  if test $ec -ne 0 || test ! -s "$2"; then#012    capture_tmp_stderr "perl";#012    return 1;#012  else#012    echo "Download complete.";#012    return 0;#012  fi#012}#012#012# do_python URL FILENAME#012do_python() {#012  echo "Trying python...";#012  python -c "import sys,urllib2 ; sys.stdout.write(urllib2.urlopen(sys.argv[1]).read())" "$1" > "$2" 2>"$tmp_stderr";#012  ec=$?;#012  # check for 404#012  grep "HTTP Error 404" "$tmp_stderr" 2>&1 >/dev/null;#012  if test $? -eq 0; then#012    http_404_error "$1";#012  fi#012#012  # check for bad return status or empty output#012  if test $ec
Oct  3 17:58:25 ip-172-33-1-191 sudo: ec2-user : (command continued) -ne 0 || test ! -s "$2"; then#012    capture_tmp_stderr "python";#012    return 1;#012  else#012    echo "Download complete.";#012    return 0;#012  fi#012}#012#012# do_wget URL FILENAME#012do_wget() {#012  echo "Trying wget...";#012  wget -O "$2" "$1" 2>"$tmp_stderr";#012  ec=$?;#012  # check for 404#012  grep "ERROR 404" "$tmp_stderr" 2>&1 >/dev/null;#012  if test $? -eq 0; then#012    http_404_error "$1";#012  fi#012#012  # check for bad return status or empty output#012  if test $ec -ne 0 || test ! -s "$2"; then#012    capture_tmp_stderr "wget";#012    return 1;#012  else#012    echo "Download complete.";#012    return 0;#012  fi#012}#012#012# exists COMMAND#012exists() {#012  if command -v "$1" >/dev/null 2>&1; then#012    return 0;#012  else#012    return 1;#012  fi#012}#012#012# http_404_error URL#012http_404_error() {#012  echo ">>>>>> Downloading ${1} resulted in an HTTP/404, aborting";#012  exit 40;#012}#012#012# should_update_chef ROOT VERSION#012should_update_chef() {#012  if test ! -d "$1"; then#012    return 0;#012  elif test "$2" = "true";
Oct  3 17:58:25 ip-172-33-1-191 sudo: ec2-user : (command continued) then#012    return 1;#012  elif test "$2" = "latest"; then#012    return 0;#012  fi#012#012  if test -f "${1}/version-manifest.txt"; then#012    chef_version="`head -n 1 ${1}/version-manifest.txt | cut -d \" \" -f 2`";#012  else#012    chef_version="`${1}/bin/chef-solo -v | cut -d \" \" -f 2`";#012  fi#012#012  echo "$chef_version" | grep "^${2}" 2>&1 >/dev/null;#012  if test $? -eq 0; then#012    return 1;#012  else#012    echo "${2}" | grep "^$chef_version" 2>&1 >/dev/null;#012    if test $? -eq 0; then#012      return 1;#012    else#012      return 0;#012    fi#012  fi#012}#012#012# unable_to_download URL FILE#012unable_to_download() {#012  echo "Unable to download $1 to $2, aborting";#012#012  if test "x${stderr_results}" != "x"; then#012    echo "\nDEBUG OUTPUT FOLLOWS:\n${stderr_results}";#012  fi#012#012  exit 10;#012}#012#012# main#012main() {#012  should_update_chef "$chef_omnibus_root" "$version"#012  if test $? -eq 0; then#012    echo "-----> Installing Chef Omnibus (${pretty_version})";#012#012    # solaris 10 lacks recent
Oct  3 17:58:25 ip-172-33-1-191 sudo: ec2-user : (command continued) enough credentials, so http url is used#012    platform="`/usr/bin/uname -s 2>/dev/null`";#012    platform_version="`/usr/bin/uname -r 2>/dev/null`";#012    if test "x${platform}" = "xSunOS" && test "x${platform_version}" = "x5.10"; then#012      chef_omnibus_url=`echo "$chef_omnibus_url" | sed -e "s/https/http/"`;#012    fi#012#012    do_download "$chef_omnibus_url" /tmp/install.sh;#012    $sudo_sh /tmp/install.sh $install_flags;#012  else#012    echo "-----> Chef Omnibus installation detected (${pretty_version})";#012  fi#012}#012#012# augment path in an attempt to find a download program#012PATH="${PATH}:/opt/local/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/sfw/bin";#012export PATH;#012#012main
Oct  3 17:58:25 ip-172-33-1-191 sudo: ec2-user : TTY=pts/1 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/bin/sh /tmp/install.sh
Oct  3 17:58:28 ip-172-33-1-191 sudo: ec2-user : TTY=pts/1 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/bin/rm -rf /tmp/kitchen/clients /tmp/kitchen/cookbooks /tmp/kitchen/data /tmp/kitchen/data_bags /tmp/kitchen/encrypted_data_bag_secret /tmp/kitchen/environments /tmp/kitchen/roles
Oct  3 17:58:29 ip-172-33-1-191 sshd[2701]: error: buffer_get_ret: trying to get more bytes 68 than in buffer 63
Oct  3 17:58:29 ip-172-33-1-191 sshd[2701]: error: buffer_get_string_ret: buffer_get failed
Oct  3 17:58:29 ip-172-33-1-191 sshd[2701]: fatal: buffer_get_string: buffer error
Oct  3 17:58:29 ip-172-33-1-191 sshd[2701]: pam_unix(sshd:session): session closed for user ec2-user
Oct  3 17:58:29 ip-172-33-1-191 sshd[2800]: fatal: mm_request_send: write: Broken pipe
Oct  3 17:58:29 ip-172-33-1-191 sshd[2800]: fatal: mm_request_send: write: Broken pipe
Oct  3 17:58:29 ip-172-33-1-191 sshd[2799]: fatal: mm_request_send: write: Broken pipe
Oct  3 17:58:29 ip-172-33-1-191 sshd[2799]: fatal: mm_request_send: write: Broken pipe
Oct  3 17:58:29 ip-172-33-1-191 sshd[2703]: fatal: mm_request_send: write: Broken pipe
Oct  3 17:58:29 ip-172-33-1-191 sshd[2703]: fatal: mm_request_send: write: Broken pipe

Client ec2-user - authorized_keys

ssh-rsa XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXMAFU84tSedKxoVOsrzUYl7BLT/ljxOl8slGVNSj9r5NGOH21mhQR6FExRrs3YXzxnUvsplx8nynv6BK1/RQcQQ9RtEuKc0FJWSGS6LU72ytmM6Sazy6uh8eyBteOT8+uREZvafUCI+eCJtLN3hwvz1feUiaMKv0Yebw1AFx+VsRDG+dmMi6cqUtHEfVWMxcyErLzb8UeWcOoSd5cPs6TdnmQxF bamboo2
fletchowns commented 7 years ago

I run into this one quite frequently. Usually it seems to be when it's scp'ing data bags:

       Thank you for installing Chef!
       Transferring files to <mybox-redhat-72>
/opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-scp-1.2.1/lib/net/scp/upload.rb:117:in `stat': No such file or directory @ rb_file_s_stat - /tmp/mybox-redhat-72-sandbox-20170109-6865-tjj5wc/data_bags/tcell/api_keys.json (Errno::ENOENT)
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-scp-1.2.1/lib/net/scp/upload.rb:117:in `set_current'
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-scp-1.2.1/lib/net/scp/upload.rb:103:in `next_item_state'
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-scp-1.2.1/lib/net/scp.rb:400:in `await_response_state'
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-scp-1.2.1/lib/net/scp.rb:369:in `block (3 levels) in start_command'
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/channel.rb:320:in `process'
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:226:in `block in preprocess'
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:226:in `each'
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:226:in `preprocess'
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:206:in `process'
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:170:in `block in loop'
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:170:in `loop'
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:170:in `loop'
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/connection/session.rb:119:in `close'
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/transport/ssh.rb:119:in `close'
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/transport/ssh.rb:101:in `cleanup!'
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/instance.rb:281:in `cleanup!'
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/command.rb:209:in `run_action_in_thread'
    from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/test-kitchen-1.13.2/lib/kitchen/command.rb:173:in `block (2 levels) in run_action'

My machine is Red Hat Enterprise Linux Server release 7.2 (Maipo) on AWS, based on ami-d1315fb1

iiro commented 7 years ago

Just a short comment: I've never bumped into this - and doing TK+kitchen-ec2 a lot every day. Using kitchen-ec2 with chef-zero and sync-RubyGem kitchen-transport-rsync (https://github.com/unibet/kitchen-transport-rsync). Official Ubuntu 14.04 LTS AMI's in use.

fletchowns commented 7 years ago

@iiro thank you for the tip! I switched my build over to the rsync transport and haven't had a net-scp error yet so far.

cheeseplus commented 6 years ago

This one is old enough and broad enough that it's largely encompassed under "net-ssh" issues that are fixed by upgrading to latest deps. I'm going to close this one but if anyone hits a similar issue with current builds we can dive into troubleshoot.