Closed c10l closed 9 years ago
electrical
commented
9 years ago Since you do a mkdir of the .ssh dir it could fail on incorrect rights on the .ssh dir. if you use beaker 2.2.0 it should be solved in there and you can remove the mkdir command as well.
madAndroid
commented
9 years ago I'm experiencing the same issue on a CentOS-6 host.. but on a CentOS-7 host, with the exact same SUT yaml configs, the docker runs work perfectly fine. I'm pretty sure Beaker/Docker ran fine on Centos6 until fairly recently, but I've not run the tests since upgrading to 2.2.0
Interestingly, if I remove docker_cmd completely, it defaults to /usr/sbin/ssh -D, and it connects okay until Beaker attempts a service sshd restart, at which point the daemon disconnects and the test fails.
I also assumed it would be caused by the mkdir /root/.ssh I'd put in place to address that missing, but it also fails without that image command being present
It fails on docker-io-1.3.2-2.el6.x86_64 from epel, as well as docker-io-1.4.1-3.el6.x86_64 from epel-testing
nodeset:
---
HOSTS:
centos6:
roles:
- master
platform: el-6-x86_64
image: centos:centos6
hypervisor: docker
docker_cmd: '["/sbin/init"]'
docker_image_commands:
- 'yum install -y tar gcc ruby-devel git'
CONFIG:
type: foss
log_level: verbose
forward_ssh_agent: true
c10l
commented
9 years ago Same here. With /sbin/init as the docker_cmd it times out same as before. Without it, the connection is lost when sshd restarts.
-bash-4.1$ bundle exec rake beaker
/opt/rubies/ruby-2.1.3/bin/ruby -I/opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib:/opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-support-3.1.2/lib /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/exe/rspec spec/acceptance --color
/opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-rspec-5.0.1/lib/beaker-rspec/helpers/serverspec.rb:43: warning: already initialized constant Module::VALID_OPTIONS_KEYS
/opt/go/agent0/.gem/ruby/2.1.3/gems/specinfra-2.12.1/lib/specinfra/configuration.rb:4: warning: previous definition of VALID_OPTIONS_KEYS was here
Hypervisor for centos6 is docker
Beaker::Hypervisor, found some docker boxes to create
Provisioning docker
provisioning centos6
Creating image
Dockerfile is FROM chef/centos-6
RUN yum clean all
RUN yum install -y sudo openssh-server openssh-clients curl ntpdate
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
RUN mkdir -p /var/run/sshd
RUN echo root:root | chpasswd
RUN yum -y install tar
EXPOSE 22
CMD /usr/sbin/sshd -D
post
/v1.15/build
{:rm=>true}
Dockerfile0000640000000000000000000000062012462403236013304 0ustar00wheelwheel00000000000000 FROM chef/centos-6
RUN yum clean all
RUN yum install -y sudo openssh-server openssh-clients curl ntpdate
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
RUN mkdir -p /var/run/sshd
RUN echo root:root | chpasswd
RUN yum -y install tar
EXPOSE 22
CMD /usr/sbin/sshd -D
Creating container from image df99d6c15b5c
post
/v1.15/containers/create
{}
{"Image":"df99d6c15b5c","Hostname":"centos6"}
Starting container 3f6916ee03df033848d784e2a101113410a9498a8e9fbe6adf374afcc8841ad7
post
/v1.15/containers/3f6916ee03df033848d784e2a101113410a9498a8e9fbe6adf374afcc8841ad7/start
{}
{"PublishAllPorts":true,"Privileged":true}
get
/v1.15/containers/3f6916ee03df033848d784e2a101113410a9498a8e9fbe6adf374afcc8841ad7/json
{}
get
/v1.15/containers/3f6916ee03df033848d784e2a101113410a9498a8e9fbe6adf374afcc8841ad7/json
{}
node available as ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@0.0.0.0 -p 49159
centos6 10:08:41$ rpm -q curl
Warning: Try 1 -- Host 0.0.0.0 unreachable: connection closed by remote host
Warning: Trying again in 1 seconds
Created ssh connection to 0.0.0.0, user: root, opts: {:password=>"root", :port=>"49159", :forward_agent=>false, :user=>"root"}
curl-7.19.7-40.el6_6.3.x86_64
centos6 executed in 1.22 seconds
centos6 10:08:42$ rpm -q ntpdate
ntpdate-4.2.6p5-2.el6.centos.x86_64
centos6 executed in 0.03 seconds
centos6 10:08:43$ echo "/usr/bin"
/usr/bin
centos6 executed in 0.01 seconds
centos6 10:08:43$ echo "/opt/puppet-git-repos/hiera/bin"
/opt/puppet-git-repos/hiera/bin
centos6 executed in 0.01 seconds
centos6 10:08:43$ echo "/opt/puppet-git-repos/hiera/lib"
/opt/puppet-git-repos/hiera/lib
centos6 executed in 0.01 seconds
centos6 10:08:43$ echo "/opt/puppet-git-repos/hiera-puppet/lib"
/opt/puppet-git-repos/hiera-puppet/lib
centos6 executed in 0.01 seconds
setting local environment on centos6
centos6 10:08:43$ echo '
PermitUserEnvironment yes' >> /etc/ssh/sshd_config
centos6 executed in 0.01 seconds
centos6 10:08:43$ /sbin/service sshd restart
Stopping sshd: /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/ruby_compat.rb:30:in `select': closed stream (IOError)
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/ruby_compat.rb:30:in `io_select'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/transport/packet_stream.rb:75:in `available_for_read?'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/transport/packet_stream.rb:87:in `next_packet'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:183:in `block in poll_message'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:178:in `loop'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:178:in `poll_message'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:461:in `dispatch_incoming_packets'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:222:in `preprocess'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:206:in `process'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:170:in `block in loop'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:170:in `loop'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:170:in `loop'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/ssh_connection.rb:96:in `try_to_execute'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/ssh_connection.rb:107:in `execute'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/host.rb:271:in `block in exec'
from /opt/rubies/ruby-2.1.3/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/host.rb:270:in `exec'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/host_prebuilt_steps.rb:489:in `block in set_env'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/shared/host_manager.rb:74:in `run_block_on'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/shared/host_manager.rb:71:in `block in run_block_on'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/shared/host_manager.rb:70:in `map'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/shared/host_manager.rb:70:in `run_block_on'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/dsl/patterns.rb:32:in `block_on'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/host_prebuilt_steps.rb:471:in `set_env'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/hypervisor.rb:111:in `configure'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/network_manager.rb:71:in `block in configure'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/network_manager.rb:70:in `each_key'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-2.2.0/lib/beaker/network_manager.rb:70:in `configure'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-rspec-5.0.1/lib/beaker-rspec/beaker_shim.rb:43:in `configure'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-rspec-5.0.1/lib/beaker-rspec/spec_helper.rb:44:in `block in <top (required)>'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core.rb:81:in `configure'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-rspec-5.0.1/lib/beaker-rspec/spec_helper.rb:5:in `<top (required)>'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-rspec-5.0.1/lib/beaker-rspec.rb:5:in `require'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-rspec-5.0.1/lib/beaker-rspec.rb:5:in `<module:BeakerRSpec>'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/beaker-rspec-5.0.1/lib/beaker-rspec.rb:1:in `<top (required)>'
from /opt/go/agent0/.gem/ruby/2.1.3/bundler/gems/gamesys_puppet_spec_helper-ddcaf8a93a98/lib/gamesys_puppet_spec_helper/spec_helper_acceptance.rb:1:in `require'
from /opt/go/agent0/.gem/ruby/2.1.3/bundler/gems/gamesys_puppet_spec_helper-ddcaf8a93a98/lib/gamesys_puppet_spec_helper/spec_helper_acceptance.rb:1:in `<top (required)>'
from /opt/go/agent0/go-agent/pipelines/remote_file/remote_file/spec/spec_helper_acceptance.rb:2:in `require'
from /opt/go/agent0/go-agent/pipelines/remote_file/remote_file/spec/spec_helper_acceptance.rb:2:in `<top (required)>'
from /opt/go/agent0/go-agent/pipelines/remote_file/remote_file/spec/acceptance/init_spec.rb:1:in `require'
from /opt/go/agent0/go-agent/pipelines/remote_file/remote_file/spec/acceptance/init_spec.rb:1:in `<top (required)>'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105:in `load'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105:in `block in load_spec_files'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105:in `each'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core/configuration.rb:1105:in `load_spec_files'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:96:in `setup'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:84:in `run'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:69:in `run'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib/rspec/core/runner.rb:37:in `invoke'
from /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/exe/rspec:4:in `<main>'
/opt/rubies/ruby-2.1.3/bin/ruby -I/opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/lib:/opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-support-3.1.2/lib /opt/go/agent0/.gem/ruby/2.1.3/gems/rspec-core-3.1.7/exe/rspec spec/acceptance --color failedCould you do a quick test with docker image electrical/centos:6.4-1 instead of the one you are using now?
Just to check if the docker image is misbehaving.
c10l
commented
9 years ago Same behaviour -- timeout with docker_cmd: '["/sbin/init"]', disconnection upon sshd restart without it.
Anything else can provide?
electrical
commented
9 years ago Very weird. because i do the same and has always been working fine for me. https://github.com/elasticsearch/puppet-elasticsearch/blob/master/spec/acceptance/nodesets/centos-6-x64.yml
madAndroid
commented
9 years ago One major difference I've noticed, is that this only fails on a centos 6.6 host for me (with docker installed via EPEL), the exact same nodeset works fine on a centos 7 host (with docker installed via the standard centos 7 repos) - I've not tested on any other operating systems though.
c10l
commented
9 years ago I'm running Docker 1.4.1 installed from epel-testing on a CentOS 6.6 host. Can't test it on CentOS 7 as we run stuff on xen and no one has been able to make that combination work yet around here. :)
electrical
commented
9 years ago I'm running on a Ubuntu 14.04 host with docker 1.4.1
c10l
commented
9 years ago How can I investigate this issue further? I'm at a complete loss here...
electrical
commented
9 years ago Must admit its very confusing. Could you try out my exact node config? wonder if that works. it it doesn't i'm fairly out of idea's.
c10l
commented
9 years ago Same results...
madAndroid
commented
9 years ago It looks as though sshd is not being started in the container when run on a centos6 host; when I attach to a running container that Beaker's created with docker_cmd: '["/sbin/init"]', I don't seem to see a running ssh daemon, as far as I can tell:
-bash-4.1$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fc4f926000d0 0364c8063090 "/sbin/init" About a minute ago Up About a minute 0.0.0.0:49153->22/tcp nostalgic_lalande
-bash-4.1$ docker exec -i -t fc4f926000d0 bash
[root@centos6 /]# ps auxff
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 85 0.0 0.0 14724 1804 ? S 17:48 0:00 bash
root 99 0.0 0.0 16624 1044 ? R+ 17:48 0:00 \_ ps auxff
root 1 0.0 0.0 17128 1244 ? Ss 17:46 0:00 /sbin/init
root 11 0.0 0.0 14860 1908 ? Ss 17:46 0:00 /bin/bash /etc/rc.d/rc.sysinit
root 28 0.0 0.0 14972 1904 ? S 17:46 0:00 \_ /bin/bash /sbin/start_udev
root 66 0.2 0.0 8416 720 ? S 17:46 0:00 \_ /sbin/udevadm settle
root 60 0.0 0.0 10640 492 ? S<s 17:46 0:00 /sbin/udevd -dBy contrast, on the centos7 host, when I attach to the container from the same nodeset YAML:
[root@ip-172-31-12-185 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
65bca78c5cc6 a1a7ac8946dc "/sbin/init" 4 seconds ago Up 3 seconds 0.0.0.0:49590->22/tcp jovial_mestorf
91515fc1153f a1a7ac8946dc "/sbin/init" 28 hours ago Up 28 hours 0.0.0.0:49185->22/tcp romantic_wilson
[root@ip-172-31-12-185 ~]# docker exec -i -t 65bca78c5cc6 bash
[root@centos6 /]# ps auxff
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 679 0.4 0.0 108348 1680 pts/2 S 18:33 0:00 bash
root 799 0.0 0.0 110200 1056 pts/2 R+ 18:33 0:00 \_ ps auxff
root 1 0.1 0.0 19284 1500 ? Ss 18:33 0:00 /sbin/init
root 67 0.4 0.0 10824 908 ? S<s 18:33 0:00 /sbin/udevd -d
root 474 0.0 0.0 4132 652 ? Ss 18:33 0:00 /sbin/agetty /dev/ttyS0 115200 vt100-nav
root 476 0.0 0.0 4116 596 tty1 Ss+ 18:33 0:00 /sbin/mingetty /dev/tty1
root 478 0.0 0.0 4116 596 tty2 Ss+ 18:33 0:00 /sbin/mingetty /dev/tty2
root 480 0.0 0.0 4116 596 tty3 Ss+ 18:33 0:00 /sbin/mingetty /dev/tty3
root 482 0.0 0.0 4116 596 tty4 Ss+ 18:33 0:00 /sbin/mingetty /dev/tty4
root 484 0.0 0.0 4116 592 tty5 Ss+ 18:33 0:00 /sbin/mingetty /dev/tty5
root 486 0.0 0.0 4116 592 tty6 Ss+ 18:33 0:00 /sbin/mingetty /dev/tty6
root 516 0.0 0.0 66740 1176 ? Ss 18:33 0:00 /usr/sbin/sshd
root 533 0.5 0.0 96516 4476 ? Ss 18:33 0:00 \_ sshd: root@notty
root 724 16.3 0.2 142756 40024 ? Ssl 18:33 0:00 \_ /usr/bin/ruby /usr/bin/puppet apply --verbose --detailed-exitcodes /tmp/apply_manifest.pp.7kNnS6
[root@centos6 /]#So, it appears that sshd is not starting correctly when run on a centos6 host, afaict .. exactly why that's happening, is beyond me, I'm afraid..
electrical
commented
9 years ago Very weird.. could it be a kernel issue? which kernel is it on centos 6 and 7 ?
madAndroid
commented
9 years ago I suspect that it could be the kernel - there are mentions of issues with 2.6.x kernels in the docker github issue tracker - but afaict, they're all either closed or resolved .. so unless it's a very recent development, it's possible that not many others are experiencing the same issue. In addition to this, I'm not sure whether this is specific to Beaker (or the docker-api gem) or whether it's a general issue with docker on centos6 (due to limitations with the 2.6.x kernels)
What's very confusing is that I could have sworn it was working on centos6 previously, I'm just not sure when it stopped working. I'm going to try with an earlier version of Beaker at some point, but I won't be able to get to this in a while... @cassianoleal - is that something you could try: earlier releases of Beaker, as well as the docker-api gem?
c10l
commented
9 years ago -bash-4.1$ uname -a
Linux puppetci-agent0 2.6.32-504.8.1.el6.x86_64 #1 SMP Wed Jan 28 21:11:36 UTC 2015 x86_64 x86_64 x86_64 GNU/LinuxAFAIK all Docker incompatibilities have been solved on this kernel (it's the latest on CentOS 6).
I'll try to downgrade the gems as @madAndroid suggested and see what happens.
For the record, I spawned a container using the Dockerfile that Beaker generates, then attached to it and sshd is definitely running:
-bash-4.1$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ed4c51ff27c0 dfb7dc321cea "/sbin/init" 4 seconds ago Up 3 seconds 22/tcp determined_archimedes
-bash-4.1$ cat Dockerfile
FROM electrical/centos:6.4
RUN yum clean all
RUN yum install -y sudo openssh-server openssh-clients curl ntpdate
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
RUN mkdir -p /var/run/sshd
RUN echo root:root | chpasswd
RUN yum install -y wget ntpdate rubygems ruby-augeas ruby-devel augeas-devel
RUN touch /etc/sysconfig/network
EXPOSE 22
CMD ["/sbin/init"]
-bash-4.1$ docker exec -i -t ed4 /bin/bash
bash-4.1# ps auxff
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 460 0.2 0.0 11304 1556 ? S 08:20 0:00 /bin/bash
root 468 0.0 0.0 13368 1000 ? R+ 08:20 0:00 \_ ps auxff
root 1 0.0 0.0 19232 1536 ? Ss 08:03 0:00 /sbin/init
root 78 0.0 0.0 10640 624 ? Ss 08:03 0:00 /sbin/udevd -d
root 242 0.0 0.0 66688 1228 ? Ss 08:03 0:00 /usr/sbin/sshd
root 467 0.0 0.0 4060 672 ? Ss 08:20 0:00 /sbin/mingetty /dev/tty[1-6]Tried it with Beaker 1.20.1 and hit the same timeout.
While Beaker was trying to SSH into the container, I attached to it to run ps and noticed that sshd was indeed not running, so I started it:
bash-4.1# service sshd start
Generating SSH1 RSA host key: [ OK ]
Starting sshd: [ OK ]And then Beaker was able to connect, but what called my attention was the first line where it says it's generating the RSA host key. From what I can tell, that's done as part of the image building process, right?
c10l
commented
9 years ago I've also tried it with docker-api 1.15.0 and saw the same results: no sshd running...
electrical
commented
9 years ago Most likely because the older dockerapi gems have an API miss match with the docker api it self?
c10l
commented
9 years ago How does that explain that I have the same results with 1.15.0 and 1.17.0 (which is the latest)?
electrical
commented
9 years ago ah sorry. i miss understood your findings.
electrical
commented
9 years ago Must admit from my side i never ran on 2.6.x kernels for docker.. always been on 3.x ( currently on 3.13.x )
c10l
commented
9 years ago This PR https://github.com/puppetlabs/beaker/pull/626 solves my issue.
madAndroid
commented
9 years ago That does seem like the cleanest option, glad to hear you've made some progress :)
On Mon, Feb 2, 2015 at 4:30 PM, cassiano notifications@github.com wrote:
This PR #626 https://github.com/puppetlabs/beaker/pull/626 solves my issue.
— Reply to this email directly or view it on GitHub https://github.com/puppetlabs/beaker/issues/652#issuecomment-72465936.
anodelman
commented
9 years ago I'll close this out when the PR626 gets merged.
anodelman
commented
9 years ago PR merged, closing.
Beaker is failing on CentOS 6, running on Docker 1.4.1 from epel-testing. It starts the container, but then is unable to ssh in to it.
Beaker verbose output:
I tried the SSH command manually on the host and got this:
This is my nodeset file: