Can't login in docker when test-kitchen is executed from docker container

[dj-wasabi]

Hi,

We have a Jenkins master running in docker. When a job is started, a jenkins-slave docker container is started and executes everything related in this job in this specific jenkins-slave container.

When running the kitchen converge step, it will create and boot a new docker container, but it can't login:

       Waiting for SSH service on localhost:32844, retrying in 3 seconds
       Waiting for SSH service on localhost:32844, retrying in 3 seconds
       Waiting for SSH service on localhost:32844, retrying in 3 seconds
$$$$$$ [SSH] connection failed, terminating (#<Errno::ECONNREFUSED: Connection refused - connect(2) for 127.0.0.1:32844>)
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
[Pipeline] mail
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 20
Finished: FAILURE

I can see that the docker container is started and I can manually from my host login into the container. But It won't work from this jenkins-slave.

This is my .kitchen.yml file:

driver:
  name: docker

provisioner:
  name: ansible_playbook
  ansible_yum_repo: "https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-6.noarch.rpm"
  hosts: localhost
  require_chef_for_busser: false
  require_ruby_for_busser: true

platforms:
  - name: centos-7.2
    driver_config:
      privileged: true
      provision_command:
        - /usr/bin/systemctl enable sshd
      run_command: /usr/sbin/init

verifier:
  ruby_bindir: '/usr/bin'

suites:
  - name: default

Versions: test-kitchen: 1.9.0 kitchen-docker: 2.4.0 kitchen-ansible: 0.43.1

Any suggestions where I should look at to make this work? Or do you need some more information?

[coderanger]

You would probably have to make sure whatever networking model is being used will work with the expectations of the driver. The driver tries to SSH to $DOCKER_IP:$MAPPED_PORT where the DOCKER_IP is probably localhost in this case. The mapped port would only be valid on the hypervisor though. In this case you probably want to skip the port mapping and talk directly to the container's IP, but there is not currently any code for that.

[ytsarev]

Why not to make it simple as in proposed https://github.com/test-kitchen/kitchen-docker/pull/203 ?

[coderanger]

@ytsarev The issue with that approach is the IP of the hypervisor needs to be fixed or you're going to have a hellacious kitchen config.

[ytsarev]

@coderanger not sure if I understood you correctly but in my case it's not IP of the hypervisor as it is but more or less standard IP of docker0 bridge. Which is pretty generic.

[coderanger]

@ytsarev That very much depends on the kind of networking model you are using. If a static, hardwired address works for you then that's cool, but I wouldn't advertise it as a generic solution.

[ytsarev]

@coderanger I agree, but at least the bridge model is default

[awiddersheim]

Running into this same issue myself. Last post on this thread is pretty old. Anyone found any good solutions to this problem?

[coderanger]

I would highly recommend using kitchen-dokken if possible to people in this situation. It doesn't use SSH so it doesn't have as many issues with networking management.

[awiddersheim]

@coderanger thank you for the recommendation. Project looks interesting. However, looks pretty Chef specific. I'm using kitchen-docker with the kitchen-ansible provisioner. My first impressions make me think I can't do the same with kitchen-dokken.

[coderanger]

Yeah, dokken is Chef-only at this time.

[awiddersheim]

@coderanger to @ytsarev's point I could probably make this work if I could change the host and/or port used when making the SSH connection.

[AaronKalair]

Hello 👋

I was having this issue too so I made this PR - https://github.com/test-kitchen/kitchen-docker/pull/283

If you get a chance to try it out, let me know if it also works for you. You'll need to add use_internal_docker_network: true to your .kitchen.yml file under the driver section

[awiddersheim]

@AaronKalair your patch worked for me. Nice solution IMO and dynamic enough to work in quite a few scenarios I would say.