Unable to connect on same host

[metal3d]

Hi, and first thanks a lot for your docker gitlab images. I run them since 14 month without to have to complain :)

I've got a problem to run ci-multi-runner on the same host. I've got this output

gitlab-ci-multi-runner 1.1.3 (a470667)
WARNING: image is not supported by selected executor and shell
Using Shell executor...
Running on 0e608eb62373...
Cloning repository...
Cloning into '/home/gitlab_ci_multi_runner/data/builds/f7eeb9e6/0/metal3d/test-ci'...
fatal: unable to access 'https://gitlab-ci-token:xxxxxx@git.develipsum.com/metal3d/test-ci.git/': Failed to connect to git.develipsum.com port 443: Connection refused

ERROR: Build failed: exit status 1

Actually git.develipsum.com is served by a reverse proxy. I can access it outside and on my host. But not inside the "runner" container.

This is my docker-compose that runs:

GitlabCIMultiRunner:
  image: sameersbn/gitlab-ci-multi-runner:1.1.3-1
  volumes:
    - ./data:/home/gitlab_ci_multi_runner/data
    - /var/run/docker.sock:/var/run/docker.sock
  environment:
    - CI_SERVER_URL=http://gitlab/ci
    - RUNNER_TOKEN=XXXXXXXXXXXXXXXXx
    - RUNNER_DESCRIPTION=Docker CI runner
    - RUNNER_EXECUTOR=docker
  restart: always
  external_links:
    - gitlab

"gitlab" link is my gitlab 8.6.7 (your image) and the runner is correctly registered.

CONTAINER ID        IMAGE                                      COMMAND                  CREATED             STATUS              PORTS                                                NAMES
0e608eb62373        sameersbn/gitlab-ci-multi-runner:1.1.3-1   "/sbin/entrypoint.sh"    10 minutes ago      Up 10 minutes                                                            gitlabrunner_GitlabCIMultiRunner_1
5396398c5a12        sameersbn/gitlab:8.6.7                     "/sbin/entrypoint.sh "   14 minutes ago      Up 14 minutes       0.0.0.0:22->22/tcp, 443/tcp, 0.0.0.0:10880->80/tcp   gitlab

Note that:

docker exec -it gitlabrunner_GitlabCIMultiRunner_1 bash
root@0e608eb62373:/home/gitlab_ci_multi_runner# ping git.develipsum.com
PING gitlab (172.17.0.1) 56(84) bytes of data.
64 bytes from gitlab (172.17.0.1): icmp_seq=1 ttl=64 time=0.214 ms
64 bytes from gitlab (172.17.0.1): icmp_seq=2 ttl=64 time=0.171 ms

As you can see, actually, runner takes the private network address and not the public one. It resolves the name to "gitlab" that is my container name. So I imagine that the runner tries to connect to 172.17.0.1:443 that is not served by nginx reverse proxy (wich uses validated certificate)

The connection is refused.

[metal3d]

OK ! Seeing #4 issue I understand that the data volume was not erased and the whole modification I made was not refreshed.

The solution is way simple. I needed to add a extra_host. This is what I set:

go-runner:
  image: sameersbn/gitlab-ci-multi-runner:1.1.3-1
  volumes:
    - ./data:/home/gitlab_ci_multi_runner/data
    - /var/run/docker.sock:/var/run/docker.sock
  environment:
    - CI_SERVER_URL=https://git.develipsum.com/ci
    - RUNNER_TOKEN=XXXXXXXXXX
    - RUNNER_DESCRIPTION=Golang CI runner
    - RUNNER_EXECUTOR=docker
    - DOCKER_IMAGE=metal3d/go:1.6
  restart: always
  extra_hosts:
    - "git.develipsum.com:195.154.76.142"

That way: git.develipsum.com is resolved by /etc/hosts correspondance

But... now... the runner cannot access my gitlab instance saying:

couldn't execute POST against https://git.develipsum.com/ci/api/v1/runners/register.json: Post https://git.develipsum.com/ci/api/v1/runners/register.json: dial tcp 195.154.76.142:443: getsockopt: no route to host

Note that if I add again "external_link" to "gitlab", my extra_host is not used because (inside the runner container) "/etc/hosts" has got 2 entries:

• 195.154.76.142 git.develipsum.com
• 172.17.1.2 gitlab git.develipsum.com

And so, the local one is used by default, so git.develipsum.com is resolved to 172.17.1.2 that is not ok.

To be precise: the problem is that the runner takes the "public" repository url that cannot be resovled inside the docker network.

[metal3d]

I just find the problem and I open another issue closing this one

[jeremyzahner]

@metal3d Did you find a valuable solution for this issue?

[metal3d]

Hi, yes. Sorry to not have explained this sooner.

Ok my problem is that "git.develipsum.com" is served by an external container (nginx), and local docker network cannot hit my nginx by using "git.develipsum.com" so I did a "trick". See the "links" section and CI_SERVER_URL also:

go-runner:
  image: sameersbn/gitlab-ci-multi-runner:1.1.3-1
  volumes:
    - ./data:/home/gitlab_ci_multi_runner/data
    - /var/run/docker.sock:/var/run/docker.sock
  environment:
    - CI_SERVER_URL=http://gitlab/ci
    - RUNNER_TOKEN=XXXXXXXXXX
    - RUNNER_DESCRIPTION=Golang CI runner
    - RUNNER_EXECUTOR=docker
    - DOCKER_IMAGE=metal3d/go:1.6
  restart: always
  links:
    - nginx_nginx_1:git.develipsum.com
    - gitlab

That way, "git.develipsum.com" is resolved using "internal docker ip" and hits nginx. CI_RUNNER is using "gitlab" container name to contact.

But runners jobs want to hit "git.develipsum.com" to "git pull" the project. So ! After runner is up, I changed toml file to add links the same way:

links = ["nginx_nginx_1:git.develipsum.com"]

That way docker runner jobs can resolve "git.develipsum.com" hitting "nginx" with the docker local ip and not the public one.

[metal3d]

That means that runner dockerfile should probably allows to create "links" that are inserted inside the toml file without the need to change it manually...

@sameersbn ? you're ok with the idea ? Maybe I can provide a PR (if I find the time)