Jenkins framework crash if inside a docker container in bridge mode

[gdippolito]

I'm experience this issue with the following setup: Jenkins 2.2 (running in docker with bridge network) Mesos-plugin 0.12 Mesos 0.28.1

Apparently the framework keep crashing after starting the connection with the mesos master. The framework is using the IP address of the container which is not reachable from the Mesos master (mesos master is running in a container in a different host).

These are the logs on the mesos master:

I0510 10:32:34.468268    12 master.cpp:2231] Received SUBSCRIBE call for framework 'jenkins-framework' at scheduler-1d3da1d1-aa5d-4c78-a65a-ccb1dbfd5c84@172.17.0.3:45128
I0510 10:32:34.475852    12 master.cpp:2302] Subscribing framework jenkins-framework with checkpointing enabled and capabilities [  ]
I0510 10:32:34.483469     9 hierarchical.cpp:265] Added framework 0ccc0b5b-b484-4821-9ace-5eddda056dcf-0157
I0510 10:32:34.514787    12 master.cpp:5324] Sending 2 offers to framework 0ccc0b5b-b484-4821-9ace-5eddda056dcf-0157 (jenkins-framework) at scheduler-1d3da1d1-aa5d-4c78-a65a-ccb1dbfd5c84@172.17.0.3:45128
I0510 10:32:34.692304     9 master.cpp:2231] Received SUBSCRIBE call for framework 'jenkins-framework' at scheduler-1d3da1d1-aa5d-4c78-a65a-ccb1dbfd5c84@172.17.0.3:45128
I0510 10:32:34.692454     9 master.cpp:2302] Subscribing framework jenkins-framework with checkpointing enabled and capabilities [  ]
I0510 10:32:34.692483     9 master.cpp:2312] Framework 0ccc0b5b-b484-4821-9ace-5eddda056dcf-0157 (jenkins-framework) at scheduler-1d3da1d1-aa5d-4c78-a65a-ccb1dbfd5c84@172.17.0.3:45128 already subscribed, resending acknowledgement
I0510 10:32:35.919252     9 master.cpp:2231] Received SUBSCRIBE call for framework 'jenkins-framework' at scheduler-1d3da1d1-aa5d-4c78-a65a-ccb1dbfd5c84@172.17.0.3:45128
I0510 10:32:35.919471     9 master.cpp:2302] Subscribing framework jenkins-framework with checkpointing enabled and capabilities [  ]
I0510 10:32:35.919534     9 master.cpp:2312] Framework 0ccc0b5b-b484-4821-9ace-5eddda056dcf-0157 (jenkins-framework) at scheduler-1d3da1d1-aa5d-4c78-a65a-ccb1dbfd5c84@172.17.0.3:45128 already subscribed, resending acknowledgement
E0510 10:32:37.481415    13 process.cpp:1958] Failed to shutdown socket with fd 24: Transport endpoint is not connected

I have tried to change the Jenkins URL to the IP address of the VM where docker is running but it seems to make no changes.

If I run the container with networking "host" mode I can successfully register the jenkins framework to the Mesos master.

[vinodkone]

I don't see the "crash" in the logs you posted?

Anyway a mesos frameowrk should advertise a routable IP to the mesos master, otherwise things don't work. You can set LIBPROCESS_ADVERTISE_IP and LIBPROCESS_ADVERTISE_PORT environment variables before starting jenkins master. These values should correspond to the the routable IP address and port, which in this case is the docker host's IP and host port forwarded to the container's port.

[geoffroyjabouley]

How are you starting your jenkins master? Are you using Marathon?

In bridged mode, your docker container shall know its host IP and run with at least 3 binded ports:

1. PORT0 = jenkins http port
2. PORT1 = jenkins jnlp port
3. PORT2 = mesos libprocess port

Then you can set, when starting the container:

export LIBPROCESS_PORT="${PORT2}"
export LIBPROCESS_ADVERTISE_IP="${HOST_IP}"
export LIBPROCESS_ADVERTISE_PORT="${LIBPROCESS_PORT}"

[gdippolito]

By crash I means that the framework attempt to register to the Mesos master and immediately fail.

There is a configuration option in the mesos-plugin section under "Manage Jenkins" called Jenkins URL I thought this option would have set up the correct Jenkins IP address and port to pass to Mesos master. I will try to pass this variable and I will let you know how it goes

[iax7]

I'm running Mesos 0.28.2 and Jenkins 2.7.3 in a docker container with plugin version 0.13.1

I have made a custom Dockerfile for this:

FROM jenkins:latest

USER root
ARG MESOS_VERSION=0.27.0
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv E56151BF && \
    echo "deb http://repos.mesosphere.com/debian jessie main" > /etc/apt/sources.list.d/mesosphere.list && \
    apt-get update && apt-get install -y --force-yes --install-recommends mesos=$MESOS_VERSION\* && \
    apt-get clean
USER jenkins

RUN /usr/local/bin/install-plugins.sh mesos:0.13.1

my Jenkins configuration is this:

This is the result in Mesos 0.28.2:

Jenkins logs:

I'm not using Marathon for this tests. Plain Old docker run:

docker run --name jenkins -p 10001:8080 -p 50001:50000 myjenkins

[lavcraft]

@iax7 you have to use zookeeper url in Mesos master [hostname:port] field

In my example:

[kopax]

Hi, I have the same issue with a jenkins host running in bridged mode.

It was working fine when I was trying in standalone mode.

@geoffroyjabouley should I use the export you just wrote? What about LIBPROCESS_IP here

[sbeaulie]

The key part of the mesos master log is:

at scheduler-1d3da1d1-aa5d-4c78-a65a-ccb1dbfd5c84@172.17.0.3:45128

172.17.0.3 is the private IP range used by your docker container in bridge mode. See ARIN private IP ranges

This is not routable from your mesos master. You have to set the IP (if you are not using FQDNs) in your Jenkins URL to something that is routable from the mesos master. Also note the dynamic port 45128 which is blocked by docker. You'd have to allow the dynamic port range to connect to the container.

[jeschkies]

@gdippolito, were you able to resolve the issue?

[jeschkies]

Closing for lack of response.