Configure file of haproxy could not be updated with bamboo

[CptJason]

I have buit an image named bamboo:v1. with the Dockerfile from github website(https://github.com/QubitProducts/bamboo/blob/master/Dockerfile) .

I setup the network mode to HOST MODE, so that I could directly to use the IP address of this physical host. The following command was to start bamboo: docker run --net=host -t -i --rm -v /home/wowotuan/config/haproxy_template.cfg:/opt/go/src/github.com/QubitProducts/bamboo/config/haproxy_template.cfg -e MARATHON_ENDPOINT=http://ip:8080,http://ip1:8080,http://ip2:8080 -e BAMBOO_ENDPOINT=http://0.0.0.0:8000 -e BAMBOO_ZK_HOST=ip:2181,ip1:2181,ip2:2181 -e BAMBOO_ZK_PATH=/bamboo -e BIND=":8000" -e CONFIG_PATH="config/production.example.json" -e BAMBOO_DOCKER_AUTO_HOST=true bamboo:v1

Once I started bamboo, all of the docker apps on the marathon could be updated into the configure file of haproxy. After that, I tried using different template files of haproxy to test the functionality of bamboo respectively , one was from bamboo github website, the other was made by myself, but All failed. The configure file of haproxy wouldn't be updated in purpose, when I scaled the app or newed an app on marathon administrative website.

The following section was configure template file of haproxy made by myself.

haproxy_template.cfg
global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL).
        # ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000

        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http
#
# map service ports of marathon apps
# ( see https://mesosphere.github.io/marathon/docs/service-discovery-load-balancing.html#ports-assignment ))
# to haproxy frontend port
#
{{ range $index, $app := .Apps }}
   {{ range $serviceIndex, $servicePort := $app.ServicePorts }}
   listen {{ $app.EscapedId }}_{{ $servicePort }}
   bind *:{{ $app.Env.BAMBOO_TCP_PORT }}
   mode tcp
   {{ if $app.HealthCheckPath }}
#   # option httpchk GET {{ $app.HealthCheckPath }}
   {{ end }}
   balance leastconn
   option forwardfor
         {{ range $page, $task := $app.Tasks }}
         server {{ $app.EscapedId }}-{{ $task.Host }}-{{ index $task.Ports $serviceIndex }} {{ $task.Host }}:{{ index $task.Ports $serviceIndex }} {{ if $app.HealthCheckPath }} check inter 30000 {{ end }} {{ end }}
   {{ end }}
 {{ end }}
# Template Customization
        stats enable
        # CHANGE: Your stats credentials
        stats auth admin:admin
        stats uri /haproxy_stats

The following setion was running log of bamboo:

/usr/lib/python2.7/dist-packages/supervisor/options.py:295: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
  'Supervisord is running as root and it is searching '
2015-06-10 09:32:18,883 CRIT Supervisor running as root (no user in config file)
2015-06-10 09:32:18,883 WARN Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
2015-06-10 09:32:18,912 INFO RPC interface 'supervisor' initialized
2015-06-10 09:32:18,912 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2015-06-10 09:32:18,912 INFO supervisord started with pid 9

2015-06-10 09:32:25,749 INFO spawned: 'bamboo' with pid 12
2015-06-10 09:32:25,762 DEBG 'bamboo' stdout output:
2015/06/10 09:32:25 Using environment override MARATHON_ENDPOINT=http://ip:8080,http://ip1:8080,http://ip2:8080
2015/06/10 09:32:25 Using environment override BAMBOO_ENDPOINT=http://0.0.0.0:8000
2015/06/10 09:32:25 Using environment override BAMBOO_ZK_HOST=ip:2181,ip1:2181,ip2:2181
2015/06/10 09:32:25 Using environment override BAMBOO_ZK_PATH=/bamboo
2015/06/10 09:32:25 Environment variable not set: STATSD_ENABLED
2015/06/10 09:32:25 Starting update loop

2015-06-10 09:32:25,769 DEBG 'bamboo' stdout output:
2015/06/10 09:32:25 bamboo_startup => 2015-06-10T09:32:25Z
2015/06/10 09:32:25 Queuing an haproxy update.

2015-06-10 09:32:26,294 DEBG 'bamboo' stdout output:
[martini] listening on :8000 (development)

2015-06-10 09:32:26,417 DEBG 'bamboo' stdout output:
2015/06/10 09:32:26 Exec cmd: haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf $(cat /var/run/haproxy.pid)

2015-06-10 09:32:26,426 DEBG 'bamboo' stdout output:
2015/06/10 09:32:26 HAProxy: Configuration updated

2015-06-10 09:32:30,004 INFO success: bamboo entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-06-10 09:33:36,682 DEBG 'bamboo' stdout output:
2015/06/10 09:33:36 read tcp ip1:2181: i/o timeout

2015-06-10 09:33:46,814 DEBG 'bamboo' stdout output:
2015/06/10 09:33:46 read tcp ip2:2181: i/o timeout

2015-06-10 09:34:07,088 DEBG 'bamboo' stdout output:
2015/06/10 09:34:07 read tcp ip:2181: i/o timeout

[timoreimann]

One thing I'm noticing is that Bamboo apparently produces i/o timeout errors for the Zookeeper connections. Are you sure the networking is working properly? Can you docker exec into the container and verify that telneting to the ZK port on your host machine works?

[akamalov]

I had the same issue. What I did for a workaround is to click on update ACL rules (WEB UI) which forced the update of the configuration. However, yes , you're right, bamboo does not update entries dynamically as you scale (as does Consul + Registrator + HAProxy)

[CptJason]

@timoreimann I am sure that Zookeeper Cluster works fine. The connect to zookeeper from container is ok.

@akamalov I had the same issue.

[CptJason]

We made marathon have a correct http_endpoints option, when it started. And then, the problem gone.

java -Djava.library.path=/usr/local/lib:/usr/lib:/usr/lib64 -Djava.util.logging.SimpleFormatter.format=%2$s%5$s%6$s%n -Xmx512m -cp /usr/bin/marathon mesosphere.marathon.Main --zk zk://ip:2181,ip1:2181,ip2:2181/marathon --master zk://ip:2181,ip1:2181,ip2:2181/mesos --hostname ip --event_subscriber http_callback --http_endpoints http://bambooip:8000/api/services

Great.