Bamboo Not Listening on Port :80 and Not Load Balancing

akamalov commented 9 years ago

Deployed Bamboo. Having problem making it load balance for tutum/hello-world containers. This is how tutum-hello-world containers were deployed:

File: test-multi-nginx-1.json

{
    "id": "/product/web",
    "apps": [
        {
            "id": "tutum-hello-world-1",
            "container": {
                "type": "DOCKER",
                "docker": {
                    "image": "tutum/hello-world",
                    "network": "BRIDGE",
                    "portMappings": [
                        {
                            "containerPort": 80,
                            "hostPort": 0,
                            "protocol": "tcp"
                        }
                    ]
                }
            },
            "instances": 2,
            "cpus": 0.25,
            "mem": 256,
            "uris": [],
            "constraints": [
                [
                    "hostname",
                    "LIKE",
                    "wwserver(104|204).alphalocal.net"
                ]
            ]
        }
    ]
}

Configuration was updated after a defaul rule was created for running tutum/hello-world containers. Here is how configuration looks now:

########################################################################################

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL).
        # ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000

        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

# Template Customization
frontend http-in
        bind *:80

        acl ::product::web::tutum-hello-world-1-aclrule 
        use_backend ::product::web::tutum-hello-world-1-cluster if ::product::web::tutum-hello-world-1-aclrule

        stats enable
        # CHANGE: Your stats credentials
        stats auth admin:admin
        stats uri /haproxy_stats

backend ::product::web::tutum-hello-world-1-cluster
        balance leastconn
        option httpclose
        option forwardfor

        server ::product::web::tutum-hello-world-1-wwserver104.alphalocal.net-5052 wwserver104.alphalocal.net:5052  
        server ::product::web::tutum-hello-world-1-wwserver204.alphalocal.net-8082 wwserver204.alphalocal.net:8082  

##
## map service ports of marathon apps
## ( see https://mesosphere.github.io/marathon/docs/service-discovery-load-balancing.html#ports-assignment ))
## to haproxy frontend port
##
## 
##   
## listen ::product::web::tutum-hello-world-1_10001
##   bind *:10001
##   mode http
##   
##   balance leastconn
##   option forwardfor
##         
##         server ::product::web::tutum-hello-world-1-wwserver104.alphalocal.net-5052 wwserver104.alphalocal.net:5052  
##         server ::product::web::tutum-hello-world-1-wwserver204.alphalocal.net-8082 wwserver204.alphalocal.net:8082  
##   
## 
########################################################################################

Unfortunately, bamboo is not listening on port :80 as it is displayed in config file. I can hit ports :5052 and :8082 just fine and get tutum/hello-world page, but I cannot hit port :80 and get it redirected to one of the tutum/hello-world services.

So, got into bamboo container to see if it is listening on port 80...and it's not!

root@f3fd87261a47:/etc/haproxy# ss -lnt
State       Recv-Q Send-Q                                                                      Local Address:Port                                                                        Peer Address:Port 
LISTEN      0      128                                                                                    :::8000                                                                                  :::*     
root@f3fd87261a47:/etc/haproxy#

Why wouldn't it listen on port 80 ? It's in a configuration...

Tried to restart haproxy:

root@f3fd87261a47:/etc/haproxy# service haproxy restart
 * Restarting haproxy haproxy
[ALERT] 161/121915 (382) : parsing [/etc/haproxy/haproxy.cfg:42] : error detected while parsing ACL '::product::web::tutum-hello-world-1-aclrule' : missing fetch method in ACL expression ''.
[ALERT] 161/121915 (382) : parsing [/etc/haproxy/haproxy.cfg:43] : error detected while parsing switching rule : no such ACL : '::product::web::tutum-hello-world-1-aclrule'.
[ALERT] 161/121915 (382) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
[ALERT] 161/121915 (382) : Fatal errors found in configuration.
   ...fail!

root@f3fd87261a47:/etc/haproxy#

Then I went ahead and removed an ACL entry:

acl ::product::web::tutum-hello-world-1-aclrule

Once I did remove and restart haproxy, errors have gone away and it started listening on port :80 However, traffic is still not hitting backend containers - tutum/hello-world - no load balancing).

What am I doing wrong?

Thanks!!

Alex

rasputnik commented 9 years ago

The 'acl' line in the config is incomplete. Perhaps bamboo can't load the ACL rules from zookeeper? What does that part of your template look like?

akamalov commented 9 years ago

Thansk for reply rasputnik (nice nick! I hope there is "rasputnitsa" somewhere around :)

Here is my bamboo.json

{
    "Marathon": {
        "Endpoint": "http://10.9.158.219:8080,http://10.9.158.220:8080,http://10.9.158.221:8080"
    },
    "Bamboo": {
        "Endpoint": "http://10.9.159.75:8000",
        "Zookeeper": {
            "Host": " 10.9.158.219:2181,10.9.158.220:2181,10.9.158.221:2181",
            "Path": "/marathon-haproxy/state",
            "ReportingDelay": 5
        }
    },
    "HAProxy": {
        "TemplatePath": "/var/lib/bamboo/etc/haproxy_template.cfg",
        "OutputPath": "/etc/haproxy/haproxy.cfg",
        "ReloadCommand": "haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf $(cat /var/run/haproxy.pid)"
    },
    "StatsD": {
        "Enabled": true,
        "Host": "localhost:8125",
        "Prefix": "bamboo-server.production.n1."
    }
}

Here is my haproxy_template.cfg

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL).
        # ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000

        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

# Template Customization
frontend http-in
        bind *:80
        {{ $services := .Services }}
        {{ range $index, $app := .Apps }} {{ if hasKey $services $app.Id }} {{ $service := getService $services $app.Id }}
        acl {{ $app.EscapedId }}-aclrule {{ $service.Acl}}
        use_backend {{ $app.EscapedId }}-cluster if {{ $app.EscapedId }}-aclrule
        {{ else }}

        # This is the default proxy criteria
        acl {{ $app.EscapedId }}-aclrule path_beg -i {{ $app.Id }}
        use_backend {{ $app.EscapedId }}-cluster if {{ $app.EscapedId }}-aclrule
        {{ end }} {{ end }}

        stats enable
        # CHANGE: Your stats credentials
        stats auth admin:admin
        stats uri /haproxy_stats

{{ range $index, $app := .Apps }} {{ if $app.Env.BAMBOO_TCP_PORT }}
listen {{ $app.EscapedId }}-cluster-tcp :{{ $app.Env.BAMBOO_TCP_PORT }}
        mode tcp
        option tcplog
        balance roundrobin
        {{ range $page, $task := .Tasks }}
        server {{ $app.EscapedId}}-{{ $task.Host }}-{{ $task.Port }} {{ $task.Host }}:{{ $task.Port }} {{ if $app.HealthCheckPath }} check inter 30000 {{ end }} {{ end }}
{{ end }}
backend {{ $app.EscapedId }}-cluster{{ if $app.HealthCheckPath }}
        option httpchk GET {{ $app.HealthCheckPath }}
        {{ end }}
        balance leastconn
        option httpclose
        option forwardfor
        {{ range $page, $task := .Tasks }}
        server {{ $app.EscapedId}}-{{ $task.Host }}-{{ $task.Port }} {{ $task.Host }}:{{ $task.Port }} {{ if $app.HealthCheckPath }} check inter 30000 {{ end }} {{ end }}
{{ end }}

##
## map service ports of marathon apps
## ( see https://mesosphere.github.io/marathon/docs/service-discovery-load-balancing.html#ports-assignment ))
## to haproxy frontend port
##
## {{ range $index, $app := .Apps }}
##   {{ range $serviceIndex, $servicePort := $app.ServicePorts }}
## listen {{ $app.EscapedId }}_{{ $servicePort }}
##   bind *:{{ $servicePort }}
##   mode http
##   {{ if $app.HealthCheckPath }}
##   # option httpchk GET {{ $app.HealthCheckPath }}
##   {{ end }}
##   balance leastconn
##   option forwardfor
##         {{ range $page, $task := $app.Tasks }}
##         server {{ $app.EscapedId }}-{{ $task.Host }}-{{ index $task.Ports $serviceIndex }} {{ $task.Host }}:{{ index $task.Ports $serviceIndex }} {{ if $app.HealthCheckPath }} check inter 30000 {{ end }} {{ end }}
##   {{ end }}
## {{ end }}

rasputnik commented 9 years ago

(thanks, rasputina says hi :) )

do you have acls defined for routing the apps (in zookeeper, exposed via the bamboo web ui at 8000)? Maybe it'd be worth wrapping that acl line with something like

 {{ if $service.Acl }}
     # acl bits get templated here
 {{ end }}

(I don't use acls yet myself - my setup is the 'map service ports' bit commented out underneath )

akamalov commented 9 years ago

Thanks for your reply rasputnink. I decided to go your way of mapping service ports. It maps the ports beautifully, however, docker doesn't seem to be mapping the port (i.e., listens on port 10001 for tutum/hello-world on a container port, but won't map it on host port) :(

akamalov commented 9 years ago

Here is how template looks now:

haproxy_template.cfg

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL).
        # ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000

        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

# Template Customization
#frontend http-in
#        bind *:80
#        {{ $services := .Services }}
#        {{ range $index, $app := .Apps }} {{ if hasKey $services $app.Id }} {{ $service := getService $services $app.Id }}
#        acl {{ $app.EscapedId }}-aclrule {{ $service.Acl}}
#        use_backend {{ $app.EscapedId }}-cluster if {{ $app.EscapedId }}-aclrule
#        {{ else }}
#
#        # This is the default proxy criteria
#        acl {{ $app.EscapedId }}-aclrule path_beg -i {{ $app.Id }}
#        use_backend {{ $app.EscapedId }}-cluster if {{ $app.EscapedId }}-aclrule
#        {{ end }} {{ end }}
#
#        stats enable
#        # CHANGE: Your stats credentials
#        stats auth admin:admin
#        stats uri /haproxy_stats
#
#{{ range $index, $app := .Apps }} {{ if $app.Env.BAMBOO_TCP_PORT }}
#listen {{ $app.EscapedId }}-cluster-tcp :{{ $app.Env.BAMBOO_TCP_PORT }}
#        mode tcp
#        option tcplog
#        balance roundrobin
#        {{ range $page, $task := .Tasks }}
#        server {{ $app.EscapedId}}-{{ $task.Host }}-{{ $task.Port }} {{ $task.Host }}:{{ $task.Port }} {{ if $app.HealthCheckPath }} check inter 30000 {{ end }} {{ end }}
#{{ end }}
#backend {{ $app.EscapedId }}-cluster{{ if $app.HealthCheckPath }}
#        option httpchk GET {{ $app.HealthCheckPath }}
#        {{ end }}
#        balance roundrobin
#        option httpclose
#        option forwardfor
#        {{ range $page, $task := .Tasks }}
#        server {{ $app.EscapedId}}-{{ $task.Host }}-{{ $task.Port }} {{ $task.Host }}:{{ $task.Port }} {{ if $app.HealthCheckPath }} check inter 30000 {{ end }} {{ end }}
#{{ end }}

##
## map service ports of marathon apps
## ( see https://mesosphere.github.io/marathon/docs/service-discovery-load-balancing.html#ports-assignment ))
## to haproxy frontend port
##
 {{ range $index, $app := .Apps }}
   {{ range $serviceIndex, $servicePort := $app.ServicePorts }}
 listen {{ $app.EscapedId }}_{{ $servicePort }}
   bind *:{{ $servicePort }}
   mode http
   {{ if $app.HealthCheckPath }}
   # option httpchk GET {{ $app.HealthCheckPath }}
   {{ end }}
   balance roundrobin
   option forwardfor
         {{ range $page, $task := $app.Tasks }}
         server {{ $app.EscapedId }}-{{ $task.Host }}-{{ index $task.Ports $serviceIndex }} {{ $task.Host }}:{{ index $task.Ports $serviceIndex }} {{ if $app.HealthCheckPath }} check inter 30000 {{ end }} {{ end }}
   {{ end }}
 {{ end }}

#######################################################################

Deployed bamboo as a container:

wwserver104 etc # docker run -d -P --name bamboo --dns=160.254.96.245 -v /var/lib/bamboo/etc:/var/lib/bamboo/etc -e CONFIG_PATH=/var/lib/bamboo/etc/bamboo.json -e BAMBOO_DOCKER_AUTO_HOST=true akamalov/qubit_bamboo

#######################################################################

Here is how my /etc/haproxy/haproxy.cfg looks like on a running bamboo container:

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL).
        # ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000

        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

# Template Customization
#frontend http-in
#        bind *:80
#        
#          
#        acl ::product::web::tutum-hello-world-1-aclrule 
#        use_backend ::product::web::tutum-hello-world-1-cluster if ::product::web::tutum-hello-world-1-aclrule
#         
#
#        stats enable
#        # CHANGE: Your stats credentials
#        stats auth admin:admin
#        stats uri /haproxy_stats
#
# 
#backend ::product::web::tutum-hello-world-1-cluster
#        balance roundrobin
#        option httpclose
#        option forwardfor
#        
#        server ::product::web::tutum-hello-world-1-wwserver104.alphalocal.net-5052 wwserver104.alphalocal.net:5052  
#        server ::product::web::tutum-hello-world-1-wwserver204.alphalocal.net-8082 wwserver204.alphalocal.net:8082  
#

##
## map service ports of marathon apps
## ( see https://mesosphere.github.io/marathon/docs/service-discovery-load-balancing.html#ports-assignment ))
## to haproxy frontend port
##

 listen ::product::web::tutum-hello-world-1_10001
   bind *:10001
   mode http

   balance roundrobin
   option forwardfor

         server ::product::web::tutum-hello-world-1-wwserver104.alphalocal.net-5052 wwserver104.alphalocal.net:5052  
         server ::product::web::tutum-hello-world-1-wwserver204.alphalocal.net-8082 wwserver204.alphalocal.net:8082

Bamboo is listening on port 10001:

wwserver104 etc # docker exec -ti bamboo ss -lnt  
State      Recv-Q Send-Q                                                                    Local Address:Port                                                                      Peer Address:Port 
LISTEN     0      128                                                                                   *:10001                                                                                *:*     
LISTEN     0      128                                                                                  :::8000                                                                                :::*     
wwserver104 etc #

However, no love on a host port mapping part:

wwserver104 etc # docker ps -a
CONTAINER ID        IMAGE                          COMMAND                CREATED             STATUS              PORTS                                                                                                                                                                NAMES
f50c99c938d0        akamalov/qubit_bamboo:latest   "/bin/sh -c /run.sh"   5 minutes ago       Up 5 minutes        0.0.0.0:32774->80/tcp, 0.0.0.0:32775->8000/tcp                                                                                                                       bamboo                                       
6662597dcdd8        tutum/hello-world:latest       "\"/bin/sh -c 'php-f   42 hours ago        Up 42 hours         0.0.0.0:5052->80/tcp                                                                                                                                                 mesos-a1263108-1e11-4384-b4f5-38315d837231

Not mapping port 10001 running on a container to host port :(

rasputnik commented 9 years ago

Afraid I don't run my bamboos in docker, but I'd guess you'd have to explicitly map those ports over in your docker run (I'd guess @activars runs all his services on a single port with the acl magic so he only needs to map the single port over).

akamalov commented 9 years ago

yes. I guess.However, this will be a major bummer. Consul/Registrator has a smoother integration. For the ACL stuff, it looks like it is an outstanding bug:

https://github.com/QubitProducts/bamboo/issues/119

akamalov commented 9 years ago

Is this project dead? Is bug #119 being addressed ?

j1n6 commented 9 years ago

Thanks for reminding me that bug. Haproxy configuration should be validated before reload. I will prioritise the fix.

We are making a bigger release that comes with more features.

On 26 Jun 2015, at 17:26, akamalov notifications@github.com wrote:

Is this project dead? Is bug #119 being addressed ?

— Reply to this email directly or view it on GitHub.

akamalov commented 9 years ago

Thanks Jing!

akamalov commented 9 years ago

Hey Jing, any ETA on when the new release will be out ? When can we expect it ?

Thanks a ton,

Alex

j1n6 commented 9 years ago

Probably late this week.

On 29 June 2015 at 13:54, akamalov notifications@github.com wrote:

Hey Jing, any ETA on when the new release will be out ? When can we expect it ?

Thanks a ton,

Alex

— Reply to this email directly or view it on GitHub https://github.com/QubitProducts/bamboo/issues/135#issuecomment-116648404 .

akamalov commented 9 years ago

Hey Jing, will the new cut be released sometimes this week ?

Thanks!!

QubitProducts / bamboo

Bamboo Not Listening on Port :80 and Not Load Balancing #135