Initial HTTP requests fail with 503 on edge router without HTTP KeepAlive

[tnolet]

As far as I can see, every first HTTP request that does NOT use HTTP KeepAlive to a Vamp managed service fails. This is noticeable even with simple static html pages in a browser.

Reproduce with the following:

DC/OS: 1.9.0 Vamp: 0.9.5-1237-g7000ce10

name: simpleservice:1.0.0
gateways:
  80/http: simpleservice/web
clusters:
  simpleservice:
    services:
      breed:
        name: simpleservice:1.0.0
        deployable: magneticio/simpleservice:1.0.0
        ports:
          web: 3000/http
      scale:
        cpu: 0.2
        memory: 128MB
        instances: 1

deploy the blueprint. Now hit the external gateway with some requests. In a browser you will see 503 on occasion. When the 503's stop, and you have a nice page you just have to wait a bit and it will happen again: the exact time is determined by the http keepalive timeout setting on the server and in the browser.

automatic reproduction can be done with apache bench:

# 100 requests, no concurrency
ab -n 100 -c 1 http://52.166.151.31/
...
Complete requests:      100
Failed requests:        61
...

# 100 requests, no concurrency with keep alive
ab -n 100 -c 1 -k http://52.166.151.31/
...
Failed requests:        0
Keep-Alive requests:    100
...

[olafmol]

It seems this issue appears when running a gateway on port 80. When configuring a gateway to run in a higher range (f.e. 9050) the issue does not appear.

A few possible reasons come to mind:

• how we reload the HAproxy, we add a "hack" to avoid dropped packets: https://github.com/magneticio/vamp-gateway-agent/blob/master/files/usr/local/vamp/haproxy-reload.sh

• how we configure the HAproxy: https://github.com/magneticio/vamp-gateway-agent/blob/master/files/usr/local/vamp/haproxy.basic.cfg

• how we init/run the HAproxy: https://github.com/magneticio/vamp-gateway-agent/blob/master/files/etc/service/vamp-gateway-agent/run and https://github.com/magneticio/vamp-gateway-agent/blob/master/files/etc/service/haproxy/run

• the generated HAproxy config template

My suggestion would be to first remove the zero-downtime reload hack and see if it clears the issue: https://github.com/magneticio/vamp-gateway-agent/blob/73f6935d307418a13b34d8767ae02aa4e77b2c38/files/usr/local/vamp/haproxy-reload.sh#L27

[olafmol]

This might give some pointers to a fix: https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/

[olafmol]

Update: the behaviour happens when defining an external vamp gateway on port 80. Because of virtual host name support we automatically bind to 0.0.0.0 port 80 in our haproxy config template, this creates this 503 issue.

Solutions/workarounds for now:

• create a virtual hostname label in the gateway definition that resolves to the URL used (https://vamp.io/documentation/using-vamp/v0.9.5/virtual-hosts/#custom-virtual-hosts) and keep the vamp gateway itself at a higher port (f.e. 9000 range)

• comment out the bind 0.0.0.0:80 line from the VGA config template, and set the vamp gateway to port 80

[olafmol]

@tnolet can we update our documentation to explain how to make our VGA gateways work on port 80, closing this issue for now.

[tnolet]

We need to make this automatic. Vamp has all the info to make the right decision here and it should for the user.

• filter all port declarations
• if 80 is declared, do not claim 80 twice (i.e. once for the dedicated gateway, once for all vhosts)

or disallow using 80 on a dedicated gateway. I'll update the docs with a note on the workaround.