Varnish probe fails with PROXY v2

[idl0r]

Hi,

we use go-proxyproto for some stub backends for testing purpose and it fails when we tell Varnish to use v2 while v1 works. The connection between Varnish and HAProxy works fine so I suspect something's wrong within go-proxyproto. Here's anything required for a testcase:

package main
import (
    "fmt"
    "net"
    "net/http"
    "github.com/pires/go-proxyproto"
)
func main() {
    handler := func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
    }
    srv := &http.Server{
        Handler: http.HandlerFunc(handler),
        Addr:    fmt.Sprintf("127.0.0.1:%d", 14240),
    }
    ln, err := net.Listen("tcp", srv.Addr)
    if err != nil {
        fmt.Println(err)
    }

    policyFunc := func(upstream net.Addr) (proxyproto.Policy, error) { return proxyproto.REQUIRE, nil }
    proxyListener := &proxyproto.Listener{Listener: ln, Policy: policyFunc}
    srv.Serve(proxyListener) //nolint:errcheck
}

go build test.go
./test

Varnish 6.4.0 (or later): example.vcl:

vcl 4.1;
backend default {
    .host = "127.0.0.1";
    .port = "14240";
    .proxy_header = 1;

            .probe = {
                .request =
                        "GET /test HTTP/1.1"
                        "Host: www.example.com"
                        "User-Agent: varnish"
                        "Connection: close";
                .interval = 5s;
                .timeout = 3s;
                .window = 8;
                .threshold = 3;
                .expected_response = 200;
        }

}

varnishd -f /etc/varnish/example.vcl -F

varnishlog -n /var/lib/varnish/$(hostname) -g raw -i Backend_health

Now varnishlog should report:

0 Backend_health - default Still healthy 4---X-RH 4 3 8 0.000505 0.000226 "HTTP/1.1 200 OK"

Now change .proxy_header from 1 to 2 in example.vcl:

    .proxy_header = 2;

And re-run the above testcase. varnishlog will now log:

0 Backend_health - default Still sick 4---X-R- 0 3 8 0.000322 0.000000 "HTTP/1.1 400 Bad Request"

So the probe fails. Looking into a tcpdump what it does is to send a QUIT via PROXYv2 before doing the probe request and I suspected that's the reason for the 400.

This might be related: https://github.com/kubernetes/kubernetes/issues/57250#issuecomment-475081213

[pires]

What version of the lib are you using?

[pires]

Also, have you tried different policies? Right now you're setting REQUIRE:

    // REQUIRE connection to send PROXY header, reject if not present
    // Note: even though the first read on the connection returns an error if
    // a PROXY header is not present, subsequent reads do not. It is the task
    // of the code using the connection to handle that case properly.

[idl0r]

0.3.2. We tried without REQUIRE at first.

[pires]

Can you share more about this QUIT message? I can't find references to this in the spec.

[idl0r]

I have no idea to be honest. That's just what I found by googling for similar issues.

[pires]

Can you please try with v0.4.0 and let me know? It addresses #60 (HAProxy) but maybe addresses your use-case as well.

[idl0r]

It still fails. Are you able to reproduce it the way I described above?

[pires]

Haven't tried, yet, I'm sorry. Maybe during the weekend.

[pires]

I haven't checked yet, sorry. Lots of other things taking priority. Anyways, I did a bit more research and it all comes back to this. Whatever this QUIT message is, it's an invalid HTTP request and/or PROXY header so HTTP 400 sounds about right.

I'm still leaving this open for others' reference and maybe I'll find time to reproduce and investigate.

[idl0r]

I'm not sure where it comes from and if that really is the problem. I couldn't find it within the Varnish sources so far. So I might be wrong.

[idl0r]

I just tested it again with Varnish 6.5.1 and 6.4.0 and it seems to work now. As of 0.4.1 it works. It's probably related to 70665b5dc439d3894d793050feec5331c4ea608f

[pires]

Sweet! Can we close then?

[idl0r]

Yup. Thanks!