Saint mode : std.healthy return true even if backend is in blacklist.

[Arnall]

Hello, in vcl_hit i use (std.healthy(req.backend_hint)) in conditional block to extend grace mode if backend is down. It works well if the backend is really down, but if the backend return a status code > 500 :

sub vcl_backend_response {
  if (beresp.status >= 500) {
        saintmode.blacklist(20s);
    return (retry);
   }

i blacklist the backend, but (std.healthy(req.backend_hint)) continue to return true for the specific request. Don't know if it's intentend to work that way, if it does is there any way to have the saintmode status for a backend (in blacklist or not)

Thanks.

[fgsch]

Are you assigning a director to backend_hint? It might be that some backends are blacklisted but not others.

Might be worth sharing your configuration.

[Arnall]

Hello,

backend be_api_dailymotion {
    .host="172.16.0.100";
    .port="8080";
    .probe = {
        .request =
            "GET /videos?channel=xxxxxxxxxx&limit=10 HTTP/1.1"
            "Host: api.dailymotion.com"
            "Connection: close";           
        .timeout = 1s;
        .interval = 5s;
        .window = 1;
        .threshold = 1;
    }   
}

sub vcl_init {
new sm_api_dailymotion = saintmode.saintmode(be_api_dailymotion,5);
    new d_api_dailymotion = directors.round_robin();
    d_api_dailymotion.add_backend(sm_api_dailymotion.backend());
}

sub vcl_recv {
if (req.http.host == "api.dailymotion.com"){
        set req.backend_hint = d_api_dailymotion.backend();
        unset req.http.Cookie;
    }
}

sub vcl_hit {   
    if (obj.ttl >= 0s) {
        // A pure unadultered hit, deliver it       
        return (deliver);
    }
    if  ( (std.healthy(req.backend_hint)) ) {
        if (obj.ttl + obj.grace > 0s) {
            // Object is in grace, deliver it
            // Automatically triggers a background fetch            
            return (deliver);
        }
    }
    else {
        // no healthy backend available, deliver old version for obj.keep period
        if (obj.ttl + obj.grace + obj.keep > 0s) {      
            return (deliver);           
        }
    }
    // fetch & deliver once we get the result   
    return (miss);
}

As you can see there's only one backend in the director. Thanks.

[daghf]

Largely due to some limitations with regards to directors in Varnish, the saintmode director does not have access to the hash value for a request in any of the client-side vcl subroutines.

The hash value is required as it is the identifying key used for checking if an object is on the trouble list or not. When this is not available, the saintmode director will fall back to the health probe of the underlying director/backend.

In short, the saintmode vmod will not be able to inspect the trouble list in vcl_hit.

(reminder to self: we should doc this)

[Arnall]

Thank you for your answer.

[fgsch]

@daghf can't we use req->digest?

[daghf]

@fgsch we don't even have a req to look at in the healthy function.

We could possibly introduce a separate vmod function that could be invoked explicitly (i.e. not via std.healthy()) to sidestep this - I haven't really looked at it.

[Arnall]

if you do it someday i'll take it :)

[fgsch]

@daghf duh! we should probably change that.

[dridi]

Reopening, it looks like we forgot to get around this one because discussions occurred after it got closed. Let's discuss it on Monday.