varnishcache / varnish-cache

Varnish Cache source code repository
https://www.varnish-cache.org
Other
3.65k stars 374 forks source link

Unix socket backend leads to high cpu usage in some cases #4203

Open jogoossens opened 1 week ago

jogoossens commented 1 week ago

Expected Behavior

No unexplained high cpu load when using unix socket on backend.

Current Behavior

We had some longtime intermittent high cpu usage with varnish software. We spent a lot of hours troubleshooting and searching, finally we changed our backend to port 8080 and the high cpu usage stayed away.

When we (re)start varnish, initially there is no high load at all, but after some time (one hour or a few hours) the vcache process experiences high cpu usage.

Possible Solution

No response

Steps to Reproduce (for bugs)

We now switched back and forth between the following settings, and this change alone makes it "stable" versus "high cpu usage".

High cpu usage

backend default_backend {
    .path = "/var/lib/nginx/default.sock";
    .connect_timeout        = 5s;
    .first_byte_timeout     = 600s;
}

No high cpu usage

backend default_backend {
    .host = "127.0.0.1";
    .port = "8080";
    .connect_timeout        = 5s;
    .first_byte_timeout     = 600s;
}

Context

We tried perf top and see high cpu usage in copying data, but it's not easy to see the source so far.

We have this problem for a long time (at least with varnish 7.0 and all following versions, probably 6.x too but unsure)

Varnish Cache version

varnishd 7.6.0

Operating system

Debian 12

Source of binary packages used (if any)

No response

nigoroll commented 1 week ago

We'll look at it, but it would be really helpful if you could get some perf data, yes. Or a reproducer. Also could you please report which VMODs you use, if any?