search

bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
22.7k stars 3.98k forks source link

com.google.devtools.build.lib.remote.http.HttpException keeps returning 499 despite ridiculously long client timeout #22847

Open grepwood opened 2 weeks ago

grepwood commented 2 weeks ago

I'm using Bazel 6.5.0 with nginx 1.24.0 with WebDAV support as a remote cache.

I've been getting com.google.devtools.build.lib.remote.http.HttpException returning HTTP 499 every time that Bazel is trying to download bigger files from the cache, like Docker image layers. By bigger I mean, each individual layer isn't bigger than 400MB.

I thought that I'll deal with this issue. So first, I've read this: https://stackoverflow.com/questions/12973304/possible-reason-for-nginx-499-error-codes So it's not a remote cache issue, it's the local Bazel not being very patient.

So I read the Bazel docs, specifically this section https://bazel.build/reference/command-line-reference#flag--remote_timeout So now I think I get it, so I add this to my .bazelrc:

build --remote_timeout=9001s
run --remote_timeout=9001s

However, I still get timeouts. What can I do to get rid of this non-fatal error?

UPDATE 1: These timeouts are different. Now I'm getting:

io.netty.handler.ssl.SslHandshakeTimeoutException: handshake timed out after 1000ms

I have no idea what am I supposed to do now :/

meisterT commented 1 week ago

Since the SSL handshake failed, on what type of machine do you run your remote cache? Is it underpowered?