bazel fails with Failed to parse downloader config

evis commented 7 months ago

Description of the bug:

During April, we encounter such a problem three times (on different machines): any bazel command fails locally for some developer with:

% bazel build //...
INFO: Invocation ID: 40c6fcdb-de05-4132-99f6-80ab60643974
ERROR: Failed to parse downloader config: bazel_downloader.cfg

Every time, it works good for everyone else, so, looks like a local problem.

At the end of March, we upgraded Bazel from 6.2.1 to 7.1.1.

Workaround that we found: restart Macbook.

Relevant part of our .bazelrc:

build --experimental_downloader_config=bazel_downloader.cfg
sync --experimental_downloader_config=bazel_downloader.cfg
query --experimental_downloader_config=bazel_downloader.cfg

bazel_downloader.cfg:

rewrite (github.com/.*) https://s3.mds.yandex.net/bazel-build/mirror/$1

block github.com

Any ideas, how to fix it? Or maybe some tips, how to debug the problem next time?

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

Mac OS

What is the output of `bazel info release`?

release 7.1.1

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

No response

What's the output of `git remote get-url origin; git rev-parse HEAD` ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

artem-zinnatullin commented 6 months ago

We're seeing this multiple times per day for some reason it happens mostly on macOS CI machines and the issue persists between retries if it bounces onto the same macOS machine.

2024/05/07 13:34:08 Downloading https://releases.bazel.build/8.0.0/rolling/8.0.0-pre.20240128.3/bazel-8.0.0-pre.20240128.3-darwin-arm64...
--
  | WARNING: option '--remote_download_outputs' was expanded to from both option '--remote_download_minimal' (source expanded from --config=ci) and option '--remote_download_toplevel' (source expanded from --config=ci)
  | ERROR: Failed to parse downloader config: tools/bazel/bazel-downloader-config.txt

KostadinAlmishev commented 6 months ago

I thing code fails here, this is called from here.

During bazel build fails with error above (and running local bazel server), I tried to read the file from different program the same way this is done in bazel. Exception didn't occurred:

scala> Files.newBufferedReader(Paths.get("bazel_downloader.cfg"))
val res10: java.io.BufferedReader = java.io.BufferedReader@22bd6b0a

scala> res10.readLine()
val res11: String = rewrite (github.com/.*) https://<example masked>/bazel-build/mirror/$1

scala> res10.readLine()
val res12: String = ""

scala> res10.readLine()
val res13: String = block github.com

scala> res10.readLine()
val res14: String = null

As a result I founded another workaround - killing local bazel server process.

Our code is mounted with fuse filesystem. Sometimes this happens, when developer remounts the system, when local bazel server is running.

Current bazel version: 7.1.1 With previous version 6.2.1 we didn't have this problem.

meteorcloudy commented 6 months ago

/cc @shs96c Can you please take look?

shs96c commented 6 months ago

I can have a look. I'm working on a reproduction so I can debug it better. If anyone has an easy way to get this to occur reliably, that would be very helpful

keith commented 5 months ago

We see this on github actions CI too. It's definitely flaky for us too.

keith commented 5 months ago

In our case the server isn't even running and on the first launch it happens:

Installing bazelisk...
INFO: Invocation ID: 24c57e4a-37f6-4b0d-a279-fdfcf00fb996
ERROR: Failed to parse downloader config: bazel/downloader-cfg.txt

edit: hrm maybe this isn't the first run

shs96c commented 5 months ago

I'm still attempting to reproduce this bug locally. When I can do that reliably, I'll be able to debug what's going on.

shs96c commented 5 months ago

I've patched Bazel to report a little more information about what's going on, and I've managed to get this to fail once with this error. It would appear that the current working directory isn't the workspace root when we see this, which is... surprising. Working on that assumption, I'll prepare a patch to try and address the issue.

bazelbuild / bazel