search

bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
22.99k stars 4.03k forks source link

Enable --shutdown_on_low_sys_mem by default to avoid system-wide memory pressure #7600

Open Globegitter opened 5 years ago

Globegitter commented 5 years ago

Description of the problem / feature request:

Now that I started on some open source projects around bazel, that might be some rules or contributions to bazel itself, as well as us using it in our monorepo I am noticing I sometimes have two or even three bazel server's running at the same time and that is really straining my machine and sometimes getting it to freeze for a very long time. The problem seems to be mostly due to memory usage of the server. Even when it is sitting idle after >10min of no usage I can still see it using up over 2GB of memory plus more virtual memory.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Not sure if this is a bug or feature request, but either way, use a machine with 16GB of memory, have things open for a normal developer setup, browser with a few tabs, vs code/IDE, slack, a terminal with a few tabs and start running tests in two or three different repositories using bazel. It does not have to be in parallel.

What operating system are you running Bazel on?

Elementary OS 5 (Ubuntu 18.04)

What's the output of bazel info release?

release 0.23.0rc3

Any other information, logs, or outputs that you want to share?

To be honest, I am not sure if there is anything that can be done directly, or if the best answer is just one has to rune bazel shutdown but I think it is also important to be aware of this issue, maybe there is also something that can be improved in bazel.

Only thing related I could find: https://github.com/bazelbuild/bazel/issues/7446

jin commented 5 years ago

Consider using --max_idle_secs to shut down idle servers. It defaults to 3 hours.

jmmv commented 5 years ago

7446 has been very interesting, but note that that's exclusively for macOS -- and your report comes from Linux.

This is a big problem indeed (servers continuing to use a lot of RAM in the background) and I don't think we have a good answer so far. One possibility we have discussed was to page the Skyframe graph to disk and thus allow limiting Bazel's high memory usage watermark, but that's non-trivial and it's unclear how such a thing would affect performance. Another approach we've talked about is to drop nodes from Skyframe and recompute them on demand. I believe a mixture of the two could solve this issue, but it's unclear yet as we don't have a design and don't know how that'd affect performance.

susinmotion commented 5 years ago

It'll take some time to figure out whether it's even possible to improve the status quo, so in the meantime we'll focus on documenting this behavior, as you suggested.

Globegitter commented 5 years ago

Thanks, as an update from my side, I have also recently found the flag --shutdown_on_low_sys_mem: https://docs.bazel.build/versions/master/command-line-reference.html#flag--shutdown_on_low_sys_mem but not tried it yet.

jmmv commented 4 years ago

Yeah, unfortunately I think we'll have to live with a combination of --max_idle_secs and --shutdown_on_low_sys_mem. The latter flag doesn't yet work on Mac, but we are planning on adding support for it relatively soon. I also think we should be enabling this flag by default (which we already do internally at Google, and the reason there is a flag seems to have been purely to allow for a controlled rollout).

laurentlb commented 4 years ago

@jmmv Shall we update the documentation? https://docs.bazel.build/versions/master/command-line-reference.html#flag--shutdown_on_low_sys_mem says "Linux only"