search

rust-lang / rls

Repository for the Rust Language Server (aka RLS)
Other
3.51k stars 257 forks source link

Linux memory usage regression #1188

Open alexheretic opened 5 years ago

alexheretic commented 5 years ago

I noticed RLS can consume quite a lot of memory lately. Upstream: https://github.com/rust-lang/rust/issues/56980

Idle memory usage

Memory usage running rls --cli. Reading taken after running once until built, quitting and re-running again and waiting for build complete.

Project Linux nightly-2018-12-14 Linux stable 1.31
cargo init --bin 80 MiB 25 MiB
rust-random/rand 500 MiB 45 MiB
rust-lang/regex 1.5 GiB 81 MiB

So current stable gives a decent idea of where it should be. I tested beta channel out of interest and had results similar to current nightly.

Status

Tests use regex d4b9419 on Arch Linux

2019-03-10

Release Idle memory usage
1.31.1 62 MB
1.32.0 1.6 GB
1.33.0 800 MB

Update 2019-07-04

Release Idle memory usage
1.34.2 740 MB
1.35.0 1.3 GB
1.36.0 1.0 GB

Update 2019-08-16

Release Idle memory usage
1.31.1 83 MB
1.37.0 2.4 GB
aloucks commented 5 years ago

Does stable include the removal of jemalloc?

Memory consumption on windows still seems to be reasonable:

80 MiB after building rust-lang/regex:

$ rustc --version
rustc 1.32.0-nightly (4a45578bc 2018-12-07)

$ rls --version
rls-preview 1.31.6 (bfa1371 2018-12-07)

image

alexheretic commented 5 years ago

Thanks for testing Windows, so perhaps it's not affected. I did try selecting system & jemalloc allocators but they both had similarly high usage for me on Linux.

Xanewok commented 5 years ago

I believe we build Rust with jemalloc (only libstd?), so the distribution artifacts (incl. RLS) should use jemalloc.

Since system allocator is being selected since recently, is it possible we use both allocators (system for locally-built RLS space and jemalloc for the dynamically linked rustc-land), leading to increased memory usage?

alexheretic commented 5 years ago

I bisected the memory regression based on nightly toolchain releases

Testing as above with the regex crate:

alexheretic commented 5 years ago

I can also deduce this regression is caused upstream if I can reproduce it using the same rls source + different compilers.

For example using current rls code (173be7729ae9ea41303fbd84f339f76554c9d9af),

plus a small fix for older compilers

``` diff --git src/build/rustc.rs src/build/rustc.rs index be3c52c..9ebc925 100644 --- src/build/rustc.rs +++ src/build/rustc.rs @@ -301,6 +301,7 @@ impl<'a> CompilerCalls<'a> for RlsRustcCalls { edition: match state.session.edition() { RustcEdition::Edition2015 => Edition::Edition2015, RustcEdition::Edition2018 => Edition::Edition2018, + _ => panic!(), }, }; let files = fetch_input_files(state.session); ```


Again testing as above with the regex crate:

alexheretic commented 5 years ago

With 1.32 coming out I thought I'd retest rls memory usage to provide some more recent numbers.

Project Linux stable 1.31 / nightly-2018-11-03 Linux stable 1.32
rust-lang/regex 80 MiB 1.4 GiB

So this regression has not been addressed and will affect the next stable release.

mati865 commented 5 years ago

Since switch to system as default allocator tools don't use jemalloc anymore: https://github.com/rust-lang/rust/issues/56980#issuecomment-455227996

I can reproduce huge memory usage that doesn't go down even with RLS built on the very up to date Linux distro (latest available GLIBC, kernel, toolchain, etc). This is pretty worrisome because it means there is leak somewhere or there is bug in memory handling. It was masked by jemalloc but that doesn't mean there was no issue.

At this point underlying issue should be identified and fixed. I'll try adding similar hack as in https://github.com/rust-lang/rust/pull/57287 but it shouldn't be considered as fix.

EDIT: Is there a way to reproduce large memory usage by manually running a rls command instead of using IDE? It's not easy to debug it when it's run from IDE.

jens1o commented 5 years ago

Yeah @mati865, you can run rls --cli in a workspace.

mati865 commented 5 years ago

@jens1o well I was hoping for something like rls arg1 arg2 arg3 ... and boom 600 MiB memory consumed 😄

alexheretic commented 5 years ago

I've tried forcing use of jemalloc & adding the stuff in https://github.com/rust-lang/rust/pull/57287 to rls main (though I don't understand it). These don't seem to help the memory usage regression though, or maybe I'm doing something wrong.

mati865 commented 5 years ago

I don't have good news. After switching RLS to use jemmalloc it's peak memory usage increased a lot but later it drops to numbers similar to system allocator. Clippy and Cargo were still using system allocator during testing.

Valgrind and Heapstack didn't give any meaningful results. Maybe somebody more skilled could find out what's wrong, I'm out of ideas.

programmerjake commented 5 years ago

10GB used for me after initial build completed on https://salsa.debian.org/Kazan-team/kazan rls-preview 1.33.0 (9f3c906 2019-01-20) installed via rustup Linux x86_64

Note that building kazan builds LLVM as part of the process, so it can take a really long time and lots of memory.

jgillich commented 5 years ago

VS Code + rls consume 3GB RAM, 1.4GB for rls itself. This is a huge issue on systems with 8GB RAM. I'll have to upgrade the RAM on my laptop because of this.

Xanewok commented 5 years ago

I just did Massif runs on rust-lang/regex, RLS itself and a cargo new project using latest RLS master built with rustc 1.35.0-nightly (53f2165c5 2019-04-04) and I can't reproduce it anymore.

Keep in mind that this is with all_targets set to true (so every bin/lib/test target is compiled and analyzed).

This is a simple output for run on RLS (some packages were cached but cargo needed to build some from scratch at the beginning):

    MB
441.5^                            #                                           
     |                            #                                           
     |                      @@@:@@#                  @                        
     |                    @@@@@:@@#             :@@@@@                        
     |                  :@@@@@@:@@#            @@@@@@@                        
     |                 @@@@@@@@:@@#          :@@@@@@@@                        
     |                 @@@@@@@@:@@#         @@@@@@@@@@                        
     |            @@:@@@@@@@@@@:@@#       @:@@@@@@@@@@                        
     |           :@@:@@@@@@@@@@:@@#     @@@:@@@@@@@@@@          @@@           
     |          :@@@:@@@@@@@@@@:@@#    :@@@:@@@@@@@@@@         @@@@           
     |          :@@@:@@@@@@@@@@:@@#    @@@@:@@@@@@@@@@        @@@@@  @@       
     |          @@@@:@@@@@@@@@@:@@#    @@@@:@@@@@@@@@@       @@@@@@ @@@       
     |         :@@@@:@@@@@@@@@@:@@#   @@@@@:@@@@@@@@@@ :@ @  @@@@@@ @@@       
     |       :@:@@@@:@@@@@@@@@@:@@#  @@@@@@:@@@@@@@@@@ @@ @ :@@@@@@ @@@       
     |    :  @@:@@@@:@@@@@@@@@@:@@# @@@@@@@:@@@@@@@@@@ @@:@ @@@@@@@@@@@       
     |    : :@@:@@@@:@@@@@@@@@@:@@# @@@@@@@:@@@@@@@@@@:@@:@@@@@@@@@@@@@       
     |   :: @@@:@@@@:@@@@@@@@@@:@@#@@@@@@@@:@@@@@@@@@@@@@:@@@@@@@@@@@@@       
     | @:::@@@@:@@@@:@@@@@@@@@@:@@#@@@@@@@@:@@@@@@@@@@@@@:@@@@@@@@@@@@@       
     | @:::@@@@:@@@@:@@@@@@@@@@:@@#@@@@@@@@:@@@@@@@@@@@@@:@@@@@@@@@@@@@@ :@@@@
     |@@:::@@@@:@@@@:@@@@@@@@@@:@@#@@@@@@@@:@@@@@@@@@@@@@:@@@@@@@@@@@@@@@@@@@@
   0 +----------------------------------------------------------------------->Gi
     0                                                                   75.77

While it peaks at ~440MB, after everything is rebuilt it then stays on idle at 55MB.

For regex package it's slightly different (reopened folder, previously RLS built the dependencies):

    MB
511.7^                          #                                             
     |                         @#                                             
     |        :                @#                      :                      
     |       ::                @#                     @:                      
     |       @:               @@#         @:        ::@@                  @@  
     |       @:               @@#       @@@:       @@:@@       @@       @@@@  
     |      @@:             : @@#      @@@@:      :@@:@@      @@@      @@@@@  
     |      @@:             : @@#     :@@@@:      @@@:@@     @@@@      @@@@@  
     |      @@:           @@@ @@#     @@@@@:     @@@@:@@    @@@@@     @@@@@@  
     |      @@:          @@@@ @@#    @@@@@@:     @@@@:@@    @@@@@    @@@@@@@  
     |      @@:          @@@@:@@#   :@@@@@@:  @@@@@@@:@@  @@@@@@@  @@@@@@@@@  
     |      @@:         @@@@@:@@# @@:@@@@@@: @@@@@@@@:@@@:@@@@@@@ @@@@@@@@@@  
     |  :  @@@:   @:    @@@@@:@@#:@@:@@@@@@: @@@@@@@@:@@@:@@@@@@@ @@@@@@@@@@: 
     | :@  @@@:  :@: @@@@@@@@@@@#@@@:@@@@@@:@@@@@@@@@:@@@:@@@@@@@@@@@@@@@@@@@@
     |@@@  @@@:  @@::@@@@@@@@@@@#@@@:@@@@@@:@@@@@@@@@:@@@:@@@@@@@@@@@@@@@@@@@@
     |@@@:@@@@: :@@:@@@@@@@@@@@@#@@@:@@@@@@:@@@@@@@@@:@@@:@@@@@@@@@@@@@@@@@@@@
     |@@@:@@@@::@@@:@@@@@@@@@@@@#@@@:@@@@@@:@@@@@@@@@:@@@:@@@@@@@@@@@@@@@@@@@@
     |@@@:@@@@:@@@@@@@@@@@@@@@@@#@@@:@@@@@@:@@@@@@@@@:@@@:@@@@@@@@@@@@@@@@@@@@
     |@@@:@@@@:@@@@@@@@@@@@@@@@@#@@@:@@@@@@:@@@@@@@@@:@@@:@@@@@@@@@@@@@@@@@@@@
     |@@@:@@@@@@@@@@@@@@@@@@@@@@#@@@:@@@@@@:@@@@@@@@@:@@@:@@@@@@@@@@@@@@@@@@@@
   0 +----------------------------------------------------------------------->Gi
     0                                                                   234.5

Then idle is 130 MB. What's interesting in this case is that there's

->35.79% (53,018,184B) 0x1063E80: __rbt_backtrace_alloc (in /home/xanewok/repos/rls/target/release/rls)
| ->29.37% (43,520,972B) 0x106AED2: __rbt_backtrace_get_view (in /home/xanewok/repos/rls/target/release/rls)
| | ->16.07% (23,815,387B) 0x1063294: elf_add (in /home/xanewok/repos/rls/target/release/rls)
| | | ->16.07% (23,815,387B) 0x1063C62: phdr_callback (in /home/xanewok/repos/rls/target/release/rls)
| | |   ->16.07% (23,815,387B) 0x9D55F1F: dl_iterate_phdr (dl-iteratephdr.c:75)
| | |     ->16.07% (23,815,387B) 0x1063DC1: __rbt_backtrace_initialize (in /home/xanewok/repos/rls/target/release/rls)
| | |       ->16.07% (23,815,387B) 0x10605A9: __rbt_backtrace_pcinfo (in /home/xanewok/repos/rls/target/release/rls)
| | |         ->16.07% (23,815,387B) 0x105FB38: backtrace::symbolize::resolve (libbacktrace.rs:171)
| | |           ->16.07% (23,815,387B) 0x105E9B8: backtrace::capture::Backtrace::resolve (capture.rs:146)
| | |             ->16.07% (23,815,387B) 0x105E03E: failure::backtrace::internal::InternalBacktrace::as_backtrace (internal.rs:61)
| | |               ->16.07% (23,815,387B) 0x105E0AA: <&T as core::fmt::Debug>::fmt (mod.rs:131)
| | |                 ->16.07% (23,815,387B) 0x8A7656A: core::fmt::write (mod.rs:1016)
| | |                   ->16.07% (23,815,387B) 0x8A77312: core::fmt::Formatter::write_fmt (mod.rs:1457)
| | |                     ->16.07% (23,815,387B) 0x105DD57: <failure::error::Error as core::fmt::Debug>::fmt (mod.rs:199)
| | |                       ->16.07% (23,815,387B) 0x8A7656A: core::fmt::write (mod.rs:1016)
| | |                         ->16.07% (23,815,387B) 0x8A7626F: <core::fmt::Arguments as core::fmt::Debug>::fmt (mod.rs:417)
| | |                           ->16.07% (23,815,387B) 0x8A7656A: core::fmt::write (mod.rs:1016)
| | |                             ->16.07% (23,815,387B) 0x112CD7C: env_logger::fmt::Builder::build::{{closure}} (mod.rs:1279)
| | |                               ->16.07% (23,815,387B) 0x1128017: std::thread::local::LocalKey<T>::with (lib.rs:804)
| | |                                 ->16.07% (23,815,387B) 0x1126DC0: <env_logger::Logger as log::Log>::log (lib.rs:775)
| | |                                   ->16.07% (23,815,387B) 0x11BEBA9: log::__private_api_log (lib.rs:1232)
| | |                                     ->16.07% (23,815,387B) 0x2C1D03: rls::build::cargo::run_cargo_ws (cargo.rs:271)
| | |                                       ->16.07% (23,815,387B) 0x2BD098: rls::build::cargo::run_cargo (cargo.rs:137)
| | |                                         ->16.07% (23,815,387B) 0x452D33: _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h34609
f02c4e16181E.llvm.1588280635495087468 (cargo.rs:62)
| | |                                           ->16.07% (23,815,387B) 0x33D463: _ZN3std9panicking3try7do_call17hb8fcd0f3e48228acE.llvm.6379468967674043134 (mod.rs:470)
| | |                                             ->16.07% (23,815,387B) 0x8A585D8: __rust_maybe_catch_panic (lib.rs:87)
| | |                                               ->16.07% (23,815,387B) 0x4C9E29: <F as alloc::boxed::FnBox<A>>::call_box (panicking.rs:272)
| | |                                                 ->16.07% (23,815,387B) 0x8A5733C: std::sys::unix::thread::Thread::new::thread_start (boxed.rs:762)
| | |                                                   ->16.07% (23,815,387B) 0x97C06D9: start_thread (pthread_create.c:463)
| | |                                                     ->16.07% (23,815,387B) 0x9D1188D: clone (clone.S:95)

which make up around 100 of the total ~130MB.

So while we're not in the gigabyte order of magnitude during idle (at least I can't confirm it, still need to test some on-and-off development and see if there is any prominent leakage) we still can improve things.

@nnethercote would you have any tips or insights on how we could debug it better, seeing as you profiled a lot of the Rust compiler itself and as the author/maintainer(?) of Massif?

nnethercote commented 5 years ago

First, use massif-visualizer. It's a graphical viewer for massif data that is much better than ms_print.

Second, if you build your own version of Valgrind from trunk (or wait until 3.15 comes out, which I believe should be soon) Massif will get better, because it can now use debug info to get better stack traces. The Massif that comes with Valgrind 3.14 and earlier doesn't know about inlining, so any inlined frames aren't shown, and in Rust code there is usually a lot of inlining. See here for how to build your own Valgrind.

Third, if you build your own (or wait until 3.15) you can use the new and improved version of DHAT, which tracks peak memory usage as well as a bunch of other things relating to heap memory. I find its presentation of the stacks at peak memory a bit nicer, but it doesn't have the time-based graph, and in your case the graph is useful because there are multiple peaks.

alexheretic commented 5 years ago

Testing with regex d4b9419 I still get ~900MB usage after idle, so no change for me with rustc 1.35.0-nightly (8159f389f 2019-04-06).

I tested with regex & rls checked out next to each other. Run in regex dir:

cargo run --manifest-path ../rls/Cargo.toml --no-default-features --release -- --cli
jhpratt commented 5 years ago

What can be done about this? The bug has existed since mid-Decmeber and is P-high. By not fixing this, it seems like Linux is being treated like it doesn't exist.

I just recorded RLS at 2.2 GB (on a machine with 8 GB) in VS Code with no files open. I had worked on Rust files previously, but it had been at least half an hour since I had any open.

I am absolutely willing to help, but given that no one is even assigned to this, I'm losing faith. If someone can point me in the right direction, that would be great.

What changed between 2018-11-03 and 2018-11-06, as indicated in this issue? I presume that's a starting place.

mati865 commented 5 years ago

@jhpratt Linux has the best overall support. This issue is just harder to debug.

What changed between 2018-11-03 and 2018-11-06, as indicated in this issue? I presume that's a starting place.

Rust changed the way it links jemalloc, there is more info in Rust issue: https://github.com/rust-lang/rust/issues/56980 But that change haven't introduced the issue, it only exposed it in official builds. Unofficial builds without jemalloc were affected before that change.

alexheretic commented 5 years ago

Without upstream progress I think we'll need #1307 which should address this as the compiler memory usage will die with the process.

u8983478934 commented 5 years ago

Will this issue ever be solved? RLS makes my PC totally freeze within minutes from opening VSCode

gh67uyyghj commented 4 years ago

@alexcrichton @alexheretic would you please release the solution in #1307 at least in linux releases because development with RLS on Linux is really a painful experience due to this memory leak bug. It becomes exponentially worse as the project grows.

fstephany commented 4 years ago

@gophobic @gh67uyyghj In the meantime you can give a try to rust-analyzer. It's frequently updated and works much better than RLS (on my setup at least :p)

gh67uyyghj commented 4 years ago

@fstephany thank you, I just installed rust-analyzer using the commands on their README but couldn't get it work, all I see in VSCode is the error message Couldn't start client Rust Analyzer Language Server :(

jhpratt commented 4 years ago

FWIW This is now bad enough that I've disabled RLS in VS Code due to the extremely frequent restarts necessary. Memory would rise to over a 1.5 GB within a matter of a minute or two. This is just developing the time crate, not anything terribly large.

bluenote10 commented 4 years ago

In case tougher reproductions are needed: In projects using godot-rust I'm hitting 6 GB (often sending my machine into swap-freezes).

Xanewok commented 4 years ago

Could you try using nightly RLS and see if that’s still the issue?

On Fri, 15 May 2020 at 10:52, Fabian Keller notifications@github.com wrote:

In case tougher reproductions are needed: In projects using godot-rust https://github.com/GodotNativeTools/godot-rust I'm hitting 6 GB (often sending my machine into swap-freezes).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/rust-lang/rls/issues/1188#issuecomment-629116245, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXTFXIW5UCGWCL3IYX42NDRRT7DVANCNFSM4GKNBBFQ .

buckle2000 commented 4 years ago

I'm using stable, and the memory usage is still terrible. Maybe language server is just a bad idea for every language. Non-long-running shared library plugins would work better.

Xanewok commented 4 years ago

The fix hasn’t been backported, so stable and beta are not fixed and have the memory issue.

On Sat, 16 May 2020 at 21:50, buckle2000 notifications@github.com wrote:

I'm using stable, and the memory usage is still terrible.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/rust-lang/rls/issues/1188#issuecomment-629697611, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXTFXND7JYG4SSLNHWNPTDRR3VBBANCNFSM4GKNBBFQ .

mineralres commented 3 years ago 
rls 1.41.0 (2cf84ba 2020-12-06)

rls 1.41.0 still alloc huge memory (about 4.5G, on a machine with 8 GB, ubuntu 18.04) .