search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
123.68k stars 17.62k forks source link

x/tools/gopls: high memory consumption #54268

Closed codisms closed 1 year ago

codisms commented 2 years ago

gopls version

Build info
----------
golang.org/x/tools/gopls v0.9.1
    golang.org/x/tools/gopls@v0.9.1 h1:SigsTL4Hpv3a6b/a7oPCLRv5QUeSM6PZNdta1oKY4B0=
    github.com/google/go-cmp@v0.5.7 h1:81/ik6ipDQS2aGcBfIN5dHDB36BwrStyeAQquSYCV4o=
    github.com/sergi/go-diff@v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
    golang.org/x/mod@v0.6.0-dev.0.20220419223038-86c51ed26bb4 h1:6zppjxzCulZykYSLyVDYbneBfbaBIQPYMevg0bEwv2s=
    golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
    golang.org/x/sys@v0.0.0-20220209214540-3681064d5158 h1:rm+CHSpPEEW2IsXUib1ThaHIjuBVZjxNgSKmBLFfD4c=
    golang.org/x/text@v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
    golang.org/x/tools@v0.1.12-0.20220713141851-7464a5a40219 h1:Ljlba2fVWOA1049JjsKii44g8nZN2GjpxMlzVc8AnQM=
    mvdan.cc/gofumpt@v0.3.0 h1:kTojdZo9AcEYbQYhGuLf/zszYthRdhDNDUi2JKTxas4=
    mvdan.cc/xurls/v2@v2.4.0 h1:tzxjVAj+wSBmDcF6zBB7/myTy3gX9xvi8Tyr28AuQgc=
go: go1.16.15

go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ubuntu/.cache/go-build"
GOENV="/home/ubuntu/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ubuntu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ubuntu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.15"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3575468840=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Editing Go files with vim and vim-go.

What did you expect to see?

Lower memory consumption.

What did you see instead?

Over 5GB of memory consumption; thrashing swap file.

Editor and settings

VIM - Vi IMproved 9.0 (2022 Jun 28, compiled Aug  1 2022 05:17:14)
Included patches: 1-124
Compiled by ubuntu@dev-veritone
Huge version without GUI.  Features included (+) or not (-):
+acl               +file_in_path      +mouse_urxvt       -tag_any_white
+arabic            +find_in_path      +mouse_xterm       -tcl
+autocmd           +float             +multi_byte        +termguicolors
+autochdir         +folding           +multi_lang        +terminal
-autoservername    -footer            -mzscheme          +terminfo
-balloon_eval      +fork()            +netbeans_intg     +termresponse
+balloon_eval_term +gettext           +num64             +textobjects
-browse            -hangul_input      +packages          +textprop
++builtin_terms    +iconv             +path_extra        +timers
+byte_offset       +insert_expand     +perl/dyn          +title
+channel           +ipv6              +persistent_undo   -toolbar
+cindent           +job               +popupwin          +user_commands
-clientserver      +jumplist          +postscript        +vartabs
-clipboard         +keymap            +printer           +vertsplit
+cmdline_compl     +lambda            +profile           +vim9script
+cmdline_hist      +langmap           -python            +viminfo
+cmdline_info      +libcall           +python3/dyn       +virtualedit
+comments          +linebreak         +quickfix          +visual
+conceal           +lispindent        +reltime           +visualextra
+cryptv            +listcmds          +rightleft         +vreplace
+cscope            +localmap          +ruby/dyn          +wildignore
+cursorbind        -lua               +scrollbind        +wildmenu
+cursorshape       +menu              +signs             +windows
+dialog_con        +mksession         +smartindent       +writebackup
+diff              +modify_fname      -sodium            -X11
+digraphs          +mouse             -sound             -xfontset
-dnd               -mouseshape        +spell             -xim
-ebcdic            +mouse_dec         +startuptime       -xpm
+emacs_tags        -mouse_gpm         +statusline        -xsmp
+eval              -mouse_jsbterm     -sun_workshop      -xterm_clipboard
+ex_extra          +mouse_netterm     +syntax            -xterm_save
+extra_search      +mouse_sgr         +tag_binary
-farsi             -mouse_sysmouse    -tag_old_static
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -g -O2 -D_REENTRANT -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: gcc -L. -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,-E -L/usr/local/lib -Wl,--as-needed -o vim -lm -ltinfo -lrt -ldl -Wl,-E -fstack-protector-strong -L/usr/local/lib -L/usr/lib/x86_64-linux-gnu/perl/5.30/CORE -lperl -ldl -lm -lpthread -lcrypt

Logs

gopls.3782930-5GiB-nonames.zip

findleyr commented 2 years ago

That gopls uses a lot of memory is a known (and difficult) problem. See for example #47855.

However, 5gb is a lot, and we should check whether that makes sense based on your usage. Can you tell me:

Unfortunately, at the moment 5gb is not unheard-of memory usage when working on large projects, such as Kubernetes. We're working on this, but it is a long-term project.

codisms commented 2 years ago

Unfortunately, I don't have a great way to answer the first question. The number is hard to know because of the libraries being referenced, but the project itself has 3500 Go files, though not all referenced in all parts of the code. One thing I can say is that we have a single file generated by gqlgen that has 256k lines in it and is probably a big culprit.

Initially loading the project causes the machine to spin for a good five minutes or so and it'll spike to 3-4 GB pretty easily. It'll settle around 2-3 after a while but spikes as the files are edited. Multiple files could be changed at the same time with multiple vim windows going on. I do have the gopls server running for quite a long time though.

findleyr commented 2 years ago

Thanks.

I expect you'll see it stay in the the ~3gb range after gopls@v0.9.2 is released. I'll take a look at the heap dump you've provided to see if I can understand this better.

findleyr commented 2 years ago

Putting this in the next milestone for additional triage.

galileo-pkm commented 2 years ago

Not much has changed with this release. gopls remains a memory hog. My "template" project without any real code and gopls eats up 2.2G. With a few medium size project in VSCode workspace it regularly goes over 8GB.

gopls v0.9.4, go.mod for the project and the diagnostics file attached if anyone cares. gopls.35536-2GiB-withnames.zip go.mod.gz

switchupcb commented 1 year ago

Do you experience this memory usage immediately, or does it take a while to build up? — @findleyr

I don't ever hit 1 GB of RAM (cause I'm not analyzing 256 KLOC). However, I noticed when running VSCode for a long time that gopls will freeze up and stop analyzing code. I kill the process (which shows higher memory consumption than usual) using Task Manager, which restarts gopls and it works for some time. Then, the cycle repeats.

Maybe a short term solution for the large projects is to have a mode that only analyzes specific files (i.e opened) or some way to exclude them (directory, whatever makes sense, etc).

Update: I hit 2GB on a 1500 line file.

findleyr commented 1 year ago

gopls@v0.12.0-pre.3 significantly reduces gopls' memory usage, up to an order of magnitude on larger repositories.

Closing this as resolved. Please open a new issue if you are still encountering memory issues with the v0.12.0 release (the final release is scheduled for this week).