go-langserver uses lot of cpu and this causes vscode to hang or slow responsiveness

chandradeepak commented 6 years ago

Macbook OS x EI Captain

Attached is the cpu profile . It shows lot of time spent in reading files not sure why that happens but i have seen this slows the find references in vscode very slow. I have almost 50 files in a project.

let me know if you need any thing else

chandradeepak commented 6 years ago

cpu.svg.zip

keegancsmith commented 6 years ago

Thanks for the report @chandradeepak as well as the profile. How does the resource usage compare to the reference finder without the language server? Our reference implementation is a modified version of the reference finder in go guru (what the plugin uses).

When did you install the language server? We did some major updates a few months ago which should help a lot with resource usage, so just in case you have an old lang server binary hanging around I'd suggest updating.

keegancsmith commented 6 years ago

The trace seems to point to most of the time spent reading files. Right now the implementation relies on the go typechecker. It shouldn't do more work than go does to compile your application. IS the time take a similar amount of time without a pkg cache?

chandradeepak commented 6 years ago

what is a pkg cache? Not sure what it is. I know my language server is latest it was installed on Nov 6th 2017.

keegancsmith commented 6 years ago

Sorry, I missed your reply. @chandradeepak if you still have this issue, how does the perf compare to just vscode-go without the langserver? You say you have 50 files, how many files are on your GOPATH though?

chandradeepak commented 6 years ago

@keegancsmith , vscode-go would be smooth until i change some files with out langserver . But it has some other issues as well. In my GOPATH i have so many repos. Do you think that is the cause ?

slimsag commented 6 years ago

@chandradeepak To clarify what @keegancsmith said above a bit: find references with the language server is more slow if your code imports a lot of Go packages, or if the packages you import themselves import a lot of Go packages (i.e. transitive dependencies). The more Go files your program ultimately uses in the end, the slower Find References is in general.

If you disable the language server in vscode-go settings and just use vscode-go without language server, how slow is find references in comparison?

Note: We do have someone who is taking a look at improving find references performance, but it's a very tough problem to solve and as such there is no ETA on when it may be completed right now.

danicat commented 6 years ago

Is anyone exploring this at this moment?

slimsag commented 6 years ago

In order to fix this issue, we will need to switch over to using the new golang.org/x/tools/go/packages package. Almost all of the CPU and memory usage from go-langserver comes from type-checking, and using that package would reduce it substantially as it would cache results to disk automatically AFAIK.

Nobody is working on this issue right now AFAIK, and it'd probably be quite a large undertaking as it would affect almost the entire codebase I think -- but we'd be very open to someone trying to contribute this. 😃I can't give any guarantees or hard dates, but I think we are likely to work on this in a few months or so if nobody beats us to it.

danicat commented 6 years ago

Looking at the documentation on go/packages it says it's not production ready and has dependencies on Go 1.11. The idea is to wait until it gets stable?

slimsag commented 6 years ago

The warning about it not being ready for widespread use yet seems to be mostly about the fact that the API may change, which is OK for us (we can vendor it anyway).

If we use it before it supports older Go versions, we'd need to use it optionally at build time. e.g. when building with Go 1.11, it would use that package and when using an older version it would use our current go/loader setup.

kelbyers commented 5 years ago

I am experiencing high CPU usage from this recently. I have vscode up and running in the background, on a different desktop, and go-langserver shows up as using 50% of a CPU on my Mac in the activity monitor.

primalmotion commented 5 years ago

Mine goes as high as 400% every 10m. It's not usable.

akzk commented 5 years ago

The go-langserver uses 800% cpu (8 cores) and 15GB mem (16GB total) on my MBP. It happens in this month, I do not know why it happens

keegancsmith commented 5 years ago

Can you share your config? Have you enabled any optional features like code completion or diagnostics?

akzk commented 5 years ago

I write Go file by vscode, here is my settings.json

{
    "workbench.editor.enablePreview": false,
    "files.autoSave": "off",
    "editor.fontSize": 13,
    "go.testFlags": ["-v", "-count=1"],
    "go.testEnvVars": {"MY_DEVBOX_HOST": "10.8.122.122"},
    "go.docsTool": "gogetdoc",
    "go.goroot": "/usr/local/Cellar/go/1.10.3/libexec",
    "go.gopath": "/Users/leonardo/Documents/go",
    "go.gocodeAutoBuild": false,
    "go.formatTool": "goreturns",
    "go.useLanguageServer": true,
    "C_Cpp.intelliSenseEngine": "Default",
    "python.pythonPath": "python3",
    "python.formatting.provider": "yapf",
    "code-runner.clearPreviousOutput": true,
    "code-runner.saveFileBeforeRun": true,
    "code-runner.executorMap": {
        "python": "env LC_ALL=en_US.UTF-8 python3",
        "cpp": "cd $dir && g++ -std=c++11 $fileName -o $fileNameWithoutExt -w && $dir$fileNameWithoutExt && rm $dir$fileNameWithoutExt"
    },
    "window.zoomLevel": 0,
    "extensions.ignoreRecommendations": false,
    "explorer.confirmDelete": false,
    "git.autofetch": true,
    "git.confirmSync": false,
    "terminal.integrated.shell.osx": "/bin/zsh",
    "gitlens.advanced.messages": {
        "suppressShowKeyBindingsNotice": true
    }
}

BTW, there are 1281 Go files in my project, too much files ？

keegancsmith commented 5 years ago

You are not opting into anything that should make go-langserver that slow. 1281 files should be fine. We run go-langserver on sourcegraph.com and don't use as much resources for large projects such as kubernetes. We won't have time to investigate these perf issues in the short-term, so I would suggest disabling the go language server in vscode-go in the mean time. There should hopefully be a new go language server in the not too distant future which won't suffer from such issues. See https://github.com/sourcegraph/go-langserver/issues/316#issuecomment-429307592

sp4rd4 commented 5 years ago

I have the same issue, it goes almost up to 600% cpu usage, I have almost 2000 files in a project. Here is config:

{
    "[markdown]": {
        "editor.acceptSuggestionOnEnter": "off",
        "editor.formatOnSave": true,
        "editor.renderWhitespace": "all",
        "editor.wordWrap": "on"
    },
    "bracketPairColorizer.colorMode": "Consecutive",
    "bracketPairColorizer.showBracketsInGutter": true,
    "bracketPairColorizer.showHorizontalScopeLine": false,
    "bracketPairColorizer.showVerticalScopeLine": false,
    "editor.fontFamily": "Source Code Pro",
    "editor.fontSize": 13,
    "editor.formatOnPaste": true,
    "editor.insertSpaces": false,
    "editor.lineHeight": 18,
    "editor.minimap.enabled": false,
    "editor.multiCursorModifier": "ctrlCmd",
    "editor.rulers": [
        120,
        80
    ],
    "editor.scrollBeyondLastLine": false,
    "editor.snippetSuggestions": "top",
    "editor.tabSize": 2,
    "explorer.confirmDelete": false,
    "explorer.confirmDragAndDrop": false,
    "explorer.openEditors.visible": 0,
    "extensions.ignoreRecommendations": false,
    "files.exclude": {
        "**/.git": true,
        "**/.vscode": true,
        "**/bower_components": true,
        "**/node_modules": true,
        "**/vendor": true
    },
    "files.trimTrailingWhitespace": true,
    "gitlens.advanced.messages": {
        "suppressCommitHasNoPreviousCommitWarning": false,
        "suppressCommitNotFoundWarning": false,
        "suppressFileNotUnderSourceControlWarning": false,
        "suppressGitVersionWarning": false,
        "suppressLineUncommittedWarning": false,
        "suppressNoRepositoryWarning": false,
        "suppressResultsExplorerNotice": true,
        "suppressShowKeyBindingsNotice": true,
        "suppressUpdateNotice": true,
        "suppressWelcomeNotice": false
    },
    "gitlens.codeLens.enabled": false,
    "gitlens.historyExplorer.enabled": true,
    "gitlens.keymap": "alternate",
    "gitlens.statusBar.enabled": false,
    "go.autocompleteUnimportedPackages": true,
    "go.lintFlags": [
        "--config=~/.gometalinter"
    ],
    "go.lintTool": "gometalinter",
    "go.useLanguageServer": true,
    "search.exclude": {
        "**/.git": true,
        "**/.vscode": true,
        "**/bower_components": true,
        "**/node_modules": true,
        "**/vendor": true
    },
    "search.location": "panel",
    "search.useIgnoreFiles": false,
    "sync.autoDownload": false,
    "sync.autoUpload": false,
    "sync.forceDownload": false,
    "sync.quietSync": false,
    "sync.removeExtensions": true,
    "sync.syncExtensions": true,
    "telemetry.enableCrashReporter": false,
    "telemetry.enableTelemetry": false,
    "window.zoomLevel": 0.4,
    "workbench.activityBar.visible": false,
    "workbench.colorTheme": "Material Theme Darker High Contrast",
    "workbench.editor.enablePreview": false,
    "workbench.iconTheme": "eq-material-theme-icons-darker",
    "workbench.startupEditor": "newUntitledFile"
}

keegancsmith commented 5 years ago

@Sp4rd4 for now just disable the language server. If your performance improves report back here.

sandipb commented 5 years ago

I ran dtruss on go-langserver to see why it is using so much CPU. I saw that it was churning over and over the Go source files in GOROOT. I moved the Go src files to a different place, and the CPU usage became ok. Of course this means that I can't get doc info for standard library stuff, but I can atleast go through my own code. I cannot disable language server, it is the only alternative which currently supports symlinks in GOPATH, and that it critical for me.

I wish the langserver would read GOROOT/src only once during startup and not watch those directories afterwards. The number of open/fstat64 going on was ridiculous.

keegancsmith commented 5 years ago

@sandipb thanks, that is a good lead on what is going wrong and surprising to me. If you re-install the langserver using the same version of Go you are developing with do you still have issues?

sandipb commented 5 years ago

I have configured vscode to compile go-langserver using the executable for the project in a version specific tools directory. So it is compiled with the same version.

Just to confirm, I used dlv as instructed by https://dave.cheney.net/2017/06/20/how-to-find-out-which-go-version-built-your-binary

Confirmed that it is the same version.

(dlv) p runtime.buildVersion
"go1.11"

primalmotion commented 5 years ago

It seems I managed to calm go-langserver down by dropping the vendor directory and building it with whatever I had in my GOPATH. It's been an hour now and it seems to behave.

I'll update if it goes crazy again

primalmotion commented 5 years ago

It's not better. It also caused a kernel panic

*** Panic Report ***
panic(cpu 6 caller 0xffffff80114b9dc3): "compressed PTE 0xffffff92279aba98 0xc000000000000000 has extra bits 0x8000000000000000: corrupted?"@/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-4903.221.2/osfmk/i386/pmap_x86_common.c:1192
Backtrace (CPU 6), Frame : Return Address
0xffffff92279ab7d0 : 0xffffff80113aca9d 
0xffffff92279ab820 : 0xffffff80114e6893 
0xffffff92279ab860 : 0xffffff80114d82ba 
0xffffff92279ab8d0 : 0xffffff8011359ca0 
0xffffff92279ab8f0 : 0xffffff80113ac4b7 
0xffffff92279aba10 : 0xffffff80113ac303 
0xffffff92279aba80 : 0xffffff80114b9dc3 
0xffffff92279abb60 : 0xffffff80114bade7 
0xffffff92279abbc0 : 0xffffff801145a734 
0xffffff92279abea0 : 0xffffff8011452097 
0xffffff92279abf20 : 0xffffff80118c1a7a 
0xffffff92279abf40 : 0xffffff80119b5aab 
0xffffff92279abfa0 : 0xffffff801135a466 

BSD process name corresponding to current thread: go-langserver

Mac OS version:
18B75

Kernel version:
Darwin Kernel Version 18.2.0: Fri Oct  5 19:41:49 PDT 2018; root:xnu-4903.221.2~2/RELEASE_X86_64
Kernel UUID: 5D53F7E4-472A-369D-97D8-4DD877A4BDFF
Kernel slide:     0x0000000011000000
Kernel text base: 0xffffff8011200000
__HIB  text base: 0xffffff8011100000
System model name: MacBookPro13,3 (Mac-A5C67F76ED83108C)

sandipb commented 5 years ago

Yeah, I can confirm. I have had one panic reboot when I let go-langserver run at top cpu load for too long.

primalmotion commented 5 years ago

I noticed that the golang servers that start to go crazy actually panic

It seems to panic, then it goes berserk on the CPU

panic serving textDocument/signatureHelp: illegal file offset
goroutine 48235 [running]:
github.com/sourcegraph/go-langserver/langserver/util.Panicf(0x1622320, 0x17f3a20, 0x17101ae, 0x2, 0xc1b2edf518, 0x1, 0x1, 0xc1b2edf528, 0x105b696)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/util/util.go:133 +0x93
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).Handle.func1(0xc0a5f1f540, 0xc1b2edfe60)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:164 +0xc4
panic(0x1622320, 0x17f3a20)
    /usr/local/Cellar/go/1.11.2/libexec/src/runtime/panic.go:513 +0x1b9
go/token.(*File).Pos(...)
    /usr/local/Cellar/go/1.11.2/libexec/src/go/token/position.go:252
github.com/sourcegraph/go-langserver/langserver.posForFileOffset(0xc0f82360c0, 0xc0725ed957, 0x82, 0x721, 0xc00026e080)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/loader.go:125 +0xd6
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).typecheck(0xc000274000, 0x17ffe80, 0xc01b4894d0, 0x17fdf80, 0xc00026e080, 0xc0725ed950, 0x89, 0x40, 0x5a, 0x0, ...)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/loader.go:81 +0x774
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).handleTextDocumentSignatureHelp(0xc000274000, 0x17ffdc0, 0xc09201ea80, 0x17fdf80, 0xc00026e080, 0xc0a5f1f540, 0xc0725ed950, 0x89, 0x40, 0x5a, ...)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/signature.go:23 +0xcc
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).Handle(0xc000274000, 0x17ffdc0, 0xc09201ea80, 0x17fdf80, 0xc00026e080, 0xc0a5f1f540, 0x0, 0x0, 0x0, 0x0)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:417 +0x2c23
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).handle(0xc000274000, 0x17ffe00, 0xc0000a6008, 0xc00026e080, 0xc0a5f1f540, 0xc000161350, 0xc022dbaf18, 0x13ceda9, 0xc00014d480)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:154 +0x66
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).handle-fm(0x17ffe00, 0xc0000a6008, 0xc00026e080, 0xc0a5f1f540, 0x8, 0xc06008b440, 0x13c59ed, 0xc022dbaf30)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:30 +0x52
github.com/sourcegraph/go-langserver/vendor/github.com/sourcegraph/jsonrpc2.(*HandlerWithErrorConfigurer).Handle(0xc000070100, 0x17ffe00, 0xc0000a6008, 0xc00026e080, 0xc0a5f1f540)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/vendor/github.com/sourcegraph/jsonrpc2/handler_with_error.go:21 +0x73
created by github.com/sourcegraph/go-langserver/langserver.lspHandler.Handle
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:54 +0xfb
[Error - 12:37:38 PM] Request textDocument/signatureHelp failed.
  Message: unexpected panic: illegal file offset
  Code: 0 
panic serving textDocument/signatureHelp: illegal file offset
goroutine 48312 [running]:
github.com/sourcegraph/go-langserver/langserver/util.Panicf(0x1622320, 0x17f3a20, 0x17101ae, 0x2, 0xc0da60b518, 0x1, 0x1, 0xc0da60b528, 0x105b696)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/util/util.go:133 +0x93
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).Handle.func1(0xc1020496d0, 0xc0da60be60)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:164 +0xc4
panic(0x1622320, 0x17f3a20)
    /usr/local/Cellar/go/1.11.2/libexec/src/runtime/panic.go:513 +0x1b9
go/token.(*File).Pos(...)
    /usr/local/Cellar/go/1.11.2/libexec/src/go/token/position.go:252
github.com/sourcegraph/go-langserver/langserver.posForFileOffset(0xc101eb4b80, 0xc06f0cc6c7, 0x82, 0x722, 0xc00026e080)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/loader.go:125 +0xd6
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).typecheck(0xc000274000, 0x17ffe80, 0xc102135500, 0x17fdf80, 0xc00026e080, 0xc06f0cc6c0, 0x89, 0x40, 0x5b, 0x0, ...)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/loader.go:81 +0x774
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).handleTextDocumentSignatureHelp(0xc000274000, 0x17ffdc0, 0xc101eb49c0, 0x17fdf80, 0xc00026e080, 0xc1020496d0, 0xc06f0cc6c0, 0x89, 0x40, 0x5b, ...)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/signature.go:23 +0xcc
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).Handle(0xc000274000, 0x17ffdc0, 0xc101eb49c0, 0x17fdf80, 0xc00026e080, 0xc1020496d0, 0x0, 0x0, 0x0, 0x0)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:417 +0x2c23
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).handle(0xc000274000, 0x17ffe00, 0xc0000a6008, 0xc00026e080, 0xc1020496d0, 0xc000161350, 0xc001950718, 0x13ceda9, 0xc00014d480)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:154 +0x66
github.com/sourcegraph/go-langserver/langserver.(*LangHandler).handle-fm(0x17ffe00, 0xc0000a6008, 0xc00026e080, 0xc1020496d0, 0x8, 0xc00518ef40, 0x13c59ed, 0xc001950730)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:30 +0x52
github.com/sourcegraph/go-langserver/vendor/github.com/sourcegraph/jsonrpc2.(*HandlerWithErrorConfigurer).Handle(0xc000070100, 0x17ffe00, 0xc0000a6008, 0xc00026e080, 0xc1020496d0)
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/vendor/github.com/sourcegraph/jsonrpc2/handler_with_error.go:21 +0x73
created by github.com/sourcegraph/go-langserver/langserver.lspHandler.Handle
    /Users/tonio/Documents/Aporeto/workspace/code/go/src/github.com/sourcegraph/go-langserver/langserver/handler.go:54 +0xfb
[Error - 12:37:38 PM] Request textDocument/signatureHelp failed.
  Message: unexpected panic: illegal file offset
  Code: 0

primalmotion commented 5 years ago

this diff should prevent the panic at least

diff --git a/langserver/loader.go b/langserver/loader.go
index f030f90..a0f50c5 100644
--- a/langserver/loader.go
+++ b/langserver/loader.go
@@ -122,6 +122,11 @@ func posForFileOffset(fset *token.FileSet, filename string, offset int) token.Po
        if f == nil {
                return token.NoPos
        }
+
+       if offset > f.Size() {
+               return token.NoPos
+       }
+
        return f.Pos(offset)
 }

primalmotion commented 5 years ago

While it certainly fixes the panic, go-langserver is still eating all cpu and memory...

arowden commented 5 years ago

I'm also having this issue on Mac High Sierra 10.13.6. Seems like a memory leak. My CPU usage is low, but memory usage climbs until the os crashes.

primalmotion commented 5 years ago

The only way to make it usable for me with vscode is to kill it every 60s in a background bash loop

sandipb commented 5 years ago

One odd thing stood out from looking at the trace log - currently, when I am typing in the code editor, it seems that vscode sends a notif: textDocument/didChange message, and then a request #n: textDocument/documentSymbol request on every character being entered in the editor. This causes an insane amount of unnecessary chatter for a line of text being input.

Is this by the protocol design?

Could this be causing some concurrency issues? e.g. before the previous requests is handled properly, the next one comes along?

Can this be, in some ways debounced on the server side?

sandipb commented 5 years ago

I saw the goroutine count top 4000 as well as memory usage go past 5GB. Attaching allocs, heap dump and goroutine dump

pprof.zip

sandipb commented 5 years ago

Adding a CPU profile for the craziness today morning.

cpu.pprof.gz

kwiesmueller commented 5 years ago

Having the same issue. From what I saw so far, it especially happens when I type in code, quickly revert it and then either wait or continue typing again. Especially when starting to call a new function. So when my cursor is between the () of call() and i started to type something and immediately backspace because it was wrong.

marlonfan commented 5 years ago

me too

gitgroman commented 5 years ago

Same here, go-langserver consumes about 8-10Gb of memory in few minutes.

jonbonazza commented 5 years ago

FWIW, I've noticed that this is an issue on OSX, but not on ubuntu-based (debian-based?) linux distros, such as Elementary OS. Not sure what the difference is there. I'll need more time to compare dependency versions between the two test machines.