search

dominikh / go-tools

Staticcheck - The advanced Go linter
https://staticcheck.dev
MIT License
6.18k stars 378 forks source link

staticcheck: fatal error: runtime: out of memory in go1.12 #419

Closed jeanbza closed 5 years ago

jeanbza commented 5 years ago

OS: Ubuntu 14.04.5 LTS Memory: 14.68GB Proc Logical CPUs: 4 Go version: go version go1.12 linux/amd64 Staticcheck version (in go.mod): honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a

When I run staticcheck ./... at the root of cloud.google.com/go, I get:

+ staticcheck -go 1.9 -ignore '
*:S1007
*:SA1019
cloud.google.com/go/firestore/internal/doc-snippets.go:*
cloud.google.com/go/functions/metadata/metadata_test.go:SA1012
cloud.google.com/go/cmd/go-cloud-debug-agent/internal/controller/client_test.go:*
cloud.google.com/go/cmd/go-cloud-debug-agent/internal/debug/dwarf/frame.go:*
cloud.google.com/go/cmd/go-cloud-debug-agent/internal/debug/dwarf/typeunit.go:*
cloud.google.com/go/cmd/go-cloud-debug-agent/internal/debug/dwarf/const.go:*
cloud.google.com/go/cmd/go-cloud-debug-agent/internal/debug/dwarf/line.go:*
cloud.google.com/go/cmd/go-cloud-debug-agent/internal/debug/server/server.go:*
cloud.google.com/go/cmd/go-cloud-debug-agent/internal/debug/server/dwarf.go:*
cloud.google.com/go/cmd/go-cloud-debug-agent/internal/debug/server/eval.go:*
cloud.google.com/go/cmd/go-cloud-debug-agent/internal/debug/server/value.go:*
cloud.google.com/go/cmd/go-cloud-debug-agent/internal/debug/elf/file.go:*
cloud.google.com/go/cmd/go-cloud-debug-agent/internal/breakpoints/breakpoints_test.go:*
cloud.google.com/go/cmd/go-cloud-debug-agent/internal/debug/gosym/pclntab_test.go:*
cloud.google.com/go/cmd/go-cloud-debug-agent/debuglet.go:*
cloud.google.com/go/translate/internal/translate/v2/translate-nov2016-gen.go:*
cloud.google.com/go/storage/bucket.go:S1002
cloud.google.com/go/spanner/value.go:S1025
cloud.google.com/go/pubsub/integration_test.go:S1012
cloud.google.com/go/internal/fields/fold.go:S1008
cloud.google.com/go/httpreplay/internal/proxy/debug.go:*
cloud.google.com/go/bigtable/internal/cbtconfig/cbtconfig.go:ST1005
cloud.google.com/go/bigtable/cmd/cbt/cbt.go:ST1005
cloud.google.com/go/asset/v1beta1/doc.go:*
cloud.google.com/go/spanner/value_test.go:S1019
cloud.google.com/go/bigtable/reader.go:S1002
cloud.google.com/go/internal/btree/btree.go:U1000
' ./...
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xa14a1e, 0x16)
    /usr/local/go/src/runtime/panic.go:617 +0x72
runtime.sysMap(0xc3c4000000, 0x80000000, 0xecb438)
    /usr/local/go/src/runtime/mem_linux.go:170 +0xc7
runtime.(*mheap).sysAlloc(0xeb30c0, 0x7db2a000, 0xeb30d0, 0x3ed95)
    /usr/local/go/src/runtime/malloc.go:633 +0x1cd
runtime.(*mheap).grow(0xeb30c0, 0x3ed95, 0x0)
    /usr/local/go/src/runtime/mheap.go:1232 +0x42
runtime.(*mheap).allocSpanLocked(0xeb30c0, 0x3ed95, 0xecb448, 0x7f92ece59df0)
    /usr/local/go/src/runtime/mheap.go:1150 +0x3a7
runtime.(*mheap).alloc_m(0xeb30c0, 0x3ed95, 0x7f92e5590100, 0xeb30d0)
    /usr/local/go/src/runtime/mheap.go:977 +0xc2
runtime.(*mheap).alloc.func1()
    /usr/local/go/src/runtime/mheap.go:1048 +0x4c
runtime.(*mheap).alloc(0xeb30c0, 0x3ed95, 0x7f92e6010100, 0x7f92e559f7f0)
    /usr/local/go/src/runtime/mheap.go:1047 +0x8a
runtime.largeAlloc(0x7db2a000, 0x1, 0x0)
    /usr/local/go/src/runtime/malloc.go:1055 +0x99
runtime.mallocgc.func1()
    /usr/local/go/src/runtime/malloc.go:950 +0x46
runtime.systemstack(0x0)
    /usr/local/go/src/runtime/asm_amd64.s:351 +0x66
runtime.mstart()
    /usr/local/go/src/runtime/proc.go:1153

goroutine 1 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc001041078 sp=0xc001041070 pc=0x458460
runtime.mallocgc(0x7db2a000, 0x9cd8e0, 0xffffffffffffff01, 0x0)
    /usr/local/go/src/runtime/malloc.go:949 +0x872 fp=0xc001041118 sp=0xc001041078 pc=0x40da32
runtime.growslice(0x9cd8e0, 0xc2905a0000, 0xc60588, 0xc60589, 0x11d5fd2, 0x6d82bc, 0x8230ec, 0x6d82bc)
    /usr/local/go/src/runtime/slice.go:181 +0x1e6 fp=0xc001041180 sp=0xc001041118 pc=0x443e66
honnef.co/go/tools/lint/lintutil.compileErrors(0xc0002f9ea0, 0xc0fde3a240, 0x32, 0x50898)
    /root/go/pkg/mod/honnef.co/go/tools@v0.0.0-20190106161140-3f1c8253044a/lint/lintutil/util.go:340 +0x4bb fp=0xc001041370 sp=0xc001041180 pc=0x6e063b
honnef.co/go/tools/lint/lintutil.compileErrors(0xc000303c00, 0xc061233260, 0xac384, 0xac384)
    /root/go/pkg/mod/honnef.co/go/tools@v0.0.0-20190106161140-3f1c8253044a/lint/lintutil/util.go:340 +0x440 fp=0xc001041560 sp=0xc001041370 pc=0x6e05c0
honnef.co/go/tools/lint/lintutil.compileErrors(0xc000316a80, 0xc3285a86f0, 0x17bf210, 0x17bf216)
    /root/go/pkg/mod/honnef.co/go/tools@v0.0.0-20190106161140-3f1c8253044a/lint/lintutil/util.go:340 +0x440 fp=0xc001041750 sp=0xc001041560 pc=0x6e05c0
honnef.co/go/tools/lint/lintutil.Lint(0xc00010c420, 0x4, 0x6, 0xc000020230, 0x1, 0x1, 0xc001041e40, 0x976e40, 0x942580, 0x1, ...)
    /root/go/pkg/mod/honnef.co/go/tools@v0.0.0-20190106161140-3f1c8253044a/lint/lintutil/util.go:287 +0x458 fp=0xc001041a78 sp=0xc001041750 pc=0x6df9c8
honnef.co/go/tools/lint/lintutil.ProcessFlagSet(0xc00010c420, 0x4, 0x6, 0xc00010c3c0)
    /root/go/pkg/mod/honnef.co/go/tools@v0.0.0-20190106161140-3f1c8253044a/lint/lintutil/util.go:178 +0x9aa fp=0xc001041f00 sp=0xc001041a78 pc=0x6deb6a
main.main()
    /root/go/pkg/mod/honnef.co/go/tools@v0.0.0-20190106161140-3f1c8253044a/cmd/staticcheck/staticcheck.go:29 +0x2d0 fp=0xc001041f98 sp=0xc001041f00 pc=0x8f07b0
runtime.main()
    /usr/local/go/src/runtime/proc.go:200 +0x20c fp=0xc001041fe0 sp=0xc001041f98 pc=0x42f8fc
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc001041fe8 sp=0xc001041fe0 pc=0x45a531
cleanup

When I run the exact same set of commands in go version go1.11.5 linux/amd64, it works. I have tried both several times and the results are consistent.

Link to 1.12 (failing) log: https://source.cloud.google.com/results/invocations/89398127-b9a7-49d4-befe-c9abbce087c9/targets Link to 1.11 (passing) log: https://source.cloud.google.com/results/invocations/a80c20a8-7aad-4b3c-a79a-9f7529ba7844/targets

I loosely wonder whether there might be a regression in types/ast or some other stdlib package that is greatly reducing memory efficiency. cc @matloob

Might be related to https://github.com/dominikh/go-tools/issues/377

dominikh commented 5 years ago

A couple of basic things:

jeanbza commented 5 years ago

The links to the logs do not work for me. Though I suppose they may for matloob?

Apologies. I had thought these were public (at one point they were). Will try to figure out why that's not the case and get back to you.

Can you share the peak RSS of staticcheck with Go 1.11.5? Is it close to the system's max memory?

Sorry for noob question, but how do I go about getting you this information?

jeanbza commented 5 years ago

Update: the logs have been made public.

matloob commented 5 years ago

I'm able to access the logs, are they available to you, @dominikh?

I'm not sure what could have changed between 1.11 and 1.12 to cause this.

But this brings up an "interesting" point: we make the assumption that we can have all the typechecked data in memory at once. That might be a valid assumption if the project you're opening/typechecking is small enough, or if there's enough memory, but it might not be true here:

I'm wondering if this is going to become a big problem as more of these tools are run in memory-constrained containers. (Though 14G is a lot.....)

dominikh commented 5 years ago

I'm able to access the logs, are they available to you, @dominikh?

I have access to them now, yes.

we make the assumption that we can have all the typechecked data in memory at once

To be fair, staticcheck currently makes the assumption that all ASTs, type information and SSA can be in memory at once. Memory usage should go down drastically once we migrate to the go/analysis framework.

Sorry for noob question, but how do I go about getting you this information?

I don't know how to do it in your Google Cloud thingy, but locally you'd invoke staticcheck via /usr/bin/time -v and check the Maximum resident set size stat.

CyrusNajmabadi commented 5 years ago

Note: we're hitting what seems to be the same problem ourselves. We blow through our 8GB of ram on travis when doing CI/CD: https://github.com/pulumi/pulumi/issues/2629

Example Stack ``` golangci-lint run fatal error: runtime: out of memory runtime stack: runtime.throw(0xe79fc5, 0x16) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/panic.go:617 +0x72 runtime.sysMap(0xc1ac000000, 0x4000000, 0x17fd0b8) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/mem_linux.go:170 +0xc7 runtime.(*mheap).sysAlloc(0x17dec20, 0x2000, 0x17dec30, 0x1) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/malloc.go:633 +0x1cd runtime.(*mheap).grow(0x17dec20, 0x1, 0x0) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/mheap.go:1232 +0x42 runtime.(*mheap).allocSpanLocked(0x17dec20, 0x1, 0x17fd0c8, 0x2b9178e66b68) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/mheap.go:1150 +0x3a7 runtime.(*mheap).alloc_m(0x17dec20, 0x1, 0xb, 0x2b9178e66b68) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/mheap.go:977 +0xc2 runtime.(*mheap).alloc.func1() /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/mheap.go:1048 +0x4c runtime.systemstack(0x0) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/asm_amd64.s:351 +0x66 runtime.mstart() /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/proc.go:1153 goroutine 30545 [running]: runtime.systemstack_switch() /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/asm_amd64.s:311 fp=0xc09962c980 sp=0xc09962c978 pc=0x4592c0 runtime.(*mheap).alloc(0x17dec20, 0x1, 0x2b917801000b, 0x2b9178e66a48) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/mheap.go:1047 +0x8a fp=0xc09962c9d0 sp=0xc09962c980 pc=0x42493a runtime.(*mcentral).grow(0x17df260, 0x0) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/mcentral.go:256 +0x95 fp=0xc09962ca18 sp=0xc09962c9d0 pc=0x417875 runtime.(*mcentral).cacheSpan(0x17df260, 0x3f) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/mcentral.go:106 +0x2ff fp=0xc09962ca78 sp=0xc09962ca18 pc=0x41737f runtime.(*mcache).refill(0x2b9160c836d0, 0xb) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/mcache.go:135 +0x86 fp=0xc09962ca98 sp=0xc09962ca78 pc=0x416e16 runtime.(*mcache).nextFree(0x2b9160c836d0, 0x41530b, 0x2b9178cf7f00, 0x100, 0x2b9178cf7fff) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/malloc.go:786 +0x88 fp=0xc09962cad0 sp=0xc09962ca98 pc=0x40b718 runtime.mallocgc(0x40, 0xcb5520, 0x30000000000001, 0xc1abffbfc0) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/malloc.go:939 +0x76e fp=0xc09962cb70 sp=0xc09962cad0 pc=0x40c02e runtime.makeslice(0xcb5520, 0x1c, 0x40, 0xc1abffbfc0) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/slice.go:49 +0x6c fp=0xc09962cba0 sp=0xc09962cb70 pc=0x4429ec bytes.(*Buffer).grow(0xc1abffe3f0, 0x1c, 0x1c) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/bytes/buffer.go:131 +0x26b fp=0xc09962cbf0 sp=0xc09962cba0 pc=0x4ec19b bytes.(*Buffer).WriteString(0xc1abffe3f0, 0xc001e0c000, 0x1c, 0xc07e0bc100, 0xd, 0xd) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/bytes/buffer.go:187 +0xe7 fp=0xc09962cc20 sp=0xc09962cbf0 pc=0x4ec4c7 go/types.writePackage(0xc1abffe3f0, 0xc07dde78b0, 0x0) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/go/types/object.go:443 +0x64 fp=0xc09962cc60 sp=0xc09962cc20 pc=0x87cfb4 go/types.writeFuncName(0xc1abffe3f0, 0xc07e0d2820, 0x0) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/go/types/object.go:483 +0xbd fp=0xc09962cd30 sp=0xc09962cc60 pc=0x87d52d go/types.(*Func).FullName(0xc07e0d2820, 0xc0002de420, 0xc1abffc540) /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/go/types/object.go:315 +0x57 fp=0xc09962cd78 sp=0xc09962cd30 pc=0x87c477 github.com/golangci/golangci-lint/vendor/github.com/golangci/go-tools/staticcheck.(*Checker).checkCalls(0xc0382ee620, 0xc1a9e100e0, 0xc0002de420) /home/travis/gopath/src/github.com/golangci/golangci-lint/vendor/github.com/golangci/go-tools/staticcheck/lint.go:2397 +0xac fp=0xc09962cf50 sp=0xc09962cd78 pc=0xba680c github.com/golangci/golangci-lint/vendor/github.com/golangci/go-tools/staticcheck.(*Checker).callChecker.func1(0xc1a9e100e0) /home/travis/gopath/src/github.com/golangci/golangci-lint/vendor/github.com/golangci/go-tools/staticcheck/lint.go:2383 +0x3d fp=0xc09962cf78 sp=0xc09962cf50 pc=0xbb542d github.com/golangci/golangci-lint/vendor/github.com/golangci/go-tools/lint.(*Linter).Lint.func1(0xc1a693c370, 0xc1a6992180, 0xc1a9e100e0) /home/travis/gopath/src/github.com/golangci/golangci-lint/vendor/github.com/golangci/go-tools/lint/lint.go:469 +0x102 fp=0xc09962cfc8 sp=0xc09962cf78 pc=0xb58902 runtime.goexit() /home/travis/.gimme/versions/go1.12.1.linux.amd64/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc09962cfd0 sp=0xc09962cfc8 pc=0x45b211 created by github.com/golangci/golangci-lint/vendor/github.com/golangci/go-tools/lint.(*Linter).Lint /home/travis/gopath/src/github.com/golangci/golangci-lint/vendor/github.com/golangci/go-tools/lint/lint.go:455 +0x1d6e ```
CyrusNajmabadi commented 5 years ago

@dominikh

To be fair, staticcheck currently makes the assumption that all ASTs, type information and SSA can be in memory at once. Memory usage should go down drastically once we migrate to the go/analysis framework.

Do you have an expectation on when that will happen? Thanks!

dominikh commented 5 years ago

@CyrusNajmabadi it will likely still take several months. In the meantime I strongly recommend experimenting with GOGC.

dominikh commented 5 years ago

I've pushed some commits that should reduce memory usage some. Nothing dramatic yet, but you may be able to save some GBs.

kolaente commented 5 years ago

I have the same problem. I tried updating to the latest version of staticcheck with these commands (using go mod in the project):

go get -u honnef.co/go/tools
go mod vendor
go install -v -mod=vendor honnef.co/go/tools/cmd/staticcheck

wich resulted in vendor/honnef.co/go/tools/staticcheck/lint.go:626:23: call.Job.Pkg.TypesSizes undefined (type *lint.Pkg has no field or method TypesSizes).

I'll check in my CI to see if it works there. For CI, I'm using drone on a machine with 16GB RAM.

Update: Doesn't work on drone either. The failing CI step runs in the golang:1.12-alpine docker image.

dominikh commented 5 years ago

@kolaente

You seem to be using an old version of golang.org/x/tools/go/packages – the TypesSizes field was added on January 17 in b4b6fe2cb82970f144debbe03ecb71e340b15446. Please update your vendored dependencies.

Also, running out of memory (what this issue is about) and failing to compile staticcheck (your comment) are not "the same problem". In the future please file a separate issue.

dominikh commented 5 years ago

I'm going to merge this issue into #376. Once we finish the port, memory usage will be greatly reduced.