x/tools/gopls: out of memory failures with terraform-cdk and azurerm provider

adeturner commented 2 years ago

What version of Go are you using (`go version`)?

go1.18.3 linux/amd64

16GB Windows laptop Running 4GB WSL2 ubuntu instance + 2GB swap: vscode

plus

node 18.4.0 Terraform v1.2.3 cdktf 0.11.2 "terraformProviders": ["hashicorp/azurerm@~> 3.0.0"],

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

go env Output

GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/myuser/.cache/go-build"
GOENV="/home/myuser/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/myuser/gopath/go/pkg/mod"
GOOS="linux"
GOPATH="/home/myuser/gopath/go"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/myuser/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/myuser/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18.3"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build871093409=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I followed: tutorials/terraform/cdktf-install

Then I customised https://github.com/hashicorp/terraform-cdk/tree/main/examples/go/azure

I ran "cdktf get" which downloads azurerm and builds a "generated" directory with the modules in it

What did you expect to see?

gopls in a vscode environment coping with the cdktf generated libraries.
Limited required memory and cpu consumption from gopls

What did you see instead?

Raising an issue to see if there is optimisation potential There are an awful lot of stubs in the azurerm provider generated code and it appears to provoke >6GB memory usage from gopls which caused OOM errors. Workaround was to reconfigure WSL2 to 8GB.

if you combine the gopls memory consumption with the vscode and cdktf memory consumption ..... it is not possible to run vscode/gopls and issue cdktf build commands at the same time.

I raised a related issue here https://github.com/hashicorp/terraform-cdk/issues/1886

findleyr commented 2 years ago

Thank you for this issue, and apologies for the slow response (for some reason this issue wasn't put into the Unreleased milestone until I updated the issue title, so it avoided our triage process...).

Generally speaking, gopls uses a lot of memory. This is something we're working on fixing, but it will be a long road.

However, in this case there may be specific problems related to the cdk that we can mitigate.

findleyr commented 1 year ago

Update after a year: that long road has been walked. gopls@v0.12.0 redesigned the core of gopls to use drastically less memory. I'd be curious if this helped the situation with this repo.

gopherbot commented 1 year ago

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

golang / go