search

pulumi / pulumi-aws

An Amazon Web Services (AWS) Pulumi resource package, providing multi-language access to AWS
Apache License 2.0
460 stars 155 forks source link

wafv2 compile issue with Go. #1117

Closed niekske closed 1 year ago

niekske commented 4 years ago

Whenever I use: _, err := wafv2.NewWebAcl(ctx, "name", &wafv2.WebAclArgs{}) Pulumi will exit with:

error: an unhandled error occurred: program exited with non-zero exit code: 1

Full log:

~/infra master *2 !6 ?1 ❯ pulumi preview --logtostderr -v=5 13s 11:49:51 Previewing update (dev): I0914 11:49:56.038843 23330 plan_executor.go:391] planExecutor.retirePendingDeletes(...): no pending deletions I0914 11:49:56.038866 23330 plan_executor.go:215] planExecutor.Execute(...): waiting for incoming events I0914 11:49:56.038901 23330 step_executor.go:321] StepExecutor worker(-2): worker coming online I0914 11:49:56.038909 23330 step_executor.go:321] StepExecutor worker(-2): worker waiting for incoming chains I0914 11:49:56.040326 23330 eventsink.go:59] AWS Auth provider used: "SharedCredentialsProvider" I0914 11:49:56.040921 23330 eventsink.go:59] Trying to get account information via sts:GetCallerIdentity I0914 11:49:56.059997 23330 eventsink.go:78] eventSink::Infoerr(<{%reset%}>go: downloading github.com/pulumi/pulumi/sdk/v2 v2.10.0 <{%reset%}>) Type Name Plan Info pulumi:pulumi:Stack dev go: downloading github.com/pulumi/pulumi/sdk/v2 v2.10.0 I0914 11:49:56.660379 23330 eventsink.go:59] Trying to get account information via sts:GetCallerIdentity I0914 11:59:11.413276 23330 eventsink.go:78] eventSink::Infoerr(<{%reset%}>go build github.com/pulumi/pulumi-aws/sdk/v3/go/aws/wafv2: /usr/local/go/pkg/tool/darwin_amd64/compile: signal: killed pulumi:pulumi:Stack dev go build github.com/pulumi/pulumi-aws/sdk/v3/go/aws/wafv2: /usr/local/go/pkg/too I0914 11:59:12.113648 23330 source_eval.go:178] EvalSourceIterator ended with an error: an unhandled error occurred: program exited with non-zero exit code: 1 I0914 11:59:12.113744 23330 plan_executor.go:219] planExecutor.Execute(...): incoming event (nil? true, &{0xc00140eac0}) I0914 11:59:12.113850 23330 eventsink.go:86] eventSink::Error(<{%reset%}>an unhandled error occurred: program exited with non-zero exit code: 1<{%reset%}>) I0914 11:59:12.113885 23330 step_executor.go:321] StepExecutor worker(-1): StepExecutor.waitForCompletion(): waiting for worker threads to exit I0914 11:59:12.113907 23330 step_executor.go:321] StepExecutor worker(-2): worker exiting due to cancellation I0914 11:59:12.113914 23330 step_executor.go:321] StepExecutor worker(-1): StepExecutor.waitForCompletion(): worker threads all exited I0914 11:59:12.113925 23330 plan_executor.go:254] planExecutor.Execute(...): step executor has completed pulumi:pulumi:Stack dev error: an unhandled error occurred: program exited with non-zero exit code: 1 I0914 11:59:12.133092 23330 ignore.go:44] Explicitly ignoring and discarding error: rpc error: code = Canceled desc = grpc: the client connection is closing I0914 11:59:12.136450 23330 host.go:451] Error closing 'go' language plugin during shutdown; ignoring: 1 error occurred:

  • operation not permitted pulumi:pulumi:Stack dev 1 error; 2 messages

Diagnostics: pulumi:pulumi:Stack (dev): go: downloading github.com/pulumi/pulumi/sdk/v2 v2.10.0 go build github.com/pulumi/pulumi-aws/sdk/v3/go/aws/wafv2: /usr/local/go/pkg/tool/darwin_amd64/compile: signal: killed

error: an unhandled error occurred: program exited with non-zero exit code: 1

The process "compile" is also running for a long time, until it's killed by something (timeout or OS), and using up all application memory of my computer:

user 23449 497.8 52.4 18575468 8783752 s003 R 11:50AM 2:13.77 /usr/local/go/pkg/tool/darwin_amd64/compile -o /var/folders/5n/njr5t8bj0ksgvwlw29ctjx7r0000gn/T/go-build075238138/b354/pkg.a -trimpath /var/folders/5n/njr5t8bj0ksgvwlw29ctjx7r0000gn/T/go-build075238138/b354=> -p github.com/pulumi/pulumi-aws/sdk/v3/go/aws/wafv2 -lang=go1.14 -complete -buildid RTkyzfG9SR4-kSnHnJKx/RTkyzfG9SR4-kSnHnJKx -goversion go1.15.2 -D -importcfg /var/folders/5n/njr5t8bj0ksgvwlw29ctjx7r0000gn/T/go-build075238138/b354/importcfg -pack -c=4 /Users/user/go/pkg/mod/github.com/pulumi/pulumi-aws/sdk/v3@v3.2.1/go/aws/wafv2/getIpSet.go /Users/user/go/pkg/mod/github.com/pulumi/pulumi-aws/sdk/v3@v3.2.1/go/aws/wafv2/getRegexPatternSet.go /Users/user/go/pkg/mod/github.com/pulumi/pulumi-aws/sdk/v3@v3.2.1/go/aws/wafv2/getRuleGroup.go /Users/user/go/pkg/mod/github.com/pulumi/pulumi-aws/sdk/v3@v3.2.1/go/aws/wafv2/getWebAcl.go /Users/user/go/pkg/mod/github.com/pulumi/pulumi-aws/sdk/v3@v3.2.1/go/aws/wafv2/ipSet.go /Users/user/go/pkg/mod/github.com/pulumi/pulumi-aws/sdk/v3@v3.2.1/go/aws/wafv2/pulumiTypes.go /Users/user/go/pkg/mod/github.com/pulumi/pulumi-aws/sdk/v3@v3.2.1/go/aws/wafv2/regexPatternSet.go /Users/user/go/pkg/mod/github.com/pulumi/pulumi-aws/sdk/v3@v3.2.1/go/aws/wafv2/ruleGroup.go /Users/user/go/pkg/mod/github.com/pulumi/pulumi-aws/sdk/v3@v3.2.1/go/aws/wafv2/webAcl.go /Users/user/go/pkg/mod/github.com/pulumi/pulumi-aws/sdk/v3@v3.2.1/go/aws/wafv2/webAclAssociation.go /Users/user/go/pkg/mod/github.com/pulumi/pulumi-aws/sdk/v3@v3.2.1/go/aws/wafv2/webAclLoggingConfiguration.go

Environment:

go version go1.15.2 darwin/amd64 pulumi v.2.10.0 pulumi-aws v3.2.1

I also noticed that the wafv2 webacl webpage is extremely slow: https://www.pulumi.com/docs/reference/pkg/aws/wafv2/webacl/. However this might be unrelated.

leezen commented 4 years ago 
I0914 11:59:11.413276 23330 eventsink.go:78] eventSink::Infoerr(<{%reset%}>go build github.com/pulumi/pulumi-aws/sdk/v3/go/aws/wafv2: /usr/local/go/pkg/tool/darwin_amd64/compile: signal: killed

It looks like the go compilation process might be OOM'ing -- any chance you could post your machine's specs?

niekske commented 4 years ago

Machine specs:

MacBook Pro (16-inch, 2019) Processor: 2,6 GHz 6-Core Intel Core i7 Memory: 16 GB 2667 MHz DDR4

I think that should be enough. Without the wafv2 line it takes about 2-3s for pulumi, but with it, it takes 30 minutes or so and indeed memory keeps increasing until it probably get killed due to OOM.

niekske commented 4 years ago

Problem still exist after updating to:

iselegant commented 3 years ago

I hitted same issue. It may be caused by deep nested configuration of wafv2.NewWebAcl. Just I found same issue about simular IaC tools, Terraform. https://github.com/hashicorp/terraform/issues/25889

yo-ga commented 3 years ago

I hit the same issue when executing the command go get github.com/pulumi/pulumi-aws/sdk/v4/go/aws/wafv2 in the docker container. But executing the same command on my desktop is successful.

Avishek-Gulshan commented 2 years ago

I am also facing the same issue, go get github.com/pulumi/pulumi-aws/sdk/v4/go/aws/wafv2 is failing when building on the container but working successfully on mac with 16 GB RAM. It seems it requires around 16 GB of memory to build. Why does it need this much memory?? can we do something to fix this?

manca-disc commented 2 years ago

@lukehoban Hi Luke -- this issue impacts us as well. Can we please prioritize it, or at least suggest a workaround? We have multiple teams that depend on proper WAF configuration and we want to use this package for it, but the build takes more than >15GB of RAM which results in OOMs in our Github-hosted runners (they have 7GB RAM limit).

Thanks!

tarun-discovery commented 2 years ago

@lukehoban what is the status on this? Taking up 20GB just for Wafv2 is clearly not good. Please help. We are not getting any updates on this

adomoshe commented 2 years ago

Any update on this?

lukehoban commented 2 years ago

As noted above, due to the AWS Terraform Provider Schema being enormous here, the AWS Classic provider SDK for this type includes a very large number of types (and associated functions). The wafv2 module contains over 20000 types in the Go SDK. The performance issues here are technically not actually about running pulumi, they are about running go build on the extremely large Go library in use here. The issue at https://github.com/pulumi/pulumi/issues/8950 is tracking improvements we might be able to make to the Go SDK codegen to limit the size and improve the performance. But there are probably limits to what we can do there as long as we use the current Terraform Provider schema to support this type.

There are a few potential options and paths forward.

For users:

For Pulumi:

adomoshe commented 2 years ago

Hi @lukehoban, thank you for the thorough response. Those are definitely good options that I did not know were available. I will post my solution and any issues I run into so others can have some insight into this flow.

adomoshe commented 2 years ago

Switching to github.com/pulumi/pulumi-aws-native/sdk/go/aws/wafv2 worked like a charm thank you! Syntax is slightly different, mostly because the struct names are just more concise now, and I was able to only change out the struct names without needing any deeper work on the AWS Classic wafv2 code. (I use the web ACL with regex pattern sets, associated to an apigateway stage)

Only tips are that

  1. ctx.RegisterStackTransformation(...) in order to autoTag resources does not work with the AWS native resources because they have their own structs for each resource instead of just a generic pulumi.StringMapInput as the AWS Classic has (I did not see another way to auto tag resources on AWS Native but I'm sure that will come as the sdk matures)
  2. Pulumi.{stackname}.yaml files need an extra line of config aws-native:region: with the same region as your aws:region:
  3. An extra permission is needed cloudformation:...

Incredibly no other config was needed and this code fit right into the already existing AWS Classic code.

Pulumi did a great job of throwing accurate errors for these very minor issues I encountered. My build ran quickly and successfully on the lowest tier CodeBuild environment.

Thanks for the great work here Pulumi 🎉

mikhailshilkov commented 1 year ago

This will be fixed by https://github.com/pulumi/pulumi-aws/pull/2589 in 6.0 of the provider