Closed niekske closed 1 year ago
I0914 11:59:11.413276 23330 eventsink.go:78] eventSink::Infoerr(<{%reset%}>go build github.com/pulumi/pulumi-aws/sdk/v3/go/aws/wafv2: /usr/local/go/pkg/tool/darwin_amd64/compile: signal: killed
It looks like the go compilation process might be OOM'ing -- any chance you could post your machine's specs?
Machine specs:
MacBook Pro (16-inch, 2019) Processor: 2,6 GHz 6-Core Intel Core i7 Memory: 16 GB 2667 MHz DDR4
I think that should be enough. Without the wafv2 line it takes about 2-3s for pulumi, but with it, it takes 30 minutes or so and indeed memory keeps increasing until it probably get killed due to OOM.
Problem still exist after updating to:
I hitted same issue. It may be caused by deep nested configuration of wafv2.NewWebAcl. Just I found same issue about simular IaC tools, Terraform. https://github.com/hashicorp/terraform/issues/25889
I hit the same issue when executing the command go get github.com/pulumi/pulumi-aws/sdk/v4/go/aws/wafv2
in the docker container. But executing the same command on my desktop is successful.
I am also facing the same issue, go get github.com/pulumi/pulumi-aws/sdk/v4/go/aws/wafv2
is failing when building on the container but working successfully on mac with 16 GB RAM.
It seems it requires around 16 GB of memory to build.
Why does it need this much memory?? can we do something to fix this?
@lukehoban Hi Luke -- this issue impacts us as well. Can we please prioritize it, or at least suggest a workaround? We have multiple teams that depend on proper WAF configuration and we want to use this package for it, but the build takes more than >15GB of RAM which results in OOMs in our Github-hosted runners (they have 7GB RAM limit).
Thanks!
@lukehoban what is the status on this? Taking up 20GB just for Wafv2 is clearly not good. Please help. We are not getting any updates on this
Any update on this?
As noted above, due to the AWS Terraform Provider Schema being enormous here, the AWS Classic provider SDK for this type includes a very large number of types (and associated functions). The wafv2
module contains over 20000 types in the Go SDK. The performance issues here are technically not actually about running pulumi
, they are about running go build
on the extremely large Go library in use here. The issue at https://github.com/pulumi/pulumi/issues/8950 is tracking improvements we might be able to make to the Go SDK codegen to limit the size and improve the performance. But there are probably limits to what we can do there as long as we use the current Terraform Provider schema to support this type.
There are a few potential options and paths forward.
For users:
pulumi import
. This provider is in preview, but is a good option to consider here. https://www.pulumi.com/registry/packages/aws-native/api-docs/wafv2/For Pulumi:
Hi @lukehoban, thank you for the thorough response. Those are definitely good options that I did not know were available. I will post my solution and any issues I run into so others can have some insight into this flow.
Switching to github.com/pulumi/pulumi-aws-native/sdk/go/aws/wafv2
worked like a charm thank you! Syntax is slightly different, mostly because the struct names are just more concise now, and I was able to only change out the struct names without needing any deeper work on the AWS Classic wafv2 code. (I use the web ACL with regex pattern sets, associated to an apigateway stage)
Only tips are that
ctx.RegisterStackTransformation(...)
in order to autoTag resources does not work with the AWS native resources because they have their own structs for each resource instead of just a generic pulumi.StringMapInput
as the AWS Classic has (I did not see another way to auto tag resources on AWS Native but I'm sure that will come as the sdk matures)Pulumi.{stackname}.yaml
files need an extra line of config aws-native:region:
with the same region as your aws:region:
cloudformation:...
Incredibly no other config was needed and this code fit right into the already existing AWS Classic code.
Pulumi did a great job of throwing accurate errors for these very minor issues I encountered. My build ran quickly and successfully on the lowest tier CodeBuild environment.
Thanks for the great work here Pulumi 🎉
This will be fixed by https://github.com/pulumi/pulumi-aws/pull/2589 in 6.0 of the provider
Whenever I use:
_, err := wafv2.NewWebAcl(ctx, "name", &wafv2.WebAclArgs{})
Pulumi will exit with:Full log:
The process "compile" is also running for a long time, until it's killed by something (timeout or OS), and using up all application memory of my computer:
Environment:
I also noticed that the wafv2 webacl webpage is extremely slow: https://www.pulumi.com/docs/reference/pkg/aws/wafv2/webacl/. However this might be unrelated.