Reduce provider binary on-disk size

t0yv0 commented 3 months ago

Consider looking at options to reduce provider on-disk size.

Per a customer comment: it grow from 5.16 (~400MB) to 6.49 (~800MB) unpacked on disk.

Benefits of a leaner on-disk provider:

faster download times
less pressure on CI environment to have sufficient disk space

Possible culprits here:

embedded schema.json including more resources and examples for said resources, in more languages such as Java; could it be feasible to strip descriptions or at least examples from the schema distributed in the binary?
more embedded provider metadata, is there any compression that can be applied?
more Go dependencies statically linked in, anything that is possible to prune?

t0yv0 commented 3 months ago

Some information. Most of the size is present in the upstream provider build:

du -sh terraform-provider-aws                                                                                                                                                                     ~/code/terraform-provider-aws
752M    terraform-provider-aws

t0yv0 commented 3 months ago

From https://github.com/t0yv0/gobuildsize report on the upstream provider, major contributing packages are:

github.com/aws/aws-sdk-go-v2/service/ec2 137603662
github.com/hashicorp/terraform-provider-aws/internal/service/lexv2models 90502114
github.com/hashicorp/terraform-provider-aws/internal/service/batch 65336638
github.com/hashicorp/terraform-provider-aws/internal/service/ec2 60966940
github.com/hashicorp/terraform-provider-aws/internal/service/bedrockagent 59476792
github.com/aws/aws-sdk-go/service/sagemaker 56084368
github.com/aws/aws-sdk-go/service/quicksight 49658874
github.com/aws/aws-sdk-go-v2/service/iot 49391552
github.com/aws/aws-sdk-go-v2/service/glue 48909028
github.com/hashicorp/terraform-provider-aws/internal/service/securitylake 44860032
github.com/hashicorp/terraform-provider-aws/internal/service/cognitoidp 44200606
github.com/hashicorp/terraform-provider-aws/internal/service/securityhub 43308668
github.com/aws/aws-sdk-go-v2/service/rds 42986552
github.com/hashicorp/terraform-provider-aws/internal/service/verifiedpermissions 42005986
github.com/hashicorp/terraform-provider-aws/internal/service/appfabric 40250200
github.com/hashicorp/terraform-provider-aws/internal/service/rekognition 39744438
github.com/hashicorp/terraform-provider-aws/internal/service/ssmcontacts 39644536
github.com/hashicorp/terraform-provider-aws/internal/service/bedrock 38030686
github.com/hashicorp/terraform-provider-aws/internal/service/networkfirewall 37901482
github.com/hashicorp/terraform-provider-aws/internal/service/lakeformation 37305190
github.com/hashicorp/terraform-provider-aws/internal/service/medialive 37158322
github.com/hashicorp/terraform-provider-aws/internal/service/devopsguru 36337252
github.com/aws/aws-sdk-go-v2/service/chime 36253706
github.com/hashicorp/terraform-provider-aws/internal/service/cloudfront 35393416
github.com/hashicorp/terraform-provider-aws/internal/service/rds 34928184
github.com/aws/aws-sdk-go/service/connect 34686812
github.com/hashicorp/terraform-provider-aws/internal/service/elasticache 34474934
github.com/hashicorp/terraform-provider-aws/internal/service/s3 33639018
github.com/hashicorp/terraform-provider-aws/internal/service/bcmdataexports 33586502
github.com/aws/aws-sdk-go-v2/service/ssm 33467788
github.com/hashicorp/terraform-provider-aws/internal/service/timestreamwrite 33138974
github.com/aws/aws-sdk-go-v2/service/redshift 32826864
github.com/hashicorp/terraform-provider-aws/internal/service/appstream 32248636
github.com/hashicorp/terraform-provider-aws/internal/service/guardduty 32197636
github.com/hashicorp/terraform-provider-aws/internal/service/m2 31757684
github.com/aws/aws-sdk-go-v2/service/securityhub 31605124
github.com/hashicorp/terraform-provider-aws/internal/service/resourceexplorer2 31479248
github.com/hashicorp/terraform-provider-aws/internal/service/osis 31141590
github.com/hashicorp/terraform-provider-aws/internal/service/redshift 31092528
github.com/hashicorp/terraform-provider-aws/internal/service/s3control 30801928
github.com/hashicorp/terraform-provider-aws/internal/service/amp 30617456
github.com/aws/aws-sdk-go-v2/service/iam 30602038

ringods commented 3 months ago

@t0yv0 additional feedback from the customer about the impact of the plugin binary file size growth:

It is mainly two things:

download times

space on disk, hat to delete 50GB of plugins from disk yesterday

t0yv0 commented 3 months ago

This makes sense Ringo, thanks for that detail. With 50GB of plugins, I am wondering if something can be done at the plugin cache level, some form of scheduled eviction, as it appears multiple copies of provider(s) are involved there.

tobiashenkel commented 3 months ago

This makes sense Ringo, thanks for that detail. With 50GB of plugins, I am wondering if something can be done at the plugin cache level, some form of scheduled eviction, as it appears multiple copies of provider(s) are involved there.

A bit of context there. This grew over the course of roughly half a year while maintaining 50+ pulumi stacks each with its own repo with its own dependencies which are regularly updated in a semi automated way.

t0yv0 commented 3 months ago

I think there's a feature request in pulumi/pulumi that could be helpful in a situation like this: https://github.com/pulumi/pulumi/issues/7505

I will cross-link and add some ideas there.

We would love to reduce the binary size of pulumi-aws but since it appears to be dominated by terraform-provider-aws binary size it appears to be a very difficult undertaking that is unlikely to get prioritized in the short term. Hence we need to be considering broadly what else can we do to alleviate the end-user problem here.

t0yv0 commented 1 month ago

Exploring a few more options here:

-40 MB compress the schema and decompress on-demand, to be paid at startup time
-200 MB strip debug information with "-s -w" reducing the usefulness of panic traces
-600 MB applying UPX compressor; I was not able to get a working binary for MacOS builds though, may only apply to Linux builds and we will need to investigate if the result is functioning correctly and starts up fast enough