search

aws / smithy-go

Smithy code generators for Go (in development)
Apache License 2.0
156 stars 49 forks source link

Explore ways to reduce size of generated output #513

Open nikpivkin opened 1 month ago

nikpivkin commented 1 month ago

Describe the feature

The files serialize.go, deserialize.go and validators.go contain a lot of generated and uniform code. Using generics will reduce the package size.

The total size of the services is 464 mb:

du -sh service
464M    service

The size of these middleware is 182 mb, which is 40% of the total package size.

find . -type f \( -name "serializers.go" -o -name "deserializers.go" -o -name "validators.go" \) -exec du -ch {} + | grep total$
182M    total

The output of the go-size-analizer for my application:

gsa ./app
+---------+-----------------------------------------------------------------------------+--------+-----------+
| PERCENT | NAME                                                                        | SIZE   | TYPE      |
+---------+-----------------------------------------------------------------------------+--------+-----------+
| 20.63%  | github.com/aws/aws-sdk-go-v2                                                | 57 MB  | vendor    |
| 13.81%  | __gopclntab __DATA_CONST                                                    | 38 MB  | section   |
| 7.59%   | __rodata __TEXT                                                             | 21 MB  | section   |
| 7.22%   | __rodata __DATA_CONST                                                       | 20 MB  | section   |
| 5.60%   | __zdebug_loc __DWARF                                                        | 15 MB  | section   |
| 5.59%   | __zdebug_info __DWARF                                                       | 15 MB  | section   |
| 2.98%   | k8s.io/api                                                                  | 8.2 MB | vendor    |
| 2.23%   | __zdebug_line __DWARF                                                       | 6.1 MB | section   |
| 1.87%   | __zapple_names __DWARF                                                      | 5.1 MB | section   |
| 1.46%   | __zdebug_ranges __DWARF                                                     | 4.0 MB | section   |
| 1.22%   | github.com/alecthomas/chroma                                                | 3.4 MB | vendor    |
| 1.02%   | github.com/aws/aws-sdk-go                                                   | 2.8 MB | vendor    |
| 1.00%   | github.com/open-policy-agent/opa                                            | 2.8 MB | vendor    |
| 0.93%   | __data __DATA                                                               | 2.6 MB | section   |
| 0.85%   | __noptrdata __DATA                                                          | 2.3 MB | section   |
| 0.74%   | __zapple_types __DWARF                                                      | 2.0 MB | section   |
| 0.68%   | __zdebug_str __DWARF                                                        | 1.9 MB | section   |
| 0.64%   |                                                                             | 1.8 MB | generated |
| 0.52%   | __zdebug_frame __DWARF                                                      | 1.4 MB | section   |
| 0.37%   | net                                                                         | 1.0 MB | std       |
| 0.36%   | google.golang.org/protobuf                                                  | 996 kB | vendor    |
| 0.35%   | github.com/google/gnostic-models                                            | 954 kB | vendor    |

Use Case

Reducing the size of Golang binary

Proposed Solution

No response

Other Information

No response

Acknowledgements

AWS Go SDK V2 Module Versions Used

-

Go version used

-

lucix-aws commented 1 month ago

I'm sure we could make various changes to deflate the codegen output, but two questions here--

  1. Is the size of the SDK v2 binary problematic in your use case?
  2. All I see represented in the size analyzer is "aws-sdk-go-v2", rather than specific services. How many/which services are you using in your application?
nikpivkin commented 1 month ago

Hi @lucix-aws

  1. If an application is distributed as a tool and widely used in containers, I think it would be nice if its size could be reduced with a little effort.

  2. The list of services used is quite long:

    go list -m -f '{{if not .Indirect}}{{.Path}}{{end}}' all | grep github.com/aws/aws-sdk-go-v2
github.com/aws/aws-sdk-go-v2
github.com/aws/aws-sdk-go-v2/config
github.com/aws/aws-sdk-go-v2/credentials
github.com/aws/aws-sdk-go-v2/service/accessanalyzer
github.com/aws/aws-sdk-go-v2/service/apigateway
github.com/aws/aws-sdk-go-v2/service/apigatewayv2
github.com/aws/aws-sdk-go-v2/service/athena
github.com/aws/aws-sdk-go-v2/service/cloudfront
github.com/aws/aws-sdk-go-v2/service/cloudtrail
github.com/aws/aws-sdk-go-v2/service/cloudwatch
github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs
github.com/aws/aws-sdk-go-v2/service/codebuild
github.com/aws/aws-sdk-go-v2/service/docdb
github.com/aws/aws-sdk-go-v2/service/dynamodb
github.com/aws/aws-sdk-go-v2/service/ec2
github.com/aws/aws-sdk-go-v2/service/ecr
github.com/aws/aws-sdk-go-v2/service/ecs
github.com/aws/aws-sdk-go-v2/service/efs
github.com/aws/aws-sdk-go-v2/service/eks
github.com/aws/aws-sdk-go-v2/service/elasticache
github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2
github.com/aws/aws-sdk-go-v2/service/elasticsearchservice
github.com/aws/aws-sdk-go-v2/service/emr
github.com/aws/aws-sdk-go-v2/service/iam
github.com/aws/aws-sdk-go-v2/service/kafka
github.com/aws/aws-sdk-go-v2/service/kinesis
github.com/aws/aws-sdk-go-v2/service/kms
github.com/aws/aws-sdk-go-v2/service/lambda
github.com/aws/aws-sdk-go-v2/service/mq
github.com/aws/aws-sdk-go-v2/service/neptune
github.com/aws/aws-sdk-go-v2/service/rds
github.com/aws/aws-sdk-go-v2/service/redshift
github.com/aws/aws-sdk-go-v2/service/s3
github.com/aws/aws-sdk-go-v2/service/secretsmanager
github.com/aws/aws-sdk-go-v2/service/sns
github.com/aws/aws-sdk-go-v2/service/sqs
github.com/aws/aws-sdk-go-v2/service/sts
github.com/aws/aws-sdk-go-v2/service/workspaces
lucix-aws commented 1 month ago

Sure, and I agree, but that doesn't answer the question of whether the binary size is actually a concrete issue.

Regardless you're certainly right that we can make size reductions in the codegen output throughout. We already achieved a significant overall reduction once with the recent authentication refactor. Since this is more of a smithy-related issue I'm going to move this to that repo and leave open.