EC2 service is too big to compile on smaller machines

tenorwill commented 9 months ago

Describe the bug

I'm experiencing a problem building a simple getec2 program that describes ec2 instances. Using the go build -v main.go results in this:

github.com/aws/aws-sdk-go-v2/service/ec2: /usr/lib/golang/pkg/tool/linux_amd64/compile: signal: killed

Sometimes the signal is not killed and it simply hangs. If I don't interrupt with ctrl-c the program consumes all memory on the host and it dies. It only happens with the ec2 service.

Build

go build -v .
github.com/aws/aws-sdk-go-v2/service/ec2

After waiting at the terminal in a second window running top/htop, i slowly see the memory and CPU creep to 100% utilization causing the server to crash.

Build Server OS

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
Amazon Linux release 2 (Karoo)

Go Version (installed using yum)

go version go1.20.7 linux/amd64

Here's the code I'm using:

File: main.go

package main

import (
    "context"
    "fmt"
    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/ec2"
)

func main() {

    cfg, err := config.LoadDefaultConfig(context.TODO())
    if err != nil {
        fmt.Println("Error loading AWS configuration:", err)
        return
    }

    client := ec2.NewFromConfig(cfg)

    input := &ec2.DescribeInstancesInput{}

    result, err := client.DescribeInstances(context.TODO(), input)
    if err != nil {
        fmt.Println("Error describing EC2 instances:", err)
        return
    }

    for _, reservation := range result.Reservations {
        for _, instance := range reservation.Instances {
            fmt.Printf("Instance ID: %s\n", *instance.InstanceId)
            fmt.Printf("Instance Type: %s\n", *instance.InstanceType)
            fmt.Printf("Public IP: %s\n", *instance.PublicIpAddress)
            fmt.Printf("Private IP: %s\n", *instance.PrivateIpAddress)
            fmt.Printf("State: %s\n", instance.State.Name)
            fmt.Println("--------")
        }
    }
}

Expected Behavior

The program builds as expected. It seems to only fail when building on an AWS EC2 instance - local building works on MacOS or Windows. I also tried from a different distribution (Ubuntu 22.04 for example). Similarly, fails on the EC2 instance running that OS. Interestingly, it does NOT fail when running it on a local VMware ESXi VM. It only seems to occur on an AWS EC2 instance and only recently.

Current Behavior

The program hangs indefinitely, causes CPU and memory to spike, crashes the server

Using the go build -v main.go results in this:

github.com/aws/aws-sdk-go-v2/service/ec2: /usr/lib/golang/pkg/tool/linux_amd64/compile: signal: killed

Reproduction Steps

Start an EC2 instance
install Go using yum: 'yum install go -y'
use the provided code in a single main.go file (or similar to describe ec2 instances)
build the program using go build -v main.go
observe the error github.com/aws/aws-sdk-go-v2/service/ec2: /usr/lib/golang/pkg/tool/linux_amd64/compile: signal: killed

Possible Solution

unknown

Additional Information/Context

Until recently, this program worked perfectly. It's odd that it only appears to be service / ec2 and not other AWS SDK services.

AWS Go SDK V2 Module Versions Used

module getec2

go 1.20

require (
    github.com/aws/aws-sdk-go-v2 v1.21.0
    github.com/aws/aws-sdk-go-v2/config v1.18.42
    github.com/aws/aws-sdk-go-v2/service/ec2 v1.122.0
)

require (
    github.com/aws/aws-sdk-go-v2/credentials v1.13.40 // indirect
    github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.11 // indirect
    github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.41 // indirect
    github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.35 // indirect
    github.com/aws/aws-sdk-go-v2/internal/ini v1.3.43 // indirect
    github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.35 // indirect
    github.com/aws/aws-sdk-go-v2/service/sso v1.14.1 // indirect
    github.com/aws/aws-sdk-go-v2/service/ssooidc v1.17.1 // indirect
    github.com/aws/aws-sdk-go-v2/service/sts v1.22.0 // indirect
    github.com/aws/smithy-go v1.14.2 // indirect
    github.com/jmespath/go-jmespath v0.4.0 // indirect
)

Compiler and Version used

go version go1.20.7 linux/amd64

Operating System and version

Amazon Linux Version 2

RanVaknin commented 9 months ago

Hi @tenorwill ,

Thanks for opening this issue.

I tested it with a t2.micro and ran into the same issue. After changing the instance type to something beefier (m5.large) this was built quickly.

Until recently, this program worked perfectly. It's odd that it only appears to be service / ec2 and not other AWS SDK services.

Your observation is valid. One possible reason I can think of, is that ec2 released more features hence the source code grew, thats why you were having a hard time building this with smaller instance type.

Thanks for bringing this to our attention, I will review this with the team. Ran~

tenorwill commented 9 months ago

Hi @RanVaknin,

Awesome, thank you kindly for verifying/validating. My experience was similar on local workstations - a MBP with loads of RAM/CPU or a local VM with more than 2GB RAM resulted in the build succeeding. But anything 2GB or less would eventually fail or result in a system crash. Hopefully it's a quick fix for you and the team.

RanVaknin commented 8 months ago

Hi @tenorwill,

We acknowledge the difficulty you’re facing with the compilation of the EC2 service in the Go SDK on smaller EC2 instances. We’re actively discussing how to address the growing complexity and size of the EC2 service model across multiple SDKs. There have been similar experiences with other language SDKs, and we’re investigating several strategies to mitigate this.

This is a complex issue that would require an involved coordination between the EC2 and SDK team, and will take time. For now, we recommend using an instance with higher resources for the compilation as a temporary workaround. We apologize for any inconvenience this may have caused and thank you for your patience as we work towards a long-term solution.

Thanks again, Ran~

tenorwill commented 8 months ago

Thanks @RanVaknin! I will change my workflow to compile my program on a more powerful machine first, then transfer the binary file to my server

aws / aws-sdk