Closed errm closed 1 month ago
Hi @errm ,
Thanks for reaching out.
Can you please give us more info about how you tested it? For example, What is your code you used to observe this behavior, what does this graph mean? What did you use to profile the increase in memory? Do you have the same memory profile info available to compare against the previous version that you upgraded from?
Thanks, Ran~
Hi @RanVaknin
The graph shows the memory usage of my application, before deploying the version with the updated dependency memory usage under load was stable ~ 575MiB after updating the aws-ask-go-v2 deps memory usage rapidly rises to ~ 900MiB where processes start to be killed for OOM. After rolling back the dependency memory usage returns to ~ 575MiB
The only thing my application is doing with s3 is a simple get of files from a bucket that are then read into a buffer.
I am working on a simplified version of our code to try and reproduce the issue and then and bisect the exact version that introduced the regression…
This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.
@RanVaknin I too have experienced this problem after migrating an application from the v1 sdk to v2:
Above is my app's memory usage on a graph. On the 18th is when I released the version that used v1.30.3 of the v2 SDK. I have a recurring job that reads images from s3 and resizes them, reuploading the resized images to the same bucket. I also serve the images via HTTP.
The 21st is when I swapped back to the v1 SDK, the memory no longer creeps. I'm looking into doing a pprof dump to see where the memory is hiding. The reason for the sudden drops in memory usage is from OOM restarts.
From a glance the steady increase in memory usage looks similar to a goroutine leak caused by an http.Client
instance having a zero value set for Timeout
. I've tried to look around the source code but there's a lot of stuff going on looking into how that HTTP client gets initialised but that might be a good place for someone who knows this repo well to check.
Acknowledgements
go get -u github.com/aws/aws-sdk-go-v2/...
)Describe the bug
I have a simple application that fetches image files from s3 and resizes them.
Recently I tried upgrading to the latest version aws-sdk-go-v2 and noticed significantly increased (~ double) memory usage in a production environment....
Expected Behavior
Memory usage should be roughly comparable with the previous versions...
Current Behavior
📝 Version with latest aws-sdk-go-v2 versions was deployed between 10:15-11:45
Reproduction Steps
I haven't produced an isolated example yet, but will update issue once I have one...
Possible Solution
No response
Additional Information/Context
No response
AWS Go SDK V2 Module Versions Used
Diff to go.mod / go.sum before/after upgrading....
Compiler and Version used
go1.22.5
Operating System and version
Linux (gcr.io/distroless/cc-debian12:nonroot)