silverton-io / buz

Serverless multi-protocol + multi-destination event collection system.
https://buz.dev/
Apache License 2.0
192 stars 23 forks source link

Add GOMEMLIMIT as env variable in Terraform for serverless implementation of Buz #592

Closed ricky-galvao closed 5 months ago

ricky-galvao commented 5 months ago

When running in serverless mode, in particular with AWS Lambda, it has been noted that the Garbage Collector (GC) does not adequately keep up with freeing past execution allocations when handling high rates of requests. As a result, memory allocation grows with reuse of the same lambda container across invocations. This eventually leads to the the container being killed by the Lambda service. This PR sets the GOMEMLIMIT environment variable (both in AWS and GCP) at a default value of 90% max allocated memory. GOMEMLIMIT is a soft memory limit that Go will recognize and start to more aggressively execute GC once reached.

Resolves silverton-io/buz/issues/591