silverton-io / buz

Serverless multi-protocol + multi-destination event collection system.
https://buz.dev/
Apache License 2.0
186 stars 21 forks source link

Make Use of GOMEMLIMIT When in Serverless Mode #591

Closed ricky-galvao closed 4 months ago

ricky-galvao commented 4 months ago

When running in serverless mode, in particular with AWS Lambda, it has been noted that the Garbage Collector (GC) does not adequately keep up with freeing past execution allocations when handling high rates of requests. As a result, memory allocation grows with reuse of the same lambda container across invocations. This eventually leads to the the container being killed by the Lambda service.

GOMEMLIMIT can be set as an environment variable. Go will recognize the value as a 'soft' memory limit and start to more aggressively execute GC once reached.