brexhq / substation

Substation is a toolkit for routing, normalizing, and enriching security event and audit logs.
https://substation.readme.io
MIT License
322 stars 16 forks source link

fix(transform): Remove Temporary Files from AWS S3 #205

Closed jshlbrd closed 2 months ago

jshlbrd commented 2 months ago

Description

Motivation and Context

This was discovered while upgrading a large AWS S3 pipeline, over time files build up in the /tmp directory of the Lambda container and cause the application to crash. This only occurs if the size of all files is more than 500 MB, but the error is obvious:

{
    "errorMessage": "transform 119330de-398867f3: write /tmp/substation4083518264: no space left on device",
    "errorType": "errorString"
}

How Has This Been Tested?

Tested E2E in the production data pipeline mentioned above.

Types of changes

Checklist: