Fixes an instance where temporary files are not removed in the AWS S3 send transform
Motivation and Context
This was discovered while upgrading a large AWS S3 pipeline, over time files build up in the /tmp directory of the Lambda container and cause the application to crash. This only occurs if the size of all files is more than 500 MB, but the error is obvious:
{
"errorMessage": "transform 119330de-398867f3: write /tmp/substation4083518264: no space left on device",
"errorType": "errorString"
}
How Has This Been Tested?
Tested E2E in the production data pipeline mentioned above.
Types of changes
[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
Description
Motivation and Context
This was discovered while upgrading a large AWS S3 pipeline, over time files build up in the
/tmp
directory of the Lambda container and cause the application to crash. This only occurs if the size of all files is more than 500 MB, but the error is obvious:How Has This Been Tested?
Tested E2E in the production data pipeline mentioned above.
Types of changes
Checklist: