Feature request: Move Tar and Zip middlewares to Fargate

awslabs / project-lakechain

:zap: Cloud-native, AI-powered, document processing pipelines on AWS.

https://awslabs.github.io/project-lakechain/

Apache License 2.0

115 stars 22 forks source link

Feature request: Move Tar and Zip middlewares to Fargate #4

Open HQarroum opened 8 months ago

HQarroum commented 8 months ago

Use case

The Tar and the Zip inflate middleware are based on Lambda with a maximum execution time of 15 minutes. It would be better and more scalable for customers to move the implementation from Lambda to Fargate, ensuring that very large archives can be inflated.

Solution/User Experience

Moving to Fargate in order to benefit from a longer time to inflate archives.

Alternative solutions

No response