Closed noppoMan closed 4 years ago
adam-fowler
commented
4 years ago @ephemer This issue was here when I joined the project. I have some ideas how we could get a general purpose version of this working but it requires a number of requisites. I wouldn’t expect to see this working in the near future.
What are you trying to do exactly? There might be special case versions that could be implemented.
adam-fowler
commented
4 years ago Actually come to think of it if you are using s3.getObject and s3.putObject I can’t see a solution without streaming. We can stream getObject using the multipart api we’ve setup but the put version requires a minimum chunk size of 5gb (aws limitation). We would need to implement streaming for putObject.
As I said streaming won’t happen over night but now as I have a good use case for it I can prioritise it appropriately.
ephemer
commented
4 years ago Hi @adam-fowler, thanks for the quick reply here! Our files aren't that big, but a few hundred MB won't fit in memory in a small lambda function container either - our use case is we'd like to be able to process large files without needing to load them entirely into memory
adam-fowler
commented
4 years ago @ephemer ok when I have something working I’ll post in here. I’m thinking of possibly extending https://github.com/swift-aws/aws-sdk-swift-core/issues/185 to support streamed payloads.
adam-fowler
commented
4 years ago @ephemer I know you're not using Lambda anymore but https://github.com/swift-aws/aws-sdk-swift-core/pull/237, https://github.com/swift-aws/aws-sdk-swift-core/pull/236
adam-fowler
commented
4 years ago Done!
Would this include streaming chunks of an S3 object? We'd really like to have that functionality because we're aiming to process huge wav files on Lambda.
This Issue is quite old, so I'm curious whether work has been done in this direction, or whether developments since its creation have meant that it's unlikely to ever happen?