A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Today Transfer Manager's download_chunks_concurrently and upload_chunks_concurrently require you to provide a filename. Unfortunately I have bytes that are already in-memory and may not fit on disk (or it may be slow to persist it to disk) and can't use Transfer Manager as a result.
A clear and concise description of what you want to happen.
Provide functions that support the same interface as download_chunks_concurrently and upload_chunks_concurrently but take a file-like object instead of a filename
Or, for write, provide a similar function that returns a stream to the caller, and then writes to the stream are uploaded via multipart upload as needed (not sure if there's any realistic similar thing here for reads)
It's ok if this functionality can only be threaded and not use multi-processing since in practice we still see a large speed increase from threading
MattIrv
commented
3 months ago
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
download_chunks_concurrentlyandupload_chunks_concurrentlyrequire you to provide a filename. Unfortunately I have bytes that are already in-memory and may not fit on disk (or it may be slow to persist it to disk) and can't use Transfer Manager as a result.A clear and concise description of what you want to happen.
download_chunks_concurrentlyandupload_chunks_concurrentlybut take a file-like object instead of a filename