search

timothymugayi / boto3-sqs-extended-client-lib

Boto3 SQS Extended Client Library for Python. An extension to the boto3 sqs client that enables sending and receiving messages up to 2GB via Amazon S3. [WARNING: This library is still under development contributors welcome]
Apache License 2.0
27 stars 23 forks source link

[ENHANCEMENT] get_text_from_s3 returning a closed file object #4

Closed Maximilian-Staab closed 4 years ago

Maximilian-Staab commented 5 years ago

Issue: A closed file object was returned.

Proposition:

  1. Read the file and return what was read.
  2. Don't use a file, use a in memory BytesIO object and write/read from there. No need to open file handles.

I used 2, because why use a file if you can avoid it? SQS bodies have to be string, so it has to read into memory at some point.

timothymugayi commented 4 years ago

@Qdulf, let's give users an option as well to stream data if it's too large to add into memory. I will create some open issues of nice to have features, that we can add in perhaps if time permits.