search

etianen / django-s3-storage

Django Amazon S3 file storage.
BSD 3-Clause "New" or "Revised" License
414 stars 94 forks source link

Add AWS_S3_USE_THREADS #136

Closed Uxio0 closed 2 years ago

Uxio0 commented 2 years ago

When using gevent, s3 operations were breaking if using threading. Allow to disable it adding AWS_S3_USE_THREADS configuration flag

etianen commented 2 years ago

This is great! I'll get this merged ASAP.

Uxio0 commented 2 years ago

Nice! Thank you for all the work!

craigds commented 2 years ago

Any chance of some more info on the original issue? What does "s3 operations were breaking if using threading" mean exactly?

I came across this due to the django-storages changelog, but we've been using gunicorn with --worker-class=gevent and uploading via boto3 for years, and AFAIK haven't had any issues. So I'd appreciate any more info you can add.

Uxio0 commented 2 years ago

Any chance of some more info on the original issue? What does "s3 operations were breaking if using threading" mean exactly?

I came across this due to the django-storages changelog, but we've been using gunicorn with --worker-class=gevent and uploading via boto3 for years, and AFAIK haven't had any issues. So I'd appreciate any more info you can add.

That's not our experience at all (using Gunicorn + gevent too, last versions at that time). Gunicorn was breaking silently and a 500 was returned when we tried to upload a image to Django admin. We tracked the issue to this configuration variable.

Good thing is that this PR only gives you more control, so by default you will be using default config. I'm really curious on why it works for you and not for us.

@craigds are you doing something special for monkey patching or just relaying on the default gunicorn patching?

Uxio0 commented 2 years ago

@craigds this would be the reason, as they use concurrent.futures.ThreadPoolExecutor (https://github.com/boto/s3transfer/blob/develop/s3transfer/futures.py#L407):

http://www.gevent.org/api/gevent.monkey.html#gevent.monkey.patch_thread