Closed thomastu closed 3 years ago
@Zharktas or @amercader, any chance you can look at this?
Yes, I'll check it out @TkTech
Just to clarify, this isn't a bug with this codebase, but I don't think the real underlying problem will get solved in upstream python, so there would be value in handling it within this specific application. Here's a small example reproducing the root cause of the bug: (assumes an s3 mock running with localstack on port 4572.)
# Start localstack (on fish shell)
env SERVICES=s3 localstack start --host
# bash: `SERVICES=s3 localstack start --host`
# Make the bucket
aws --endpoint-url=http://localhost:4572 s3 mb s3://my-bucket
"""This snippet reproduces what is effectively happening in
https://github.com/TkTech/ckanext-cloudstorage/blob/master/ckanext/cloudstorage/storage.py#L246
"""
from libcloud.storage.providers import get_driver
from libcloud.storage.types import Provider
from tempfile import SpooledTemporaryFile
if __name__ == "__main__":
S3Driver = get_driver(Provider.S3_US_WEST)
driver = S3Driver("key", "secret", host="localhost", port=4572, secure=False)
container = driver.get_container("my-bucket")
with SpooledTemporaryFile() as f:
f.write(b"hello world!")
f.seek(0)
# This will hang forever - debugging shows that
# "<method-wrapper 'next' of cStringIO.StringO object at 0x103435a78>"
# is just getting written over and over again
container.upload_object_via_stream(f, "test.txt")
Motivation
I'm seeing a crash when using S3 after PR #35. Files now hang infinitely when trying to upload. This is on CKAN 2.8 using python 2.7 with the following cloudstorage related settings:
Diagnosis
I think the crash is ultimately coming from how the
SpooledTemporaryFile
implementsnext
in python 2.7. It looks likeSpooledTemporaryFile.next()
returns a callable method, when it should just be wrapping the function (i.e.SpooledTemporaryFile.next()
returnsself._file.next
when it should returnself._file.next()
.) In cloudstorage.storage._get_underlying_file, the underlyingfile.stream
is aSpooledTemporaryFile
, and so when it gets passed off to libcloud and callsupload_object_via_stream
, libcloud will callnext()
on this file stream expecting it to eventually stop iterating. Sincenext()
just returns a callable, it goes on forever. I verified this by hacking around and changing L25 above to be the followingand uploads started working once more.