search

juliomalegria / django-chunked-upload

Upload large files to Django in multiple chunks, with the ability to resume if the upload is interrupted.
MIT No Attribution
214 stars 71 forks source link

Will this work with s3 #28

Open tawanda opened 7 years ago

tawanda commented 7 years ago

will this download the whole file to local disc first and then send it to s3?

mrmachine commented 7 years ago

I'm trying to switch a project to S3 (using https://github.com/jschneier/django-storages) and I'm getting an exception:

Traceback:
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py" in get_response
  112.                    ', ' response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/db/transaction.py" in inner
  371.                 return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/views/generic/base.py" in view
  69.             return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/csrf.py" in wrapped_view
  57.         return view_func(*args, **kwargs)
File "/opt/syrinscape/djangosite/api/views.py" in dispatch
  251.         return super(CreatorUploadView, self).dispatch(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/views/generic/base.py" in dispatch
  87.         return handler(request, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/chunked_upload/views.py" in post
  93.             return self._post(request, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/chunked_upload/views.py" in _post
  205.         chunked_upload.append_chunk(chunk, chunk_size=chunk_size, save=False)
File "/usr/local/lib/python2.7/dist-packages/chunked_upload/models.py" in append_chunk
  79.         self.file.open(mode='ab')  # mode = append+binary
File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/files.py" in open
  76.         self.file.open(mode)
File "/usr/local/lib/python2.7/dist-packages/django/core/files/base.py" in open
  126.             raise ValueError("The file cannot be reopened.")

It seems that django-chunked-upload is attempting to work around a bug in Django 1.4 where inner files were not being closed properly (https://github.com/juliomalegria/django-chunked-upload/blob/master/chunked_upload/models.py#L74-L76), but I didn't see an issue or much explanation about this change.

In my local testing, after closing the inner files, I am unable to re-open them again using self.file.open().

Using self.file.close() and self.file.open(mode='ab') appears to work, at least under Django 1.8 with the default file based storage backend, so perhaps this fix is no longer required as Django 1.4 is ancient and has not been supported for a while now.

However, with django-storages, self.file.close() followed by self.file.open(mode='ab') doesn't throw an exception, but appears to not seek the file to the end (for appending).

@juliomalegria could you provide any guidance on what changes I might need to make to django-chunked-upload to support S3/boto backends via django-storages, and whether or not the Django 1.4 fix is still required?

I think that using S3 for storage is very common these days, especially as people are switching projects to Docker to make them scalable which generally requires a shared remote storage backend for media, so it would be great if django-chunked-upload supported that.

juliomalegria commented 7 years ago

The hack fix for Django 1.4 was removed in 6889442. Did that solve the issue with S3?