search

etianen / django-s3-storage

Django Amazon S3 file storage.
BSD 3-Clause "New" or "Revised" License
414 stars 94 forks source link

File is closed after s3 upload #141

Closed stumitchell closed 2 years ago

stumitchell commented 2 years ago

hi I am seeing this issue with your library

https://github.com/jschneier/django-storages/issues/382

stumitchell commented 2 years ago 
class CustomS3Storage(S3Storage):

    def _save(self, name, content):
        """
        We create a clone of the content file as when this is passed to boto3 it wrongly closes
        the file upon upload where as the storage backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified size
        content_autoclose = self.new_temporary_file()

        # Write our original content into our copy that will be closed by boto3
        content_autoclose.write(content.read())
        content_autoclose.file = content.file

        # Upload the object which will auto close the content_autoclose instance
        result = super()._save(name, content_autoclose)

        # Cleanup if this is fixed upstream our duplicate should always close
        if not content_autoclose.closed:
            content_autoclose.close()

        return result

storage = CustomS3Storage()

Seems to help

etianen commented 2 years ago

I'll push out a patch release momentarily

jakobkarlstrand commented 2 years ago

I'll push out a patch release momentarily

Hi, is there a solution for this? I get ValueError I/O while uploading to S3

etianen commented 2 years ago

Can I have your full traceback?

On Tue, 19 Jul 2022 at 08:57, jaskop @.***> wrote:

I'll push out a patch release momentarily

Hi, is there a solution for this? I get ValueError I/O while uploading to S3

— Reply to this email directly, view it on GitHub https://github.com/etianen/django-s3-storage/issues/141#issuecomment-1188720942, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEKCHQPN2T664R3YFV6MLVUZNFNANCNFSM52ECLTPA . You are receiving this because you modified the open/close state.Message ID: @.***>

jakobkarlstrand commented 2 years ago

Actually, it worked now. However I now get an UnicodeDecodeError at /api/jour-reports

Traceback (most recent call last):
  File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 145, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 143, in _get_response
    response = response.render()
  File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/django/template/response.py", line 105, in render
    self.content = self.rendered_content
  File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/rest_framework/response.py", line 70, in rendered_content
    ret = renderer.render(self.data, accepted_media_type, context)
  File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/rest_framework/renderers.py", line 100, in render
    ret = json.dumps(
  File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/rest_framework/utils/json.py", line 25, in dumps
    return json.dumps(*args, **kwargs)
  File "/opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 234, in dumps
    return cls(
  File "/opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/opt/homebrew/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/rest_framework/utils/encoders.py", line 50, in default
    return obj.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 50: invalid start byte
etianen commented 2 years ago

I don't think this has anything to do with django-s3-storages.

Your original problem was fixed in the latest release, as you've found. This seems to be some sort of unicode problem with data stored in a model somewhere.

On Tue, 19 Jul 2022 at 20:20, jaskop @.***> wrote:

Actually, it worked now. However I now get an UnicodeDecodeError at /api/jour-reports

Traceback (most recent call last): File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 34, in inner response = get_response(request) File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 145, in _get_response response = self.process_exception_by_middleware(e, request) File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 143, in _get_response response = response.render() File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/django/template/response.py", line 105, in render self.content = self.rendered_content File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/rest_framework/response.py", line 70, in rendered_content ret = renderer.render(self.data, accepted_media_type, context) File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/rest_framework/renderers.py", line 100, in render ret = json.dumps( File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/rest_framework/utils/json.py", line 25, in dumps return json.dumps(args, kwargs) File @./3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/init.py", line 234, in dumps return cls( File @./3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File @./3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/Users/jakobkarlstrand/Documents/Programming/weknowit/HaningeBostader/backend/venv/lib/python3.9/site-packages/rest_framework/utils/encoders.py", line 50, in default return obj.decode() UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 50: invalid start byte

— Reply to this email directly, view it on GitHub https://github.com/etianen/django-s3-storage/issues/141#issuecomment-1189463606, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABEKCBWDJO4FSMA7U2OKO3VU35WXANCNFSM52ECLTPA . You are receiving this because you modified the open/close state.Message ID: @.***>

stumitchell commented 2 years ago

Hi I am still seeing this issue in version 0.13.7 I will continue to investigate

etianen commented 2 years ago

@stumitchell Cab you let me know if this is fixed for you in the latest commit to the master branch?

jakobkarlstrand commented 2 years ago

For me it was the following error:

I tried to return the file, or information of the file, in a response. Check if you are doing something similar