search

jazzband / django-dbbackup

Management commands to help backup and restore your project database and media files
BSD 3-Clause "New" or "Revised" License
970 stars 221 forks source link

AttributeError: 'SpooledTemporaryFile' object has no attribute 'readinto' #378

Open ramtinabadi opened 3 years ago

ramtinabadi commented 3 years ago

I get the following error while trying backup a Postgres db and upload it to the S3. The S3 setup works fine.

`Writing file to default-something-2020-12-18-002957.psql AttributeError: 'SpooledTemporaryFile' object has no attribute 'readinto' File "/home/django/venv/lib/python3.8/site-packages/dbbackup/utils.py", line 118, in wrapper func(*args, kwargs) File "/home/django/venv/lib/python3.8/site-packages/dbbackup/management/commands/dbbackup.py", line 61, in handle self._save_new_backup(database) File "/home/django/venv/lib/python3.8/site-packages/dbbackup/management/commands/dbbackup.py", line 88, in _save_new_backup self.write_to_storage(outputfile, filename) File "/home/django/venv/lib/python3.8/site-packages/dbbackup/management/commands/_base.py", line 88, in write_to_storage self.storage.write_file(file, path) File "/home/django/venv/lib/python3.8/site-packages/dbbackup/storage.py", line 82, in write_file self.storage.save(name=filename, content=filehandle) File "/home/django/venv/lib/python3.8/site-packages/django/core/files/storage.py", line 52, in save return self._save(name, content) File "/home/django/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 446, in _save obj.upload_fileobj(reader, ExtraArgs=params) File "/home/django/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 619, in object_upload_fileobj return self.meta.client.upload_fileobj( File "/home/django/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 539, in upload_fileobj return future.result() File "/home/django/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result return self._coordinator.result() File "/home/django/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result raise self._exception File "/home/django/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 255, in _main self._submit(transfer_future=transfer_future, kwargs) File "/home/django/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 558, in _submit self._submit_multipart_request( File "/home/django/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 622, in _submit_multipart_request for part_number, fileobj in part_iterator: File "/home/django/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 270, in yield_upload_part_bodies fileobj, full_size = self._get_upload_part_fileobj_with_full_size( File "/home/django/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 337, in _get_upload_part_fileobj_with_full_size data = fileobj.read(kwargs['part_size']) File "/home/django/venv/lib/python3.8/site-packages/django/core/files/utils.py", line 17, in readinto = property(lambda self: self.file.readinto)

Traceback (most recent call last): File "manage.py", line 22, in main() File "manage.py", line 18, in main execute_from_command_line(sys.argv) File "/home/django/venv/lib/python3.8/site-packages/django/core/management/init.py", line 401, in execute_from_command_line utility.execute() File "/home/django/venv/lib/python3.8/site-packages/django/core/management/init.py", line 395, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/home/django/venv/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv self.execute(*args, cmd_options) File "/home/django/venv/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute output = self.handle(*args, *options) File "/home/django/venv/lib/python3.8/site-packages/dbbackup/utils.py", line 118, in wrapper func(args, kwargs) File "/home/django/venv/lib/python3.8/site-packages/dbbackup/management/commands/dbbackup.py", line 61, in handle self._save_new_backup(database) File "/home/django/venv/lib/python3.8/site-packages/dbbackup/management/commands/dbbackup.py", line 88, in _save_new_backup self.write_to_storage(outputfile, filename) File "/home/django/venv/lib/python3.8/site-packages/dbbackup/management/commands/_base.py", line 88, in write_to_storage self.storage.write_file(file, path) File "/home/django/venv/lib/python3.8/site-packages/dbbackup/storage.py", line 82, in write_file self.storage.save(name=filename, content=filehandle) File "/home/django/venv/lib/python3.8/site-packages/django/core/files/storage.py", line 52, in save return self._save(name, content) File "/home/django/venv/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 446, in _save obj.upload_fileobj(reader, ExtraArgs=params) File "/home/django/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 619, in object_upload_fileobj return self.meta.client.upload_fileobj( File "/home/django/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 539, in upload_fileobj return future.result() File "/home/django/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result return self._coordinator.result() File "/home/django/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result raise self._exception File "/home/django/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 255, in _main self._submit(transfer_future=transfer_future, **kwargs) File "/home/django/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 558, in _submit self._submit_multipart_request( File "/home/django/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 622, in _submit_multipart_request for part_number, fileobj in part_iterator: File "/home/django/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 270, in yield_upload_part_bodies fileobj, full_size = self._get_upload_part_fileobj_with_full_size( File "/home/django/venv/lib/python3.8/site-packages/s3transfer/upload.py", line 337, in _get_upload_part_fileobj_with_full_size data = fileobj.read(kwargs['part_size']) File "/home/django/venv/lib/python3.8/site-packages/django/core/files/utils.py", line 17, in readinto = property(lambda self: self.file.readinto) AttributeError: 'SpooledTemporaryFile' object has no attribute 'readinto' `

Settings.py

DBBACKUP_CONNECTORS = {
    'default': {
        'USER': os.environ['DB_USER'],
        'PASSWORD': os.environ['DB_PASSWORD'],
        'HOST': os.environ['DB_HOST']
    }
}

DBBACKUP_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DBBACKUP_STORAGE_OPTIONS = {
    "access_key": os.environ['AWS_ACCESS_KEY_ID'],
    "secret_key": os.environ['AWS_SECRET_ACCESS_KEY'],
    "bucket_name": os.environ['AWS_BUCKET_NAME'],
    'default_acl': 'private'
}

Versions

Django-dbbackup

External tools

jonathan-s commented 3 years ago

It looks like you're bumping into this issue described in the PR on comment here: https://github.com/python/cpython/pull/3249#issuecomment-719880320 As described in the comment a viable solution could be to monkeypatch SpooledTemporaryFile. If you've got time to test that I'd be keen to hear back.

jonathan-s commented 3 years ago

A longer term fix would fix this in django itself..