search

jazzband / django-dbbackup

Management commands to help backup and restore your project database and media files
BSD 3-Clause "New" or "Revised" License
975 stars 221 forks source link

Restore tempfile created: 0 B #463

Open rsommerard opened 1 year ago

rsommerard commented 1 year ago

Bug Report

Running the dbrestore command raise an error which is due to Restore tempfile created: 0 B.

Describe the bug

I use an FTP server as backup storage. Backup is store without issue on the FTP while the restore fail. If I download the file and run the command by filling the local path via the -I params it works well.

File is not encrypted, not compressed.

The command fail at: https://github.com/jazzband/django-dbbackup/blob/af443c5026a480fffd33c4bbb6b4b1981e6700cb/dbbackup/db/base.py#L157-L163

And here is the traceback:

Finding latest backup
Restoring backup for database 'default' and server 'None'
Restoring: default-me-myproject-2022-10-05-141541.psql.bin
Restore tempfile created: 0 B
Are you sure you want to continue? [Y/n] Y
Traceback (most recent call last):
  File "/home/me/myproject/.venv/lib/python3.8/site-packages/dbbackup/db/base.py", line 157, in run_command
    process = Popen(
  File "/usr/lib/python3.8/subprocess.py", line 808, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "/usr/lib/python3.8/subprocess.py", line 1477, in _get_handles
    p2cread = stdin.fileno()
io.UnsupportedOperation: fileno

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/me/.vscode/extensions/ms-python.python-2022.14.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/home/me/.vscode/extensions/ms-python.python-2022.14.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/home/me/.vscode/extensions/ms-python.python-2022.14.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/home/me/.vscode/extensions/ms-python.python-2022.14.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/home/me/.vscode/extensions/ms-python.python-2022.14.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/home/me/.vscode/extensions/ms-python.python-2022.14.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/home/me/myproject/manage.py", line 25, in <module>
    execute_from_command_line(sys.argv)
  File "/home/me/myproject/.venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/me/myproject/.venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/me/myproject/.venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/me/myproject/.venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/me/myproject/.venv/lib/python3.8/site-packages/dbbackup/management/commands/dbrestore.py", line 68, in handle
    self._restore_backup()
  File "/home/me/myproject/.venv/lib/python3.8/site-packages/dbbackup/management/commands/dbrestore.py", line 118, in _restore_backup
    self.connector.restore_dump(input_file)
  File "/home/me/myproject/.venv/lib/python3.8/site-packages/dbbackup/db/base.py", line 105, in restore_dump
    return self._restore_dump(dump)
  File "/home/me/myproject/.venv/lib/python3.8/site-packages/dbbackup/db/postgresql.py", line 124, in _restore_dump
    stdout, stderr = self.run_command(cmd, stdin=dump, env=self.restore_env)
  File "/home/me/myproject/.venv/lib/python3.8/site-packages/dbbackup/db/base.py", line 180, in run_command
    raise exceptions.CommandConnectorError(
dbbackup.db.exceptions.CommandConnectorError: Error running:  pg_restore --dbname=postgresql://postgres:postgres@localhost:30032/postgres --single-transaction --clean 
fileno

To Reproduce

  1. Run a local FTP server docker run -e FTP_USER_NAME=foo -e FTP_USER_PASS=pass -e FTP_USER_HOME=/home/foo -p 21:21 -p 30000-30009:30000-30009 -d stilliard/pure-ftpd
  2. Add the configuration 
    DBBACKUP_CONNECTORS = {
"default": {"CONNECTOR": "dbbackup.db.postgresql.PgDumpBinaryConnector"}
}
DBBACKUP_STORAGE = "storages.backends.ftp.FTPStorage"
DBBACKUP_STORAGE_OPTIONS = {"location": "ftp://foo:pass@localhost:21"}
  3. Backup the db: python manage.py dbbackup
  4. Restore: python manage.py dbrestore

Versions

Misc

The mediarestore command works with the same params (no compression, not encrypted)

rsommerard commented 1 year ago

FYI: for encrypted and compressed file, the same issue appears. A workaround is to add input_file.seek(0) after the input_file assignation in the decrypt block like so in dbrestore.py file: https://github.com/jazzband/django-dbbackup/compare/master...rsommerard:django-dbbackup:fix/0b-tempfile

This workaround only works for encrypted files process.

Here a solution when the file is not encrypted or compressed, don't know if it is a good one: https://github.com/rsommerard/django-dbbackup/commit/2d84111514d30ffb0c554ec7abe78e8fa4d68d1a

tobiasmcnulty commented 1 year ago

Confirming we are seeing the issue when both --uncompress and --decrypt are passed to dbrestore; since there is no input_file.seek(0) in between decrypting and decompressing the file, uncompress() doesn't read anything and hence returns an empty file.

Another workaround that doesn't require a code change is to disable compression when encryption is enabled. In our case that resulted in no measurable difference in the backup file size.