Retry on connection failures

ghost commented 2 years ago

Description

I tried to make a backup using Vorta and make my computer run through the night, but after some time upload process ended with a bug. It happened two times, while uploading and at 42GB and at 103GB.

Creating backup again makes it continue when ended, but I don't know if the backup will not be corrupted.

OS

Fedora 36, KDE Plasma 5.25.5

Version of Vorta

v0.8.7

What did you install Vorta with?

Flatpak

Version of Borg

No response

Logs

2022-09-20 05:21:13,902 - vorta.borg.borg_job - WARNING - Remote: client_loop: send disconnect: Broken pipe
2022-09-20 05:21:13,917 - vorta.borg.borg_job - ERROR - Local Exception
2022-09-20 05:21:13,920 - vorta.borg.borg_job - ERROR - Traceback (most recent call last):
  File "borg/archiver.py", line 183, in wrapper
  File "borg/archiver.py", line 668, in do_create
  File "borg/archiver.py", line 605, in create_inner
  File "borg/archiver.py", line 813, in _rec_walk
  File "borg/archiver.py", line 813, in _rec_walk
  File "borg/archiver.py", line 813, in _rec_walk
  File "borg/archiver.py", line 779, in _rec_walk
  File "borg/archiver.py", line 681, in _process_any
  File "borg/archive.py", line 1407, in process_file
  File "borg/archive.py", line 1234, in process_file_chunks
  File "borg/archive.py", line 1222, in chunk_processor
  File "borg/cache.py", line 947, in add_chunk
  File "borg/remote.py", line 476, in do_rpc
  File "borg/remote.py", line 709, in call
  File "borg/remote.py", line 826, in call_many
borg.remote.ConnectionClosed: Connection closed by remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "borg/archiver.py", line 5089, in main
  File "borg/archiver.py", line 5020, in run
  File "borg/archiver.py", line 183, in wrapper
  File "borg/remote.py", line 634, in __exit__
  File "borg/remote.py", line 476, in do_rpc
  File "borg/remote.py", line 709, in call
  File "borg/remote.py", line 779, in call_many
  File "borg/remote.py", line 719, in send_buffer
  File "borg/remote.py", line 417, in write
BrokenPipeError: [Errno 32] Broken pipe

m3nu commented 2 years ago

Hi, this is a network interruption. You can just proceed with your backup. Borg will do a checkpoint every 30 minutes, so you don't loose much progress.

Not to self: We could implement automatic retrying for such cases, like Borgmatic already does.

ghost commented 2 years ago

Thanks, good to know. Automatic retrying should be there, because leaving machine to do big backup on slow connection, only to get back next day and see some error leaves bad impressions. Also some dialog saying what's going on would be nice too.

real-yfprojects commented 2 years ago

If borg exits with an error code greater than 1, vorta will retry the backup after maximum an hour. This is not the case with the error code 1 since that only indicates a warning, not a failed backup.

Can you enclose some more log lines after the error?

borgbase / vorta