search

garethgeorge / backrest

Backrest is a web UI and orchestrator for restic backup.
GNU General Public License v3.0
1.73k stars 50 forks source link

error: Failed, orchestrator killed while operation was in progress #398

Closed Gt1ult closed 3 months ago

Gt1ult commented 4 months ago

I am using the last version of backrest via docker. I am doing the backup via shh from one server to another on the same network. I am doing the backup of 600gb of pictures, but the process is lasting a lot as I had so far 6 times the error "error: Failed, orchestrator killed while operation was in progress." The destination server is connected via wifi to the network using Fenvi FU-AX1800 dongle (main OS is openmediavault). At the moment I am running the command "restic repair index" before resuming backup. Is it correct? I can I avoid this annoying issue? Thanks

jakemauer commented 4 months ago

Just wanted to chime in that I'm running into this issue as well. Using the backrest docker container with an SFTP repo.

ianwyatt commented 4 months ago

I’m seeing the same error, running backrest docker on Unraid backing up to a external drive

Gt1ult commented 3 months ago

Any advice to avoid this issue, it seems it does not appear with restic cli

garethgeorge commented 3 months ago

This error indicates that backrest was shutdown unexpectedly before an operation completed typically

Gt1ult commented 3 months ago

it was not shot down by me for sure, neither from docker I suppose

garethgeorge commented 3 months ago

If possible, please upload logs from the backrest process around the time that you run into the error e.g. from docker logs -f or similar. In particular I'd like to see the log messages immediately before the container restart and immediately after it's next startup.

From the behavior you all are seeing it sounds like backrest is shutting down unexpectedly (there should be either a crash or graceful but unexpected shutdown somewhere in the logs) and similarly backrest checks for and logs any partial operations it finds in the operation log on startup. That should give me some insight into what's going on here.

It's worth noting: there are some expected cases where this happens e.g. an accidental power outage, etc. But it can also be a sign of a backrest bug i.e. a crash. You can also check container uptime to see if it restarted since you last created it if using docker.

Note: on linux logs are available from the systemd unit e.g the command should be journalctl -u backrest.service

garethgeorge commented 3 months ago

I suspect this may have been related to the race condition patched out in the last release, closing for now. Please reopen if still seeing crashes.