Show native restic's output

[rmoriz]

Expected behavior

When running autorestic -v backup -a I expect to see restic's native progress output. With autorestic currently I only see the command that is being executed, e.g. > Executing: /opt/homebrew/bin/restic backup --tag ar:location:.. /path but nothing from restic itself. So I have no indication how long it will take, how many files were processed etc.

Environment

• macOS 12.1
• Apple M1
• autorestic 1.5.1 (with two remotes (sftp) configured)
• restic 0.12.1
• zsh

Additional context

Documentation at https://autorestic.vercel.app/cli/general says:

Verbose mode will show the output of the native restic commands that are otherwise not printed out. Useful for debugging or logging in automated tasks.

[mdeboer]

I can confirm this also happens on Linux with 1.5.1 (with local and b2 remotes).

With a quick look it seems to happen here where the output is buffered until the command is done instead of being output continuously. My guess is this could hog quite some memory over time with larger jobs (and thus more output).

https://github.com/cupcakearmy/autorestic/blob/27758a03fa78982de3522a343f2fc32d1ab8c602/internal/utils.go#L43-L51

[cupcakearmy]

You are both right. An option would be to stream the output directly to stdout probably.

[mdeboer]

That would be really helpful. I ended up postponing the initial backup to a new backend to when I was asleep. Could have taken an hour or maybe 8, no idea 🤣

My bet is this won't be too hard to implement, streaming it directly should solve it and then just return the exit code in the method instead of the output so action can be taken when an error occurred.

[varac]

Awesome, works great :partying_face:

[varac]

Hugh, it works for backup, but not for the check cmd:

$ autorestic -vvv --ci check 
Using config paths: . /root autorestic
Using config:   /etc/autorestic/.autorestic.yml
Using env:  /etc/autorestic/.autorestic.env
Using lock: /etc/autorestic/.autorestic.lock.yml
> Executing: /root/bin/restic --verbose check
Initializing backend "pinenaspub"...
> Executing: /root/bin/restic --verbose init
Error: exit status 1

[aniqueta]

I'm having a similar issue to above. The check command fails with Error: exit status 1 however backup with the same configuration works without issue though… So without more logging I cannot determine why this is not working. I am also trying to initiate a new backend for the same repositories, and using the check command to trigger restic init fails due to this issue. Thanks for assisting with this.

[peterrus]

I am seeing output for backup but only the 'non streaming' messages like using parent snapshot xyz123. I am not seeing the live progress output that I would see when running Restic directly on the CLI. You can however force the printing of a single status update using kill -s USR1 <process id of Restic>. This should work on most UNIX-y systems and does not (contrary to how it looks) kill the Restic process, it just sends a USR1 signal as described here: https://restic.readthedocs.io/en/latest/manual_rest.html (ctrl+f USR1).

Edit: After giving this some more thought: I am fairly sure that Restic does not print the live progress output because it detects that it is not running on an interactive terminal. I believe setting the RESTIC_PROGRESS_FPS=1 environment variable will give you live output every second. I have not tried this with autorestic, but it does work with a python wrapper around Restic I once hacked on.

@cupcakearmy it might be interesting to set this automatically when not running in cron mode.

[claytono]

@peterrus I tested it and RESTIC_PROGRESS_FPS does work as you suggested with autorestic.

[DoNotResuscitate]

@aniqueta I am still getting this issue with the check command. I had setup a backblaze b2 backend, and the initial check worked fine. I have now made a backup, and a following check command fails with the Error: exit status 1

Edit: Turns out when doing autorestic exec -av -- check it showed me I had a stale lock file. Autorestic really needs to show the full native restic output.