Healthchecks 'success' not sent when borg returns 1 (which is only a warning)

[adhawkins]

If borg completes with a warning (say a file disappeared during the backup, so could not be processed) it appears that the healthcheck 'finished' ping isn't sent.

Borg's return codes are documented here:

https://borgbackup.readthedocs.io/en/stable/usage/general.html#return-codes

It looks like the 'success' ping is only sent if borg returns zero:

https://github.com/KenKundert/emborg/blob/87d1624bf6b2c2d03602c80eafe1d1e2446f23bc/emborg/hooks.py#L60-L66

[KenKundert]

Okay, I think this is fixed and checked in to GitHub. Give it a try and let me know whether it meets your expectations.

Thanks!

[adhawkins]

The code change looks good, but I'll have to see if I can engineer a test for this, as I don't get these '1' return codes very often.

[KenKundert]

I had the same concern, but then accidentally stumbled on the answer minutes before starting to work on this. Simply create a file in the directory to be backed up and make it unreadable (chmod 000). Then run the backup.

On December 10, 2021 1:23:38 AM PST, Andy Hawkins @.***> wrote:

The code change looks good, but I'll have to see if I can engineer a test for this, as I don't get these '1' return codes very often.

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/KenKundert/emborg/issues/55#issuecomment-990768441

[adhawkins]

Update, I inadvertantly re-created it by deleting the file I'd created before the backup had got round to actually processing it. My healthchecks still shows that backup job as having started but not completed, which suggests no 'finished' ping was sent.

borg 1.1.17
emborg 1.28.1 (2021-12-10)

Running Borg create command ...
/backups/pcbackup/badfile: stat: [Errno 2] No such file or directory: '/backups/pcbackup/badfile'
------------------------------------------------------------------------------
Archive name: 2021-12-13-14-01-28
Archive fingerprint: ***
Time (start): Mon, 2021-12-13 14:01:30
Time (end):   Mon, 2021-12-13 14:15:47
Duration: 14 minutes 16.54 seconds
Number of files: 37611
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                1.48 TB              1.41 TB            143.24 kB
All archives:               17.76 TB             16.94 TB              1.40 TB

                       Unique chunks         Total chunks
Chunk index:                  585253              7126998
------------------------------------------------------------------------------
emborg error: create: unexpected exit status (1).

/ # emborg -c pcbackup log
emborg: version: 1.28.1 (2021-12-10)
emborg: invoked as: /usr/bin/emborg -c pcbackup create --fast --stats
emborg: log opened on Monday, 13 December 2021 at 2:29:37 PM GMT.
reading: /root/.config/emborg/settings
reading: /root/.config/emborg/pcbackup
working dir = /
running create command
staging run_before_backup[0] pre-backup script
running: borg --version; echo "emborg $(emborg --version)"
exit status: 0
signaling start of backups to healthchecks.io: https://healthchecks.gently.org.uk:7000/ping/ed3d4796-d0e5-477e-9246-adcc120c370a/start.
Setting BORG_PASSPHRASE.
Borg-related environment variables: {
    'BORG_PASSPHRASE': '<redacted>',
    'BORG_DISPLAY_PASSPHRASE': 'no',
}
running:
    borg create \
        --stats \
        --stats \
        --remote-ratelimit 3000 \
        'jyny0mfs@jyny0mfs.repo.borgbase.com:repo::{now:%Y-%m-%d-%H-%M-%S}' \
        /backups/pcbackup
running in: /

Running Borg create command ...
starts at: 2021-12-13T14:29:37.811826+00:00
Borg terminates with exit status: 1
Unsetting BORG_PASSPHRASE.
emborg error: create: unexpected exit status (1).
emborg: terminates with status 2.
emborg: log closed on Monday, 13 December 2021 at 2:31:01 PM GMT.

I can see in the log that it signalled the start, but there is no reference to any signalling of the end of the backup.

It also doesn't seem to be running borg-space at the end of the backup as I would expect it to.

[adhawkins]

Done some digging, and it seems that if borg exits with status '1', then this causes the exception handler to be triggered.

This just raises the exception up the chain, meaning that none of the 'end of backup' stuff is run. I suspect this also occurs if there's a 'real' error, meaning the healthchecks 'fail' event will never be signalled.

[KenKundert]

Whoops, I missed one case, the case where narrate is set. I have fixed and uploaded the new version to GitHub. The problem was not with the exception handling, rather run_borg() should have been modified so that it did not raise an exception when the Borg exit status was 1. That was only partially done with the first version. Hopefully this new version works as expected.

It may take me a while to promote this version to PyPi. I was unaware of Borg's exit status policy, but now that I know about it I think Emborg should adhere to the same policy. So I will take some time to implement changes to Emborg's exit status. Once that is done and documented I will promote this new version to PyPi.

[adhawkins]

Ok, that now does seem to send the completed ping to healthchecks. However, I see you're now sending the status, and healthchecks treats a status of 1 as a failure, triggering an email to that effect.

Would probably be better to only send a non-zero status to healthchecks if the status is >1, otherwise send a successful completion?

[KenKundert]

Sorry, I don't really use healthchecks, so I did not notice this detail. I have updated Emborg again. I have fixed your most recent issue and changed the behavior of emborg's exit status to match that of borg.

[adhawkins]

Ok, all looks good, both with and without -v.

Will keep an eye out for an official release, and update the Alpine package when I see it.

[KenKundert]

Thanks Andy, should be a couple of days. Let me know if you see anything unusual.

[KenKundert]

I have released the updated version to PyPi.

Thanks again!