search

KenKundert / emborg

Interactive command line interface to Borg Backup
GNU General Public License v3.0
94 stars 8 forks source link

Does not pickup settings #51

Closed stratosgear closed 2 years ago

stratosgear commented 2 years ago

I have the following settings:

root@raspberrypi4:/home/pi/.config/emborg# cat settings 
# These settings are common to all configurations

# configurations
configurations = 'docker'
default_configuration = 'docker'

# encryption
encryption = 'keyfile'            # borg encryption method
    # Common choices are 'repokey' and 'keyfile'.
    # With 'repokey' the encryption key is copied into repository, use this
    # only if the remote repository is owned by you and is secure.
    # With 'keyfile' the encryption key is only stored locally. Be sure to
    # export it and save a copy in a safe place, otherwise you may not be
    # able to access your backups if you lose your disk.
# specify either passphrase or avendesora_account
passphrase = 'xxx'              # passphrase for encryption key
avendesora_account = '<<account-name>>'  # avendesora account holding passphrase

# basic settings
# specify notify if batch and notifier if interactive
notify = 'xxx@xxx.xx'        # who to notify when things go wrong
notifier = 'notify-send -u normal {prog_name} "{msg}"'
                                         # interactive notifier program
remote_ratelimit = 2000                  # bandwidth limit in kbps
prune_after_create = True                # automatically run prune after a backup
check_after_create = 'latest'            # automatically run check after a backup

# repository settings
repository = 'xxx:/home/xxx/borg_backups/{host_name}-{user_name}-{config_name}'
prefix = '{host_name}-'
    # These may contain {<name>} where <name> is any of host_name, user_name,
    # prog_name config_name, or any of the user specified settings.
    # Double up the braces to specify parameters that should be interpreted
    # directly by borg, such as {{now}}.
compression = 'lz4'

run_before_backup = 'service docker stop'
run_after_backup = 'service docker start'

healthchecks_uuid = 'xxx-xxx-xxx-xxx-xxx'

# filter settings
exclude_if_present = '.nobackup'
one_file_system = True
exclude_caches = True

# prune settings
keep_within = '1d'                       # keep all archives created in interval
keep_hourly = 48                         # number of hourly archives to keep
keep_daily = 14                          # number of daily archives to keep
keep_weekly = 8                          # number of weekly archives to keep
keep_monthly = 12                        # number of monthly archives to keep
keep_yearly = 1                          # number of yearly archives to keep

When I run emborg create, my docker service is not stopped while the backup takes place (I checked from another terminal). I assume the run_after_backup is not run neither.

Also, when I emborg settings I see:

root@raspberrypi4:/home/pi/.config/emborg# emborg settings
                   archive: '{host_name}-{{now}}'
                            'raspberrypi4-{now}'
        avendesora_account: '<<account-name>>'
        check_after_create: 'latest'
               compression: 'lz4'
                config_dir: '/root/.config/emborg'
               config_name: 'docker'
            configurations: 'docker'
     default_configuration: 'docker'
                encryption: 'keyfile'
            exclude_caches: True
        exclude_if_present: '.nobackup'
                  excludes: """
                                **/*~
                                **/__pycache__
                                **/*.pyc
                                **/.*.swp
                                **/.*.swo
                            """
                  home_dir: '/root'
                keep_daily: 14
               keep_hourly: 48
              keep_monthly: 12
               keep_weekly: 8
               keep_within: '1d'
               keep_yearly: 1
                   log_dir: '/root/.local/share/emborg'
                  notifier: 'notify-send -u normal {prog_name} "{msg}"'
                    notify: 'xxx@xxx.xx'
           one_file_system: True
                passphrase: '<set>'
                    prefix: '{host_name}-'
                            'raspberrypi4-'
        prune_after_create: True
          remote_ratelimit: 2000
                repository: 'xxx:/home/xxx/borg_backups/{host_name}-{user_name}-{config_name}'                                                                                                       
                            'xxx:/home/xxx/borg_backups/raspberrypi4-root-docker'
                  src_dirs: """
                                /var/lib/docker/volumes/xxx1
                                /var/lib/docker/volumes/xxx2
                                /var/lib/docker/volumes/xxx3
                            """

Same issue with the healthchecks_uuid config. There is no ping taking place!

emborg is run as the root user (so I can read the root owned docker volumes)

What am I doing wrong and why does emborg does not "see" my settings?

KenKundert commented 2 years ago

Hmmm, that is a puzzle. Presumably if you looked in the log file you would not see commentary indicating that the commands are being run. It appears as if Emborg is not getting those settings, though I do not understand why. I noticed that you showed me your shared settings file ( ~/.config/emborg/settings ) but not the settings file for the docker configuration ( ~/.config/emborg/docker ). Can you show me that file, and your log file?

stratosgear commented 2 years ago

Contents of docker is very simple:

root@raspberrypi4:~/.config/emborg# cat docker 
# Settings for home configuration
# use of absolute paths is recommended
src_dirs = '''
/var/lib/docker/volumes/xx1
/var/lib/docker/volumes/xx2
/var/lib/docker/volumes/xx3
'''

excludes = '''
    **/*~
    **/__pycache__
    **/*.pyc
    **/.*.swp
    **/.*.swo
'''                      # list of files or directories to skip

Are you talking about these logs?:

root@raspberrypi4:~/.config/emborg# emborg create
root@raspberrypi4:~/.config/emborg# emborg log
emborg: version 1.26.0 (2021-09-03)
emborg: invoked as: /usr/local/bin/emborg create
emborg: invoked on: Tuesday, 14 September 2021 at 1:04:19 PM CEST
working dir = /
running create command
Setting BORG_PASSPHRASE.
Borg-related environment variables: {
    'BORG_PASSPHRASE': '<redacted>',
    'BORG_DISPLAY_PASSPHRASE': 'no',
}
running:
    borg create \
        --exclude '**/*~' \
        --exclude '**/__pycache__' \
        --exclude '**/*.pyc' \
        --exclude '**/.*.swp' \
        --exclude '**/.*.swo' \
        --stats \
        --compression lz4 \
        --exclude-caches \
        --exclude-if-present .nobackup \
        --one-file-system \
        --remote-ratelimit 2000 \
        'xxx:/home/xxx/borg_backups/raspberrypi4-root-docker::raspberrypi4-{now}' \
        /var/lib/docker/volumes/xx1 \
        /var/lib/docker/volumes/xx2 \
        /var/lib/docker/volumes/xx3
running in: /
starts at: 2021-09-14T13:04:19.643807+02:00
ends at: 2021-09-14T13:04:47.738919+02:00
elapsed = 0:00:28.095112
Borg stderr:
    ------------------------------------------------------------------------------
    Archive name: raspberrypi4-2021-09-14T13:04:20
    Archive fingerprint: 25828854683b83bb613dadb0cfd9519ca984e984ad97641c0dbd
    Time (start): Tue, 2021-09-14 13:04:23
    Time (end):   Tue, 2021-09-14 13:04:45
    Duration: 22.83 seconds
    Number of files: 5865
    Utilization of max. archive size: 0%
    ------------------------------------------------------------------------------
                           Original size      Compressed size    Deduplicated size
    This archive:              155.44 MB             79.29 MB             25.03 MB
    All archives:              721.32 MB            380.92 MB            142.93 MB

                           Unique chunks         Total chunks
    Chunk index:                    5276                29547
    ------------------------------------------------------------------------------

update date file
Checking archive ...
Using existing BORG_PASSPHRASE.
Borg-related environment variables: {
    'BORG_PASSPHRASE': '<redacted>',
    'BORG_DISPLAY_PASSPHRASE': 'no',
}
running:
    borg list \
        --remote-ratelimit 2000 \
        --prefix raspberrypi4- \
        --json \
        xxx:/home/xxx/borg_backups/raspberrypi4-root-docker
running in: /root/.config/emborg
starts at: 2021-09-14T13:04:47.746434+02:00
ends at: 2021-09-14T13:04:50.732242+02:00
elapsed = 0:00:02.985808
Borg stdout:
    {
        "archives": [
            {
                "archive": "raspberrypi4-2021-09-13T21:57:14",
                "barchive": "raspberrypi4-2021-09-13T21:57:14",
                "id": "b609bf131f9b780557eeeab6eb1f3cf69f337a43a3741b65df7cdf",
                "name": "raspberrypi4-2021-09-13T21:57:14",
                "start": "2021-09-13T21:57:17.000000",
                "time": "2021-09-13T21:57:17.000000"
            },
            {
                "archive": "raspberrypi4-2021-09-13T22:07:55",
                "barchive": "raspberrypi4-2021-09-13T22:07:55",
                "id": "a33ef033705d18b5f86a4b6d7ccc871ab62ee3b66177229cee8deb",
                "name": "raspberrypi4-2021-09-13T22:07:55",
                "start": "2021-09-13T22:07:57.000000",
                "time": "2021-09-13T22:07:57.000000"
            },
            {
                "archive": "raspberrypi4-2021-09-13T22:17:47",
                "barchive": "raspberrypi4-2021-09-13T22:17:47",
                "id": "96df9a3a9c0c2c2cdee6150f8f135eefaa86bf4c8c0b56fa4817c",
                "name": "raspberrypi4-2021-09-13T22:17:47",
                "start": "2021-09-13T22:17:49.000000",
                "time": "2021-09-13T22:17:49.000000"
            },
            {
                "archive": "raspberrypi4-2021-09-13T22:26:45",
                "barchive": "raspberrypi4-2021-09-13T22:26:45",
                "id": "3b231ca280c9f06e7c543f63426ccf3f77aab3649f31adf09cfc",
                "name": "raspberrypi4-2021-09-13T22:26:45",
                "start": "2021-09-13T22:26:47.000000",
                "time": "2021-09-13T22:26:47.000000"
            },
            {
                "archive": "raspberrypi4-2021-09-14T13:04:20",
                "barchive": "raspberrypi4-2021-09-14T13:04:20",
                "id": "258288549d1002cdadb0aef1cfd9519ca984e984ad97641c0dbd",
                "name": "raspberrypi4-2021-09-14T13:04:20",
                "start": "2021-09-14T13:04:23.000000",
                "time": "2021-09-14T13:04:23.000000"
            }
        ],
        "encryption": {
            "keyfile": "/root/.config/borg/keys/xxx__home_xxx_borg_backups_raspberrypi4_root_docker",
            "mode": "keyfile"
        },
        "repository": {
            "id": "a8246409a0ed9fa26a043437dc11256e256d3b0f0862e2c8e0908b9",
            "last_modified": "2021-09-14T13:04:46.000000",
            "location": "ssh://xxx/home/xxx/borg_backups/raspberrypi4-root-docker"
        }
    }

Using existing BORG_PASSPHRASE.
Borg-related environment variables: {
    'BORG_PASSPHRASE': '<redacted>',
    'BORG_DISPLAY_PASSPHRASE': 'no',
}
running:
    borg check \
        --remote-ratelimit 2000 \
        --prefix raspberrypi4- \
        xxx:/home/xxx/borg_backups/raspberrypi4-root-docker::raspberrypi4-2021-09-14T13:04:20
running in: /root/.config/emborg
starts at: 2021-09-14T13:04:50.735911+02:00
ends at: 2021-09-14T13:04:56.335699+02:00
elapsed = 0:00:05.599788
Pruning archives ...
Using existing BORG_PASSPHRASE.
Borg-related environment variables: {
    'BORG_PASSPHRASE': '<redacted>',
    'BORG_DISPLAY_PASSPHRASE': 'no',
}
running:
    borg prune \
        --stats \
        --keep-within 1d \
        --keep-hourly 48 \
        --keep-daily 14 \
        --keep-weekly 8 \
        --keep-monthly 12 \
        --keep-yearly 1 \
        --remote-ratelimit 2000 \
        --prefix raspberrypi4- \
        xxx:/home/xxx/borg_backups/raspberrypi4-root-docker
running in: /root/.config/emborg
starts at: 2021-09-14T13:04:56.339941+02:00
ends at: 2021-09-14T13:04:59.367321+02:00
elapsed = 0:00:03.027380
Borg stderr:
    ------------------------------------------------------------------------------
                           Original size      Compressed size    Deduplicated size
    Deleted data:                    0 B                  0 B                  0 B
    All archives:              721.32 MB            380.92 MB            142.93 MB

                           Unique chunks         Total chunks
    Chunk index:                    5276                29547
    ------------------------------------------------------------------------------

Unsetting BORG_PASSPHRASE.
reading: /root/.config/emborg/settings
emborg: terminates with status 0 on Tuesday, 14 September 2021 at 1:04:59 PM CEST.

Note that I have redacted some strings (with xxx) and some of the ids.

KenKundert commented 2 years ago

I am at a complete loss. What you are seeing seems impossible. The settings command prints the value of every variable found in the settings files, so if it is in the file we should see in the output of the settings command. Is it possible that you are looking at the wrong file? I noticed that when you showed the contents of docker file the directory was given as ~/.config/emborg, but when you showed the contents of the settings file the directory was listed as /home/pi/.config/emborg. I think /home/pi is the default user's home directory, not root's. I believe root's home directory is /root.

stratosgear commented 2 years ago

You are very observant!!!

I was at a complete loss, too, and I have spent quite some time trying to figure this one out!

Indeed, the settings file I was editing was of the pi user and not of the root user. My bad for not cding to the root's .config folder when I was sudo su

Sorry, for bothering you with this, but I was getting crazy!

Thank you so much!