Archives on listing don't survive to container re-creation

[tmm360]

I'm using Docker Swarm to maintain a borgmatic instance. I'm still experimenting with it, but I'm having some issues understanding how cache is managed.

This is an extract of my docker swarm document:

version: "3"

services:

  admin_borgmatic:
    image: ghcr.io/borgmatic-collective/borgmatic
    environment:
      - BORG_PASSPHRASE=<change_me> # <-- change
      - TZ=Europe/Zurich
    volumes:
      - ./borgmatic/.ssh:/root/.ssh           # ssh key for remote repositories         ## <-- put ssh key
      - ./borgmatic/admin:/etc/borgmatic.d    # borgmatic config file(s)
      - admin_borg-cache:/root/.cache/borg    # checksums used for deduplication
      - admin_borg-config:/root/.config/borg  # borg config and keyfiles

      # backup sources
      - source-volume:/mnt/source-volume:ro                      # my source volume

volumes:
  admin_borg-cache:
  admin_borg-config:
  source-volume:
    external: true

my config.yaml file:

location:
  source_directories:
    - /mnt/source-volume

  repositories:
    - ssh://<myUserId>@<myUserId>.repo.borgbase.com/./repo

storage:
  compression: auto,zstd
  archive_name_format: '{hostname}-{now:%Y-%m-%d-%H%M%S}'
  retries: 5
  retry_wait: 5

retention:
  keep_daily: 7
  keep_weekly: 4
  keep_monthly: 12
  keep_yearly: 10

consistency:
  checks:
    - name: repository
      frequency: 4 weeks
    - name: archives
      frequency: 8 weeks
  check_last: 3

hooks:
  before_backup:
    - echo "Starting a backup job."
  after_backup:
    - echo "Backup created."
  on_error:
    - echo "Error while creating a backup."

Backup works without problems, I can list, extract, etc.
Problems come when I decide to stop for test the container. Docker swarm creates another, but accessing into the new one if I execute borgmatic list I only receive an empty list of archives (even if the repository is recognized).

What is strange is that running borg list ssh://<myUserId>@<myUserId>.repo.borgbase.com/./repo I can see all of them, the olds and eventually also the new if created, but borgmatic is only able to list archives created with this same container. Am I missing something? How can I recreate locally the cache of borgmatic to recover also listing of missing archives?

[witten]

Drive-by comment.. Also, a disclaimer: I've never actually used Swarm.

My understanding is that a given volume isn't shared across Swarm nodes by default. It just gets mounted on the node where the container runs. So for instance, when you start the borgmatic container initially, the cache volumes gets mounted onto one particular node's local filesystem. But then when you recreate the container on another node, it doesn't have access to any of the first node's cache files. Or.. are you using a different volume driver that actually distributes volume data across Swarm nodes?

But in any case, this cache issue shouldn't be a huge problem.. Worst case, it should just be inefficient as cache data gets recreated each time borgmatic spins up on a different node. See this FAQ entry for more information.

I can see all of them, the olds and eventually also the new if created, but borgmatic is only able to list archives created with this same container.

I think this is your problem:

  archive_name_format: '{hostname}-{now:%Y-%m-%d-%H%M%S}'

You see the {hostname} there? That means every archive gets created with the current Swarm node's hostname within the archive name. Then, due to borgmatic's automatic archive filtering, those archives get filtered out of your list results when you run borgmatic on a different node with a different hostname! You can fix that by removing {hostname} from your archive_name_format and using something else that doesn't change per-host.

Not sure what to do about your cache issue though...

[tmm360]

Thank you very much! Issue was with hostname filtering. Adding an explicit hostname on docker config solves the problem. :)

[witten]

Glad to hear that did it!