Backup of a big storage eating up space in `var/lib/docker/overlay`

[MrModest]

Describe the bug I have three different drives:

1. SSD for root (mounted as ext4 in /)
2. SSD for applications (mounted as zfs pool in /mnt/pools/fast)
3. HDD for big files (mounted as zfs pool in /mnt/pools/slow)

I can't backup a folder from the bigger one (3) with ~300Gb of size because backrest taking up space on the root drive (1).

[!TIP] My assumption is that backrest writing some temporary files inside the container somewhere which end up with copying the entire folder to the root system. If I would know which folder it is, I'd mount it to the bigger drive, so no space in the root drive will be used.

To Reproduce Steps to reproduce the behavior:

1. Use a relatively small drive for root system (or wherever you store your docker root folder)
2. Mount a bigger drive to the container
3. Try to backup a folder (that takes more space than entire root drive) from the bigger drive to another folder in the same bigger drive
4. See your root drive run out of space with /var/lib/docker/overlay folder taking all space

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Platform Info

• OS: Ubuntu 22.04.4 LTS (x86_64)
• Backrest Version: 1.4.0
• ZFS version: 2.1.5-1ubuntu6~22.04.4
• Docker Storage Driver: overlay2 (tried before with zfs - had similar issue)

Additional context Compose file:

services:
  backrest:
    image: garethgeorge/backrest:${APP_VERSION}
    container_name: backrest
    hostname: backrest
    volumes:
      - ${APP_DATA_PATH}/data:/data
      - ${APP_DATA_PATH}/config:/config
      - ${APP_DATA_PATH}/cache:/cache
      - ${APP_DATA_PATH}/rclone:/app/rclone

      - /home:/hostfs/home:ro
      - /mnt/pools:/hostfs/mnt/pools:ro

      - ${REPO_PATH}/repos:/repos
    env_file:
      - .env
    environment:
      - BACKREST_DATA=/data # the restic binary and the database
      - BACKREST_CONFIG=/config/config.json # backrest config file.
      - XDG_CACHE_HOME=/cache # restic cache
      - TZ=${TIMEZONE} # used for cron jobs
      - RCLONE_CONFIG=/app/rclone/rclone.conf
    restart: unless-stopped

networks:
  default:
    external: true
    name: nginxnetwork

ENVs:

• APP_VERSION=v1.4.0
• APP_DATA_PATH=/mnt/pools/fast/apps-data/backrest
• REPO_PATH=/mnt/pools/slow/backups/backrest

Docker daemon.json

{
  "storage-driver": "overlay2",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "1m",
    "max-file": "1"
  },
  "metrics-addr" : "0.0.0.0:9323",
  "experimental" : true
}

output of df -h

Filesystem                         Size  Used Avail Use% Mounted on
tmpfs                              783M   84M  699M  11% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   57G   57G     0 100% /
tmpfs                              3.9G     0  3.9G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
/dev/sda2                          2.0G  255M  1.6G  14% /boot
/dev/sda1                          1.1G  6.1M  1.1G   1% /boot/efi
fast-tank                          221G  6.1G  215G   3% /mnt/pools/fast
fast-tank/docker                   215G  7.5M  215G   1% /mnt/pools/fast/docker
fast-tank/docker/data-root         217G  1.7G  215G   1% /mnt/pools/fast/docker/data-root
slow-tank                          1.8T  430G  1.4T  24% /mnt/pools/slow
fast-tank/apps-data                215G  128K  215G   1% /mnt/pools/fast/apps-data
fast-tank/apps-data/backrest       215G   25M  215G   1% /mnt/pools/fast/apps-data/backrest
fast-tank/apps-data/homepage       215G  128K  215G   1% /mnt/pools/fast/apps-data/homepage
fast-tank/apps-data/immich         217G  1.4G  215G   1% /mnt/pools/fast/apps-data/immich
fast-tank/apps-data/nginx          215G   45M  215G   1% /mnt/pools/fast/apps-data/nginx
fast-tank/apps-data/observability  216G  554M  215G   1% /mnt/pools/fast/apps-data/observability
fast-tank/apps-data/portainer      215G  512K  215G   1% /mnt/pools/fast/apps-data/portainer
fast-tank/apps-data/paperless      215G   92M  215G   1% /mnt/pools/fast/apps-data/paperless
tmpfs                              783M  4.0K  783M   1% /run/user/1004

ncdu checking used space

config's repo:

{
  "id":  "immich__local",
  "uri":  "/mnt/pools/slow/backups/backrest/repos/immich",
  "password":  "<PASSWORD>",
  "prunePolicy":  {
    "maxUnusedPercent":  10
  },
  "checkPolicy":  {
    "schedule":  {
      "cron":  "0 0 3-27/8 * *"
    },
    "readDataSubsetPercent":  0
  },
  "commandPrefix":  {
    "ioNice":  "IO_BEST_EFFORT_LOW"
  }
}

config's plan

{
  "id":  "immich__local",
  "repo":  "immich__local",
  "paths":  [
    "/hostfs/mnt/pools/slow/apps-data/immich/external",
    "/hostfs/mnt/pools/slow/apps-data/immich/upload/library",
    "/hostfs/mnt/pools/slow/apps-data/immich/upload/upload"
  ],
  "schedule":  {
    "cron":  "0 3 * * 2"
  },
  "retention":  {
    "policyKeepLastN":  3
  },
  "backup_flags":  [
    "--compression max"
  ]
}

A bit more details about the setup:

• https://github.com/MrModest/homeserver/blob/main/docs/Files%20Structure.md
• https://github.com/MrModest/homeserver/tree/main/roles/deploy_backrest

[MrModest]

Also noticed a weird behavior: after restarting the container, the created repo folder is gone 🤔

Can't see it in /mnt/pools/slow/backups/backrest/repos anymore

[garethgeorge]

Thanks for the detailed report context! Just as a quick sanity check :) I'm noticing your configured storage URI is a local path /mnt/pools/slow/backups/backrest/repos/immich but your storage is mounted at simply /repos in the container

Should this be something like /repos/immich / would that explain what you're seeing?

[MrModest]

Sorry, I'm not sure if I get you fully, but if you check the compose file's volumes, you will see that I mount the entire repos folder which can contain not only the immich subfolder:

/mnt/pools/slow/backups/backrest/repos:/repos

There're also another repository in the /repos folder. But I haven't seen any issues with it so far since it backups ~2Gb of data :)

[garethgeorge]

I'm thinking in your repo config

{
  "id":  "immich__local",
  "uri":  "/mnt/pools/slow/backups/backrest/repos/immich",
  "password":  "<PASSWORD>",
  "prunePolicy":  {
    "maxUnusedPercent":  10
  },
  "checkPolicy":  {
    "schedule":  {
      "cron":  "0 0 3-27/8 * *"
    },
    "readDataSubsetPercent":  0
  },
  "commandPrefix":  {
    "ioNice":  "IO_BEST_EFFORT_LOW"
  }
}

I think your uri should be /repos if that's the directory you're mounting in the container, I think you're possibly ending up creating a repo inside the container's filesystem (somewhere under /mnt) instead of in/repos` on the host?

[MrModest]

I don't think so. I can clearly see (from both: host and container perspectives) another repo (called main) that is also inside /repos dir.

For example:

[MrModest]

oh, I'm stupid! I just got what are you trying to say, and I believe you are right. let me try it again 😅

And that also explains why the repo disappears after the container restarts.

I didn't catch your point because my first repo was configured correctly, but when I was configuring the second one, something happened with my mind :D

[MrModest]

Thank you for noticing this, and sorry for taking your time. I will close the issue :)