Unifi Controller does not Produce Backups

[Jeb-InnoVision]

Operating system

Ubuntu Linux 20.04.3 LTS

UniFi Tag

6.2.23

Docker run

docker-compose up -d

Bug description

The Unifi Controller is not creating backups, autobackups, nor will it download/create a backup when requested within the controller itself.

Backup files are not being generated in the volumes of the OS.

Placing old backups (created before moving to Docker Unifi) in the volumes directory are not visible as well.

Compose File docker-unifi-backup-issue_docker-compose.txt

Steps to reproduce

[ Setting up Autobackup ] Log into Unifi Controller as Super User Settings > Backup > [Configure settings for autobackup] *No backups are created as time progresses (based on settings)

[ Downloading Backup File ] Log into Unifi Controller as Super User Settings > Backup Select "Download Backup" button Button just spins (for hours) with no resolution Refreshing the page allows me to press the button again

Relevant log output

None. No errors or issues appear in the logs related to creating backups.

[mountaindude]

Fwiw, this bug is also occurring in 6.5.55 (which is latest as of writing this).

[jacobalberty]

@mountaindude can you provide a copy of your docker-compose or docker run command line?

This works on my setup so I want to look at narrowing down the common difference between my setup and you guys.

[mountaindude]

Of course!

Running Docker 20.10.12, build e91ed57 on Ubuntu 18.04.6 LTS

docker-compose is pretty much standard except I hardcoded the container names, plus a few extra labels (should not affect things).

version: '2.3'
services:
  mongo:
    image: mongo:3.6
    # container_name: ${COMPOSE_PROJECT_NAME}_mongo
    container_name: unifi_mongo
    networks:
      - unifi
    restart: always
    volumes:
      - db:/data/db
      - dbcfg:/data/configdb
  controller:
    # image: "jacobalberty/unifi:${TAG:-latest}"
    image: jacobalberty/unifi:latest
    # container_name: ${COMPOSE_PROJECT_NAME}_controller
    container_name: unifi_controller
    depends_on:
      - mongo
    init: true
    networks:
      - unifi
    restart: always
    volumes:
      - dir:/unifi
      - data:/unifi/data
      - log:/unifi/log
      - cert:/unifi/cert
      - init:/unifi/init.d
      - run:/var/run/unifi
      # Mount local folder for backups and autobackups
      - ./backup:/unifi/data/backup
    user: unifi
    sysctls:
      net.ipv4.ip_unprivileged_port_start: 0
    environment:
      DB_URI: mongodb://mongo/unifi
      STATDB_URI: mongodb://mongo/unifi_stat
      DB_NAME: unifi
      TZ: Europe/Stockholm
    ports:
      - "3478:3478/udp" # STUN
      - "6789:6789/tcp" # Speed test
      - "8080:8080/tcp" # Device/ controller comm.
      - "8443:8443/tcp" # Controller GUI/API as seen in a web browser
      - "8880:8880/tcp" # HTTP portal redirection
      - "8843:8843/tcp" # HTTPS portal redirection
      - "10001:10001/udp" # AP discovery
    labels:
      - "diun.enable=true"
      - "diun.watch_repo=true"
      - "diun.exclude_tags=.*arm32v7;.*arm64v8"

  logs:
    image: bash
    # container_name: ${COMPOSE_PROJECT_NAME}_logs
    container_name: unifi_logs
    depends_on:
      - controller
    command: bash -c 'tail -F /unifi/log/*.log'
    restart: always
    volumes:
      - log:/unifi/log

volumes:
  db:
  dbcfg:
  data:
  log:
  cert:
  init:
  dir:
  run:

networks:
  unifi:

I also notice some log errors relating to the scheduled backups. Not sure if it's related.

[2022-01-20T14:00:01,290] <schedule-backup> ERROR system - zipDir error
java.io.FileNotFoundException: /usr/lib/unifi/data/backup/autobackup/autobackup_6.5.55_20220120_1300_1642683600011.unf (No such file or directory)
    at java.io.FileOutputStream.open0(Native Method) ~[?:1.8.0_312]
    at java.io.FileOutputStream.open(FileOutputStream.java:270) ~[?:1.8.0_312]
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213) ~[?:1.8.0_312]
    at java.io.FileOutputStream.<init>(FileOutputStream.java:101) ~[?:1.8.0_312]
    at com.ubnt.ace.B.o00000(Unknown Source) [ace.jar:?]
    at com.ubnt.service.system.returnObject.o00000(Unknown Source) [ace.jar:?]
    at com.ubnt.service.system.backup.for.o00000(Unknown Source) [ace.jar:?]
    at com.ubnt.service.system.backup.for.o00000(Unknown Source) [ace.jar:?]
    at com.ubnt.service.system.backup.for.o00000(Unknown Source) [ace.jar:?]
    at com.ubnt.service.system.backup.oO0O.o00000(Unknown Source) [ace.jar:?]
    at com.ubnt.service.system.backup.oO0O.o00000(Unknown Source) [ace.jar:?]
    at com.ubnt.service.system.backup.oO0O$_OOo.super(Unknown Source) [ace.jar:?]
    at com.ubnt.service.schedule.L$_Oo.execute(Unknown Source) [ace.jar:?]
    at it.sauronsoftware.cron4j.TaskExecutor$Runner.run(Unknown Source) [cron4j-2.2.5.jar:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
[2022-01-20T14:00:01,291] <schedule-backup> ERROR schedule - Failed to execute[backup]: /usr/lib/unifi/data/backup/autobackup/autobackup_6.5.55_20220120_1300_1642683600011.unf does not exist
java.lang.IllegalArgumentException: /usr/lib/unifi/data/backup/autobackup/autobackup_6.5.55_20220120_1300_1642683600011.unf does not exist
    at org.apache.commons.io.FileUtils.sizeOf(FileUtils.java:2541) ~[commons-io-2.6.jar:2.6]
    at com.ubnt.service.system.returnObject.o00000(Unknown Source) ~[ace.jar:?]
    at com.ubnt.service.system.backup.for.o00000(Unknown Source) ~[ace.jar:?]
    at com.ubnt.service.system.backup.for.o00000(Unknown Source) ~[ace.jar:?]
    at com.ubnt.service.system.backup.for.o00000(Unknown Source) ~[ace.jar:?]
    at com.ubnt.service.system.backup.oO0O.o00000(Unknown Source) ~[ace.jar:?]
    at com.ubnt.service.system.backup.oO0O.o00000(Unknown Source) ~[ace.jar:?]
    at com.ubnt.service.system.backup.oO0O$_OOo.super(Unknown Source) ~[ace.jar:?]
    at com.ubnt.service.schedule.L$_Oo.execute(Unknown Source) [ace.jar:?]
    at it.sauronsoftware.cron4j.TaskExecutor$Runner.run(Unknown Source) [cron4j-2.2.5.jar:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]

[mountaindude]

@jacobalberty Do you want to track these issues in separate posts? I.e. the inability to download backups and the auto-backup errors in the logs?

I checked and it looks like the auto backup errors in the log are related to the fact that /usr/lib/unifi/data/backup/autobackup/ does not exist.

unifi@64602e3bbf81:/unifi$ ls -la /usr/lib/unifi/data/backup/
total 8
drwxr-xr-x 2 root  root  4096 Jan 20 13:14 .
drwxr-xr-x 4 unifi unifi 4096 Jan 20 13:42 ..
unifi@64602e3bbf81:/unifi$

Digging a bit deeper, permissions on that backup directory looks funky. Not sure what they should be though...unifi maybe?

unifi@64602e3bbf81:/usr/lib/unifi/data$ pwd
/usr/lib/unifi/data
unifi@64602e3bbf81:/usr/lib/unifi/data$ ls -la
total 52
drwxr-xr-x 4 unifi unifi  4096 Jan 20 13:42 .
drwxr-xr-x 6 unifi unifi  4096 Jan 20 13:14 ..
drwxr-xr-x 2 root  root   4096 Jan 20 13:14 backup
drwxr-xr-x 2 unifi unifi  4096 Jan 20 13:14 db
-rw-r--r-- 1 unifi unifi 18344 Jan 20 13:43 firmware.json
-rw-r--r-- 1 unifi unifi  2248 Jan 20 13:14 keystore
-rw-r--r-- 1 unifi unifi  1424 Jan 20 13:42 model_lifecycles.json
-rw-r--r-- 1 unifi unifi  1461 Jan 20 13:42 system.properties
-rw-r--r-- 1 unifi unifi  1461 Jan 20 13:42 system.properties.bk
unifi@64602e3bbf81:/usr/lib/unifi/data$

From within the running container, these are the non-standard/Linux-out-of-the-box users available:

unifi:x:999:999::/home/unifi:/bin/sh
mongodb:x:101:102::/var/lib/mongodb:/usr/sbin/nologin

On the host we instead have an old account no longer used on that id

netdata:x:999:998:netdata:/var/lib/netdata:

...and the directory on the host where docker-compose lives looks like this. Feels more and more like a permissions issue..

root@abc1:/opt/docker/container/unifi-controller# ls -la
total 16
drwxr-xr-x  3 root root 4096 Jan 20 13:14 .
drwxr-xr-x 47 root root 4096 Nov 21 20:04 ..
drwxr-xr-x  2 root root 4096 Jan 20 13:14 backup
-rw-r--r--  1 root root 2590 Jan 20 13:10 docker-compose.yaml
root@abc1:/opt/docker/container/unifi-controller#

[jacobalberty]

In your configuration yeah it should be unifi.unifi as the owner user/group . So that whole /usr/lib/unifi/data folder should be a symlink to /unifi/data

So the problem you're running into is the same one I've been struggling to come up with a solution to in #362 / #373 I do not have a good solution as of yet. On my setup I manually chown those directories to be owned by the unifi uid/gid (default 999/999) I've investigated using a setuid binary to change permissions style solution but I haven't found one that felt right to me yet.

[mountaindude]

Ok makes sense. I think - got to admit I am not 100% comfortable with the Linux user/group model... But good enough I think.

I'll move things around on this server, freeing up 999:999 on the host etc. Then we'll if either or even both the issues go away.

[mountaindude]

Yup, that did the trick.

I can now do manual backup downloads as expected. There was an initial error in the logs, complaining about /usr/lib/unifi/data/backup/meta.json not existing:

[2022-01-20T18:31:16,860] <backup> ERROR system - Failed to load file[/usr/lib/unifi/data/backup/meta.json] as JSON java.io.FileNotFoundException: /usr/lib/unifi/data/backup/meta.json (No such file or directory)

But I guess the backup file was created anyway, because it downloaded just fine. 2nd and later attempts worked without any errors in the logs.

I've set an auto-backup to execute in ca 25 minutes, will report back what happened.

I'd be happy to do a short writeup on the steps/commands needed to solve this. In case others run into it too and want a quick'n'easy solution.

Where would that best go? In a markdown file in the examples folder? Or in the main README.md?

Maybe a reference from the readme to the more exhaustive description in the examples folder?

[jacobalberty]

Maybe a reference from the readme to the more exhaustive description in the examples folder?

That sounds good to me

[mountaindude]

The scheduled auto-backup that just ran resulted in an error in the log files:

[2022-01-20T19:00:00,075] <schedule-backup> ERROR system - Failed to load file[/usr/lib/unifi/data/backup/autobackup/autobackup_meta.json] as JSON java.io.FileNotFoundException: /usr/lib/unifi/data/backup/autobackup/autobackup_meta.json (No such file or directory)

But the backup was created anyway, so I suspect this happens just the first time when that file doesn't exist. Coming backups will probably happen without log errors. I have another one scheduled in 55 minutes, will verify then.

Will include the above in the howto/writeup too.

[jacobalberty]

This is quite likely fixed in latest. I have not merged it into the v6.5 stuff yet but it should be working in latest as of right now. and it will definitely be in v7.0.x

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[nerozero]

Hi, I'm experience same issue, The autobackups are not produced while backup can be easily download from a web interface.