search

linuxserver / docker-unifi-network-application

GNU General Public License v3.0
617 stars 46 forks source link

[BUG] automatic backup failed. #96

Open janusn opened 1 month ago

janusn commented 1 month ago

Is there an existing issue for this?

Current Behavior

I have enabled automatic daily config backup on the system pane.

The automatic backup never happens and I find the following errors on the /usr/lib/unifi/logs/server.log every day.

[2024-07-09 04:35:00,055] <schedule-backup> ERROR system - zipDir error
java.io.FileNotFoundException: /usr/lib/unifi/data/backup/autobackup/autobackup_8.2.93_20240709_0435_1720499700006.unf (No such file or directory)
    at java.base/java.io.FileOutputStream.open0(Native Method)
    at java.base/java.io.FileOutputStream.open(FileOutputStream.java:293)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:235)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:123)
    at com.ubnt.ace.Object.ô00000(Unknown Source)
    at com.ubnt.service.system.o0Oo.private(Unknown Source)
    at com.ubnt.service.system.backup.K.õ00000(Unknown Source)
    at com.ubnt.service.system.backup.K.ö00000(Unknown Source)
    at com.ubnt.service.system.backup.K.Ó00000(Unknown Source)
    at com.ubnt.service.system.backup.OO0OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO.Õoô000(Unknown Source)
    at com.ubnt.service.system.backup.OO0OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO.öOô000(Unknown Source)
    at com.ubnt.service.system.backup.OO0OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO$_o0.Ò00000(Unknown Source)
    at com.ubnt.service.schedule.M$_Oo.execute(Unknown Source)
    at it.sauronsoftware.cron4j.TaskExecutor$Runner.run(Unknown Source)
    at java.base/java.lang.Thread.run(Thread.java:840)
[2024-07-09 04:35:00,155] <schedule-backup> ERROR schedule - Failed to execute[backup]: File system element for parameter 'path' does not exist: '/usr/lib/unifi/data/backup/autobackup/autobackup_8.2.93_20240709_0435_1720499700006.unf'
java.lang.IllegalArgumentException: File system element for parameter 'path' does not exist: '/usr/lib/unifi/data/backup/autobackup/autobackup_8.2.93_20240709_0435_1720499700006.unf'
    at org.apache.commons.io.file.PathUtils.requireExists(PathUtils.java:1467)
    at org.apache.commons.io.file.PathUtils.sizeOf(PathUtils.java:1634)
    at org.apache.commons.io.FileUtils.lambda$sizeOf$13(FileUtils.java:2861)
    at org.apache.commons.io.function.Uncheck.get(Uncheck.java:197)
    at org.apache.commons.io.FileUtils.sizeOf(FileUtils.java:2861)
    at com.ubnt.service.system.o0Oo.private(Unknown Source)
    at com.ubnt.service.system.backup.K.õ00000(Unknown Source)
    at com.ubnt.service.system.backup.K.ö00000(Unknown Source)
    at com.ubnt.service.system.backup.K.Ó00000(Unknown Source)
    at com.ubnt.service.system.backup.OO0OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO.Õoô000(Unknown Source)
    at com.ubnt.service.system.backup.OO0OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO.öOô000(Unknown Source)
    at com.ubnt.service.system.backup.OO0OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO$_o0.Ò00000(Unknown Source)
    at com.ubnt.service.schedule.M$_Oo.execute(Unknown Source)
    at it.sauronsoftware.cron4j.TaskExecutor$Runner.run(Unknown Source)
    at java.base/java.lang.Thread.run(Thread.java:840)

Expected Behavior

Automatic backup saved daily.

Steps To Reproduce

  1. Install a container from this image.
  2. start the container and connect to the webUI.
  3. configure the settings or restore from a backup file from another instance.
  4. navigate to the Settings->System->Backup pane
  5. enabled automatic backup and schedule to the earliest time slot.
  6. wait for automatic backup runs.

Environment

- OS:Ubuntu 24.04.0
- How docker service was installed: following the steps on this webpage: https://docs.docker.com/engine/install/debian/

CPU architecture

x86-64

Docker creation

---
services:
  unifi-network-application:
    image: lscr.io/linuxserver/unifi-network-application:latest
    container_name: unifi-network-application
    deploy:
      resources:
        limits:
          memory: 2G
    environment:
      - PUID=1018
      - PGID=1018
      - TZ=Etc/UTC
# Only evaluated on first run.
# commented out for security.
#      - MONGO_USER=unifi
      - MONGO_HOST=mongo
      - MONGO_PORT=27017
      - MONGO_DBNAME=unifidb
      - MEM_LIMIT=1024 #optional
      - MEM_STARTUP=1024 #optional
#      - MONGO_TLS= #optional
#      - MONGO_AUTHSOURCE= unifidb

# Only evaluated on first run.
# commented out for security.
#     env_file:
#       ./secrets.env
    volumes:
      - ./config:/config
    ports:
      - 8443:8443
      - 3478:3478/udp
      - 10001:10001/udp
      - 8080:8080
      - 1900:1900/udp #optional
      - 8843:8843 #optional
      - 8880:8880 #optional
      - 6789:6789 #optional
      - 5514:5514/udp #optional
    restart: unless-stopped
  mongo:
    image: "mongo:7.0.11"
    container_name: unifi-mongo
    restart: unless-stopped
    user: 1018:1018
    volumes:
      - "./config/db:/data/db:rw"
      - "./config/configdb:/data/configdb:rw"

Container logs

[migrations] started
[migrations] no migrations found
usermod: no changes
───────────────────────────────────────

      ██╗     ███████╗██╗ ██████╗
      ██║     ██╔════╝██║██╔═══██╗
      ██║     ███████╗██║██║   ██║
      ██║     ╚════██║██║██║   ██║
      ███████╗███████║██║╚██████╔╝
      ╚══════╝╚══════╝╚═╝ ╚═════╝

   Brought to you by linuxserver.io
───────────────────────────────────────

To support LSIO projects visit:
https://www.linuxserver.io/donate/

───────────────────────────────────────
GID/UID
───────────────────────────────────────

User UID:    1018
User GID:    1018
───────────────────────────────────────

[custom-init] No custom files found, skipping...
[ls.io-init] done.
github-actions[bot] commented 1 month ago

Thanks for opening your first issue here! Be sure to follow the relevant issue templates, or risk having this issue marked as invalid.

j0nnymoe commented 1 month ago

Did you restore a backup when initially setting this up?

Also, not sure why you have your backup volume there as it's not part of our compose but you've not declared it as a folder within the container.

janusn commented 1 month ago

Did you restore a backup when initially setting this up?

Yes, it was migrated from linuxserver/unifi-controller and the configs were restored from that container. Also, not sure why you have your backup volume there as it's not part of our compose but you've not declared it as a folder within the container.

/usr/lib/unifi/data is a symbolic link to /config/data and /usr/lib/unifi/logs is a symbolic link to /config/logs

I didn't create these links nor the folders. The only container mount I created is from the line under volumes in compose.yaml file, i.e. - ./config:/config (Sorry that I pasted an extra line under it by mistake.)

Could you tell me where the supported backup location is?

j0nnymoe commented 1 month ago

You've not defined the volume correctly.

Thought let's start with what your full path for your /config mount is?

Users that have had this issue in the past is due to permissions or running on remote file systems.

janusn commented 1 month ago

You've not defined the volume correctly.

Thought let's start with what your full path for your /config mount is?

As shown on the compose.yaml, it is on ./config. I can see there are files and folders created by UNA as shown below:

$ tree /docker/unifi/unifi-network-application/config/
/docker/unifi/unifi-network-application/config/
├── data
│   ├── backup
│   │   ├── 8.0.24.unf
│   │   ├── 8.2.93_default.unf
│   │   ├── 8.2.93.unf
│   │   └── meta.json
│   ├── db
│   │   ├── previous_version
│   │   └── version
│   ├── firmware.json
│   ├── keystore
│   ├── model_lifecycles.json
│   ├── system.properties
│   ├── system.properties.bk
│   └── uidb.json
└── logs
├── access.log
├── access.log.1
├── hotspot.log
├── inform_request.log
├── migration.log
├── remote
├── server.log
├── startup.log
├── state.log
├── tasks.log
└── tasks.log.1

Users that have had this issue in the past is due to permissions or running on remote file systems.

Might be.

Update on the situation:

I have created the directory /usr/lib/unifi/data/backup/autobackup inside the container and chown to abc. An automatic backup was finally appered at my local time 00:00. It seems the Reschedule settings on Backup pane has no effect at all. It only backup at midnight.

aptalca commented 1 month ago

As shown on the compose.yaml, it is on ./config

./config is a relative path, not absolute as . refers to the current working directory. It's prudent to use absolute paths in compose yaml and that's what we are asking for.

Based on your tree output, one would assume the ./config is referring to /docker/unifi/unifi-network-application/config/ as absolute path. However that assumption seems incorrect because your compose also maps the same relative path ./config for mongodb (well, subfolders of it). However, the absolute path on your host /docker/unifi/unifi-network-application/config/ does not show the mongodb subfolders inside of it according to the tree output.

It looks like you mapped incorrect folders and as a result you are looking in the wrong place.

You should replace your mount paths with absolute paths (and ideally different paths for the two containers) so you (and we) know exactly what folders you're mapping.

I have created the directory /usr/lib/unifi/data/backup/autobackup inside the container and chown to abc.

You should not do that.

The path /usr/lib/unifi/data inside the container is a symlink to the path /config/data, which is supposed to be your bind mount. Therefore, the path /usr/lib/unifi/data/backup/autobackup really is /config/data/backup/autobackup.

# docker exec unifi-network-application ls -al /usr/lib/unifi/data
lrwxrwxrwx 1 root root 12 Jul  3 16:28 /usr/lib/unifi/data -> /config/data
# docker exec unifi-network-application ls -al /config/data/backup/autobackup
total 272928
drwxr-xr-x 2 abc users     4096 Jul  6 20:00 .
drwxr-xr-x 3 abc users       60 Jun  5 08:15 ..
-rw-r--r-- 1 abc users 20627680 Mar 30 20:00 autobackup_8.1.113_20240331_0000_1711843200001.unf
-rw-r--r-- 1 abc users 20008640 Apr  6 20:00 autobackup_8.1.113_20240407_0000_1712448000001.unf
-rw-r--r-- 1 abc users 19557440 Apr 13 20:00 autobackup_8.1.113_20240414_0000_1713052800002.unf
-rw-r--r-- 1 abc users 19069184 Apr 20 20:00 autobackup_8.1.113_20240421_0000_1713657600002.unf
-rw-r--r-- 1 abc users 18769648 Apr 27 20:00 autobackup_8.1.113_20240428_0000_1714262400002.unf
-rw-r--r-- 1 abc users 18645392 May  4 20:00 autobackup_8.1.127_20240505_0000_1714867200001.unf
-rw-r--r-- 1 abc users 18864816 May 11 20:00 autobackup_8.1.127_20240512_0000_1715472000002.unf
-rw-r--r-- 1 abc users 18313440 May 18 20:00 autobackup_8.1.127_20240519_0000_1716076800001.unf
-rw-r--r-- 1 abc users 18240176 May 25 20:00 autobackup_8.1.127_20240526_0000_1716681600011.unf
-rw-r--r-- 1 abc users 18148784 Jun  1 20:00 autobackup_8.1.127_20240602_0000_1717286400002.unf
-rw-r--r-- 1 abc users 17905728 Jun  8 20:00 autobackup_8.2.93_20240609_0000_1717891200001.unf
-rw-r--r-- 1 abc users 17711392 Jun 15 20:00 autobackup_8.2.93_20240616_0000_1718496000001.unf
-rw-r--r-- 1 abc users 17754640 Jun 22 20:00 autobackup_8.2.93_20240623_0000_1719100800001.unf
-rw-r--r-- 1 abc users 17820896 Jun 29 20:00 autobackup_8.2.93_20240630_0000_1719705600002.unf
-rw-r--r-- 1 abc users 17999168 Jul  6 20:00 autobackup_8.2.93_20240707_0000_1720310400001.unf
-rw-r--r-- 1 abc users     3526 Jul  6 20:00 autobackup_meta.json

My suggestions are:

  1. Make sure you pull/repull the latest image so it contains that correct symlink
  2. Recreate it with correct absolute paths
janusn commented 1 month ago

It looks like you mapped incorrect folders and as a result you are looking in the wrong place.

Sorry for the confusion. The actual compose.yaml was a merge of 3 compose.yaml files located in 3 folders. I merged them manually here for simplicity. To my oversight, I should have fixed the relative paths of the two volumes before posting. My fault.

Here is the root compose.yaml:

---
services:
  unifi-network-application:
    extends:
      file: ./unifi-network-application/unifi-network-application.yaml
      service: unifi-network-application
    depends_on:
      mongo:
        condition: service_started
        restart: false
  mongo:
    extends:
      file: ./mongo/mongo.yaml
      service: mongo

The two - "./config:/config" lines were in fact referring 2 separated folders on host. UNA was fully functional besides auto-backups.

Using relative paths was for the purpose of an easy migration between machines/virtual hosts. With the same user and group setup, the whole stack can be migrated with a single rsync command. Absolute paths require a lot of edit afterwards.

You should replace your mount paths with absolute paths (and ideally different paths for the two containers) so you (and we) know exactly what folders you're mapping.

I have created the directory /usr/lib/unifi/data/backup/autobackup inside the container and chown to abc.

You should not do that.

The path /usr/lib/unifi/data inside the container is a symlink to the path /config/data, which is supposed to be your bind mount. Therefore, the path /usr/lib/unifi/data/backup/autobackup really is /config/data/backup/autobackup.

yes, i understand they are symlinks to the mounted folders on the host. And the symlinks are intact on my container. It just didn't create the autobackup folder for an unknown reason.

# docker exec unifi-network-application ls -al /usr/lib/unifi/data
lrwxrwxrwx 1 root root 12 Jul  3 16:28 /usr/lib/unifi/data -> /config/data
# docker exec unifi-network-application ls -al /config/data/backup/autobackup
total 272928
drwxr-xr-x 2 abc users     4096 Jul  6 20:00 .
drwxr-xr-x 3 abc users       60 Jun  5 08:15 ..
-rw-r--r-- 1 abc users 20627680 Mar 30 20:00 autobackup_8.1.113_20240331_0000_1711843200001.unf
-rw-r--r-- 1 abc users 20008640 Apr  6 20:00 autobackup_8.1.113_20240407_0000_1712448000001.unf
-rw-r--r-- 1 abc users 19557440 Apr 13 20:00 autobackup_8.1.113_20240414_0000_1713052800002.unf
-rw-r--r-- 1 abc users 19069184 Apr 20 20:00 autobackup_8.1.113_20240421_0000_1713657600002.unf
-rw-r--r-- 1 abc users 18769648 Apr 27 20:00 autobackup_8.1.113_20240428_0000_1714262400002.unf
-rw-r--r-- 1 abc users 18645392 May  4 20:00 autobackup_8.1.127_20240505_0000_1714867200001.unf
-rw-r--r-- 1 abc users 18864816 May 11 20:00 autobackup_8.1.127_20240512_0000_1715472000002.unf
-rw-r--r-- 1 abc users 18313440 May 18 20:00 autobackup_8.1.127_20240519_0000_1716076800001.unf
-rw-r--r-- 1 abc users 18240176 May 25 20:00 autobackup_8.1.127_20240526_0000_1716681600011.unf
-rw-r--r-- 1 abc users 18148784 Jun  1 20:00 autobackup_8.1.127_20240602_0000_1717286400002.unf
-rw-r--r-- 1 abc users 17905728 Jun  8 20:00 autobackup_8.2.93_20240609_0000_1717891200001.unf
-rw-r--r-- 1 abc users 17711392 Jun 15 20:00 autobackup_8.2.93_20240616_0000_1718496000001.unf
-rw-r--r-- 1 abc users 17754640 Jun 22 20:00 autobackup_8.2.93_20240623_0000_1719100800001.unf
-rw-r--r-- 1 abc users 17820896 Jun 29 20:00 autobackup_8.2.93_20240630_0000_1719705600002.unf
-rw-r--r-- 1 abc users 17999168 Jul  6 20:00 autobackup_8.2.93_20240707_0000_1720310400001.unf
-rw-r--r-- 1 abc users     3526 Jul  6 20:00 autobackup_meta.json

My suggestions are:

  1. Make sure you pull/repull the latest image so it contains that correct symlink
  2. Recreate it with correct absolute paths

Thanks a lot for your help. Let me think about how the whole migration strategy again.

ehannes commented 1 month ago

Hi!

I had the same problem and I solved it as suggested above by manually creating the autobackup directory.

@janusn about running the backup at specific times:

... An automatic backup was finally appered at my local time 00:00. It seems the Reschedule settings on Backup pane has no effect at all. It only backup at midnight.

I had my backup to run now at 23:00, so for me the reschedule settings works. Though I get another error in the log that might be related:

<webapi-8> WARN  sanitize - Invalid key exists in Setting payload, key=data_retention_time_enabled

I have my backup setting set to Backup Retention: 90 days. I'm not sure if the backup I got now actually includes 90 days of data or not.

LinuxServer-CI commented 3 days ago

This issue has been automatically marked as stale because it has not had recent activity. This might be due to missing feedback from OP. It will be closed if no further activity occurs. Thank you for your contributions.