search

nextcloud / all-in-one

📦 The official Nextcloud installation method. Provides easy deployment and maintenance with most features included in this one Nextcloud instance.
https://hub.docker.com/r/nextcloud/all-in-one
GNU Affero General Public License v3.0
5.3k stars 616 forks source link

Borg Backup will not run due to database container not correctly shut down #5197

Closed MooRogue closed 1 month ago

MooRogue commented 1 month ago

Steps to reproduce

  1. Run borg backup for Nextcloud AIO (whether automatically or manually)

Expected behavior

Borg backup runs

Actual behavior

Borg backup fails, gives following error message

2024-09-04T02:57:40.616290194Z Cannot create a backup now.
2024-09-04T02:57:40.616345414Z Reason is that the database export failed the last time.
2024-09-04T02:57:40.616348993Z Most likely was the database container not correctly shut down via the AIO interface.
2024-09-04T02:57:40.616350673Z 
2024-09-04T02:57:40.616354473Z You might want to try the database export again manually by running the three commands:
2024-09-04T02:57:40.616397062Z sudo docker start nextcloud-aio-database
2024-09-04T02:57:40.616408066Z sleep 10
2024-09-04T02:57:40.616410554Z sudo docker stop nextcloud-aio-database -t 1800
2024-09-04T02:57:40.616412104Z 
2024-09-04T02:57:40.616430484Z Afterwards try to create a backup again and it should hopefully work.
2024-09-04T02:57:40.616478677Z If it should still fail, feel free to report this to https://github.com/nextcloud/all-in-one/issues and post the database container logs and the borgbackup container logs into the thread. Thanks!

--

Other information

Host OS

Ubuntu 24.041

Output of sudo docker info

Client:
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.24.6+ds1-0ubuntu2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 33
  Running: 28
  Paused: 0
  Stopped: 5
 Images: 73
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version:
 runc version:
 init version:
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-41-generic
 Operating System: Ubuntu 24.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 20
 Total Memory: 31.01GiB
 Name: vivy
 ID: f21501cb-9d03-45cc-8d3e-5df092fb85d8
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Docker run command or docker-compose file that you used

Docker-compose file

services:
  nextcloud-aio-mastercontainer:
    image: nextcloud/all-in-one:latest
    init: true
    restart: always
    container_name: nextcloud-aio-mastercontainer # This line is not allowed to be changed as otherwise AIO will not work correctly
    volumes:
      - nextcloud_aio_mastercontainer:/mnt/docker-aio-config # This line is not allowed to be changed as otherwise the built-in backup solution will not work
      - /var/run/docker.sock:/var/run/docker.sock:ro # May be changed on macOS, Windows or docker rootless. See the applicable documentation. If adjusting, don't forget to also set 'WATCHTOWER_DOCKER_SOCKET_PATH'!
    ports:
      - 81:80 # Can be removed when running behind a web server or reverse proxy (like Apache, Nginx, Cloudflare Tunnel and else). See https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md
      - 8080:8080
      - 8443:8443 # Can be removed when running behind a web server or reverse proxy (like Apache, Nginx, Cloudflare Tunnel and else). See https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md
    environment: # Is needed when using any of the options below
      # - AIO_DISABLE_BACKUP_SECTION=false # Setting this to true allows to hide the backup section in the AIO interface. See https://github.com/nextcloud/all-in-one#how-to-disable-the-backup-section
      - APACHE_PORT=11000 # Is needed when running behind a web server or reverse proxy (like Apache, Nginx, Cloudflare Tunnel and else). See https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md
      - APACHE_IP_BINDING=0.0.0.0 # Should be set when running behind a web server or reverse proxy (like Apache, Nginx, Cloudflare Tunnel and else) that is running on the same host. See https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md
      # - BORG_RETENTION_POLICY=--keep-within=7d --keep-weekly=4 --keep-monthly=6 # Allows to adjust borgs retention policy. See https://github.com/nextcloud/all-in-one#how-to-adjust-borgs-retention-policy
      # - COLLABORA_SECCOMP_DISABLED=false # Setting this to true allows to disable Collabora's Seccomp feature. See https://github.com/nextcloud/all-in-one#how-to-disable-collaboras-seccomp-feature
      - NEXTCLOUD_DATADIR=/srv/nextcloud_data # Allows to set the host directory for Nextcloud's datadir. ⚠️⚠️⚠️ Warning: do not set or adjust this value after the initial Nextcloud installation is done! See https://github.com/nextcloud/all-in-one#how-to-change-the-default-location-of-nextclouds-datadir
      - NEXTCLOUD_MOUNT=/mnt/ # Allows the Nextcloud container to access the chosen directory on the host. See https://github.com/nextcloud/all-in-one#how-to-allow-the-nextcloud-container-to-access-directories-on-the-host
      # - NEXTCLOUD_UPLOAD_LIMIT=10G # Can be adjusted if you need more. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-upload-limit-for-nextcloud
      # - NEXTCLOUD_MAX_TIME=3600 # Can be adjusted if you need more. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-max-execution-time-for-nextcloud
      # - NEXTCLOUD_MEMORY_LIMIT=512M # Can be adjusted if you need more. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-php-memory-limit-for-nextcloud
      # - NEXTCLOUD_TRUSTED_CACERTS_DIR=/path/to/my/cacerts # CA certificates in this directory will be trusted by the OS of the nexcloud container (Useful e.g. for LDAPS) See See https://github.com/nextcloud/all-in-one#how-to-trust-user-defined-certification-authorities-ca
      # - NEXTCLOUD_STARTUP_APPS=deck twofactor_totp tasks calendar contacts notes # Allows to modify the Nextcloud apps that are installed on starting AIO the first time. See https://github.com/nextcloud/all-in-one#how-to-change-the-nextcloud-apps-that-are-installed-on-the-first-startup
      # - NEXTCLOUD_ADDITIONAL_APKS=imagemagick # This allows to add additional packages to the Nextcloud container permanently. Default is imagemagick but can be overwritten by modifying this value. See https://github.com/nextcloud/all-in-one#how-to-add-os-packages-permanently-to-the-nextcloud-container
      # - NEXTCLOUD_ADDITIONAL_PHP_EXTENSIONS=imagick # This allows to add additional php extensions to the Nextcloud container permanently. Default is imagick but can be overwritten by modifying this value. See https://github.com/nextcloud/all-in-one#how-to-add-php-extensions-permanently-to-the-nextcloud-container
      # - NEXTCLOUD_ENABLE_DRI_DEVICE=true # This allows to enable the /dev/dri device in the Nextcloud container. ⚠️⚠️⚠️ Warning: this only works if the '/dev/dri' device is present on the host! If it should not exist on your host, don't set this to true as otherwise the Nextcloud container will fail to start! See https://github.com/nextcloud/all-in-one#how-to-enable-hardware-transcoding-for-nextcloud
      # - NEXTCLOUD_KEEP_DISABLED_APPS=false # Setting this to true will keep Nextcloud apps that are disabled in the AIO interface and not uninstall them if they should be installed. See https://github.com/nextcloud/all-in-one#how-to-keep-disabled-apps
      # - TALK_PORT=3478 # This allows to adjust the port that the talk container is using. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-talk-port
      # - WATCHTOWER_DOCKER_SOCKET_PATH=/var/run/docker.sock # Needs to be specified if the docker socket on the host is not located in the default '/var/run/docker.sock'. Otherwise mastercontainer updates will fail. For macos it needs to be '/var/run/docker.sock'
    # networks: # Is needed when you want to create the nextcloud-aio network with ipv6-support using this file, see the network config at the bottom of the file
      # - nextcloud-aio # Is needed when you want to create the nextcloud-aio network with ipv6-support using this file, see the network config at the bottom of the file

  # # Optional: Caddy reverse proxy. See https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md
  # # You can find further examples here: https://github.com/nextcloud/all-in-one/discussions/588
  # caddy:
  #   image: caddy:alpine
  #   restart: always
  #   container_name: caddy
  #   volumes:
  #     - ./Caddyfile:/etc/caddy/Caddyfile
  #     - ./certs:/certs
  #     - ./config:/config
  #     - ./data:/data
  #     - ./sites:/srv
  #   network_mode: "host"

volumes:
  nextcloud_aio_mastercontainer:
    name: nextcloud_aio_mastercontainer # This line is not allowed to be changed as otherwise the built-in backup solution will not work

# # Optional: If you need ipv6, follow step 1 and 2 of https://github.com/nextcloud/all-in-one/blob/main/docker-ipv6-support.md first and then uncomment the below config in order to activate ipv6 for the internal nextcloud-aio network.
# # Please make sure to uncomment also the networking lines of the mastercontainer above in order to actually create the network with docker-compose
# networks:
#   nextcloud-aio:
#     name: nextcloud-aio # This line is not allowed to be changed as otherwise the created network will not be used by the other containers of AIO
#     driver: bridge
#     enable_ipv6: true
#     ipam:
#       driver: default
#       config:
#         - subnet: fd12:3456:789a:2::/64 # IPv6 subnet to use

Other valuable info

Log file from database container:

2024-09-04T03:31:09.653141872Z Setting postgres values...
2024-09-04T03:31:09.663904839Z chmod: /var/run/postgresql: Operation not permitted
2024-09-04T03:31:09.665314255Z 
2024-09-04T03:31:09.665325112Z PostgreSQL Database directory appears to contain a database; Skipping initialization
2024-09-04T03:31:09.665327344Z 
2024-09-04T03:31:09.768054524Z 2024-09-03 20:31:09.767 PDT [14] LOG:  starting PostgreSQL 16.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20240309) 13.2.1 20240309, 64-bit
2024-09-04T03:31:09.768091110Z 2024-09-03 20:31:09.767 PDT [14] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-09-04T03:31:09.768093823Z 2024-09-03 20:31:09.767 PDT [14] LOG:  listening on IPv6 address "::", port 5432
2024-09-04T03:31:09.768944050Z 2024-09-03 20:31:09.768 PDT [14] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-09-04T03:31:09.772363392Z 2024-09-03 20:31:09.772 PDT [24] LOG:  database system was interrupted; last known up at 2024-09-03 20:24:29 PDT
2024-09-04T03:31:09.928866895Z 2024-09-03 20:31:09.928 PDT [24] LOG:  database system was not properly shut down; automatic recovery in progress
2024-09-04T03:31:09.930325471Z 2024-09-03 20:31:09.930 PDT [24] LOG:  redo starts at 12/D25D0F60
2024-09-04T03:31:09.935932106Z 2024-09-03 20:31:09.935 PDT [24] LOG:  invalid record length at 12/D269A848: expected at least 24, got 0
2024-09-04T03:31:09.935969293Z 2024-09-03 20:31:09.935 PDT [24] LOG:  redo done at 12/D269A810 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
2024-09-04T03:31:09.948701762Z 2024-09-03 20:31:09.948 PDT [14] LOG:  database system is ready to accept connections

Other info:

  1. Running

    sudo docker start nextcloud-aio-database
sleep 10
sudo docker stop nextcloud-aio-database -t 1800

    Did not help. Database container constantly has "database system was not properly shut down;"

  2. The "redo starts at 12/D25D0F60 constantly changes on every database restart

  3. Nextcloud itself appears to be running fine despite the errors/warnings on the database container. Just can't do backups

szaimen commented 1 month ago

Hi, can you pleaes upgrade docker and its dependencies via sudo apt update && sudo apt upgrade and check if that fixes things afterwards?

MooRogue commented 1 month ago

ah, upgrading docker to Docker version 27.2.0, build 3ab4256 fixed things. It looks like the built-in Ubuntu 24.04.1 package didn't work properly. Thanks szaimen!