Open webstudiobond opened 1 week ago
Hi, I fear I cannot reproduce the issue. So I need more information like logs, detailed reproduction steps, etc.
Also, please fill out the whole bug report template: https://raw.githubusercontent.com/nextcloud/all-in-one/main/.github/ISSUE_TEMPLATE/Bug_report.md
Ubuntu 22.04.4 LTS
sudo docker info
Client: Docker Engine - Community
Version: 27.2.0
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.16.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.29.2
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 61
Running: 58
Paused: 0
Stopped: 3
Images: 55
Server Version: 27.2.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
runc version: v1.1.13-0-g58aa920
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.0-119-generic
Operating System: Ubuntu 22.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.61GiB
Name: redacted
ID: redacted
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
services:
nextcloud-aio-mastercontainer:
image: nextcloud/all-in-one:latest
init: true
restart: always
# This line is not allowed to be changed as otherwise AIO will not work correctly
container_name: nextcloud-aio-mastercontainer
volumes:
# This line is not allowed to be changed as otherwise the built-in backup solution will not work
- nextcloud_aio_mastercontainer:/mnt/docker-aio-config
- /var/run/docker.sock:/var/run/docker.sock:ro # May be changed on macOS, Windows or docker rootless. See the applicable documentation. If adjusting, don't forget to also set 'WATCHTOWER_DOCKER_SOCKET_PATH'!
network_mode: bridge
ports:
- 127.0.0.1:14080:8080
environment:
# Setting this to true allows to hide the backup section in the AIO interface. See https://github.com/nextcloud/all-in-one#how-to-disable-the-backup-section
AIO_DISABLE_BACKUP_SECTION: false
# Is needed when running behind a web server or reverse proxy (like Apache, Nginx, Caddy, Cloudflare Tunnel and else). See https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md
APACHE_PORT: 11000
# Should be set when running behind a web server or reverse proxy (like Apache, Nginx, Caddy, Cloudflare Tunnel and else) that is running on the same host. See https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md
APACHE_IP_BINDING: 127.0.0.1
# Allows to adjust borgs retention policy. See https://github.com/nextcloud/all-in-one#how-to-adjust-borgs-retention-policy
BORG_RETENTION_POLICY: --keep-within=7d --keep-weekly=4 --keep-monthly=6
# Setting this to true allows to disable Collabora's Seccomp feature. See https://github.com/nextcloud/all-in-one#how-to-disable-collaboras-seccomp-feature
COLLABORA_SECCOMP_DISABLED: false
# Allows to set the host directory for Nextcloud's datadir. ⚠️⚠️⚠️ Warning: do not set or adjust this value after the initial Nextcloud installation is done! See https://github.com/nextcloud/all-in-one#how-to-change-the-default-location-of-nextclouds-datadir
# NEXTCLOUD_DATADIR: /mnt/ncdata
NEXTCLOUD_DATADIR: /mnt/ncdata
# Allows the Nextcloud container to access the chosen directory on the host. See https://github.com/nextcloud/all-in-one#how-to-allow-the-nextcloud-container-to-access-directories-on-the-host
NEXTCLOUD_MOUNT: /mnt
# Can be adjusted if you need more. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-upload-limit-for-nextcloud
NEXTCLOUD_UPLOAD_LIMIT: 10G
# Can be adjusted if you need more. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-max-execution-time-for-nextcloud
NEXTCLOUD_MAX_TIME: 3600
# Can be adjusted if you need more. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-php-memory-limit-for-nextcloud
NEXTCLOUD_MEMORY_LIMIT: 512M
# CA certificates in this directory will be trusted by the OS of the nexcloud container (Useful e.g. for LDAPS) See See https://github.com/nextcloud/all-in-one#how-to-trust-user-defined-certification-authorities-ca
# NEXTCLOUD_TRUSTED_CACERTS_DIR: /path/to/my/cacerts
# Allows to modify the Nextcloud apps that are installed on starting AIO the first time. See https://github.com/nextcloud/all-in-one#how-to-change-the-nextcloud-apps-that-are-installed-on-the-first-startup
NEXTCLOUD_STARTUP_APPS: -deck twofactor_totp -tasks -calendar -contacts -notes
# This allows to add additional packages to the Nextcloud container permanently. Default is imagemagick but can be overwritten by modifying this value. See https://github.com/nextcloud/all-in-one#how-to-add-os-packages-permanently-to-the-nextcloud-container
NEXTCLOUD_ADDITIONAL_APKS: imagemagick
# This allows to add additional php extensions to the Nextcloud container permanently. Default is imagick but can be overwritten by modifying this value. See https://github.com/nextcloud/all-in-one#how-to-add-php-extensions-permanently-to-the-nextcloud-container
NEXTCLOUD_ADDITIONAL_PHP_EXTENSIONS: imagick
# This allows to enable the /dev/dri device in the Nextcloud container. ⚠️⚠️⚠️ Warning: this only works if the '/dev/dri' device is present on the host! If it should not exist on your host, don't set this to true as otherwise the Nextcloud container will fail to start! See https://github.com/nextcloud/all-in-one#how-to-enable-hardware-transcoding-for-nextcloud
# NEXTCLOUD_ENABLE_DRI_DEVICE: false
# Setting this to true will keep Nextcloud apps that are disabled in the AIO interface and not uninstall them if they should be installed. See https://github.com/nextcloud/all-in-one#how-to-keep-disabled-apps
NEXTCLOUD_KEEP_DISABLED_APPS: false
# This allows to adjust the port that the talk container is using. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-talk-port
# TALK_PORT: 3478
# Needs to be specified if the docker socket on the host is not located in the default '/var/run/docker.sock'. Otherwise mastercontainer updates will fail. For macos it needs to be '/var/run/docker.sock'
WATCHTOWER_DOCKER_SOCKET_PATH: /var/run/docker.sock
# https://github.com/nextcloud/all-in-one/tree/main/community-containers#how-to-use-this
# AIO_COMMUNITY_CONTAINERS: libretranslate vaultwarden
# Is needed when using SELinux
# security_opt: ["label:disable"]
volumes:
# If you want to store the data on a different drive, see https://github.com/nextcloud/all-in-one#how-to-store-the-filesinstallation-on-a-separate-drive
nextcloud_aio_mastercontainer:
# This line is not allowed to be changed as otherwise the built-in backup solution will not work
name: nextcloud_aio_mastercontainer
External storage (all works)
Rights, owner, group. View in container
docker exec -it nextcloud-aio-nextcloud /bin/bash
26f92dbe8f48:/var/www/html# ls -la /mnt
...
drwxr-x--- 1 www-data www-data 0 Sep 4 12:14 hetzner
drwxr-x--- 1 1038 www-data 0 Sep 4 12:07 imext
drwxr-x--- 1 1038 www-data 0 Sep 4 12:07 imlib
drwxr-x--- 6 www-data root 4096 Sep 4 12:15 ncdata
drwxr-x--- 1 www-data www-data 0 Sep 4 12:14 ncgdrive
Backup scheduled for 15:53 UTC Scheduled backup in progress After the scheduled automatic backup is complete and all services are started
[no app in context] Warning: Error while getting quota info, using root quota
PROPFIND /remote.php/dav/files/sergei/
from 10.10.1.4 by sergei at 4 сент. 2024 г., 18:57:01
25a2079aa412:/var/www/html# ls -la /mnt
...
drwxr-xr-x 2 root root 0 Sep 3 15:16 hetzner
drwxr-xr-x 2 root root 0 Sep 3 15:16 imext
drwxr-xr-x 2 root root 0 Sep 3 15:16 imlib
drwxr-x--- 6 www-data root 4096 Sep 4 18:55 ncdata
drwxr-xr-x 2 root root 0 Sep 3 15:16 ncgdrive
Now manually stop and start the container nextcloud-aio-nextcloud
docker stop nextcloud-aio-nextcloud
docker start nextcloud-aio-nextcloud
After that, everything works again
25a2079aa412:/var/www/html# ls -la /mnt
...
drwxr-x--- 1 www-data www-data 0 Sep 4 18:56 hetzner
drwxr-x--- 1 1038 www-data 0 Sep 4 18:56 imext
drwxr-x--- 1 1038 www-data 0 Sep 4 18:56 imlib
drwxr-x--- 6 www-data root 4096 Sep 4 19:10 ncdata
drwxr-x--- 1 www-data www-data 0 Sep 4 18:56 ncgdrive
Thanks for the information!
How are the drives mounted? Via /etc/fstab or via a different way?
Disks are mounted via systemd (rclone is used) This is for one of the disks, the others are similar.
cat /etc/systemd/system/mnt-ncgdrive.mount
[Unit]
Description=Mount for Nextcloud Google Drive
Wants=network-online.target
After=network-online.target
[Mount]
Type=rclone
What=gdrive:nc
Where=/mnt/ncgdrive
Options=rw,allow-other,default-permissions,config=/root/.config/rclone/rclone.conf,uid=33,gid=33,umask=027,dir-perms=750,file-perms=640,vfs-cache-mode=full,cache-dir=/var/rclone,vfs-cache-max-age=1h,vfs-cache-max-size=10G,vfs-read-chunk-size=128M,vfs-read-chunk-size-limit=1G,drive-use-trash=false
[Install]
WantedBy=multi-user.target
Can you mount them via /etc/fstab?
If that makes it work, it is most likely a systemd startup issue: for example the drives need to be mounted before docker starts.
During autobacks, disks are not unmounted in the system. They remain mounted, so they are not affected in any way. For example, immich running in docker continues to use disks.
Hm...
Can you run sudo ls -la /mnt
on the host and post the outout here?
And yes, I checked. Mounted disks manually, mounted them in /etc/fstab - the problem remains only in nextcloud and only when a scheduled backup is performed. If the backup is done manually from the AIO interface, if you stop the containers in AIO and then start them, everything works as it should.
ls -la /mnt
total 16
drwxr-xr-x 9 root root 4096 Sep 3 12:46 .
drwxr-xr-x 22 root root 4096 Aug 31 21:47 ..
drwxr-x--- 1 www-data www-data 0 Sep 4 18:56 hetzner
drwxr-x--- 1 immich www-data 0 Sep 4 18:56 imext
drwxr-x--- 1 immich www-data 0 Sep 4 18:56 imlib
drwxr-xr-x 2 root root 0 Sep 3 15:16 mega
drwxr-x--- 6 www-data root 4096 Sep 4 19:10 ncdata
drwxr-x--- 2 www-data www-data 4096 Aug 31 22:40 ncext
drwxr-x--- 1 www-data www-data 0 Sep 4 18:56 ncgdrive
I fear I cannot explain the behaviour - restarting the nextcloud container should not suddenly make it work if the drives are correctly mounted already before the container is started.
I don't understand the reasoning behind it either. Moreover, if after a scheduled backup and errors, if you execute the
docker stop nextcloud-aio-nextcloud
docker start nextcloud-aio-nextcloud
then everything works again
Could you please tell me where to see what exactly is being backed up (which folders, disks)? Maybe I can configure borgbackup in the system then
Can you please first try to add Before=docker.service
to your systemd mount service and check if that helps?
I don't understand the reasoning behind it either. Moreover, if after a scheduled backup and errors, if you execute the
docker stop nextcloud-aio-nextcloud docker start nextcloud-aio-nextcloud
then everything works again
Yes, this points towards a mounting order issue as I pointed out above...
Could you please tell me where to see what exactly is being backed up (which folders, disks)? Maybe I can configure borgbackup in the system then
You can inspect the borgbackup container via sudo docker inspect nextcloud-aio-borgbackup
. This will also list all mount points...
BTW, I just googled a bit around and found this wiki: https://github.com/rclone/rclone/wiki/Systemd-rclone-mount. Maybe it helps you to figure things out.
Before=docker.service
Checked it, it didn't work
Yeah, I saw that, too, but this is no different from running it manually rclone mount
, which I also did. And the disk remains mounted, it remains available for other services in the system and in the docker.
I followed the documentation https://rclone.org/commands/rclone_mount/
Can you maybe test if a different kind of mount works via fstab like eg a local disk as all your mounts are currently using rclone iirc?
Normal local does not disappear in nextcloud, only those mounted via rclone
drwxr-x--- 2 www-data www-data 4096 Aug 31 22:40 ncext
However, these are also regular local, but through rclone they are encrypted and do not disappear in the immich docker
drwxr-x--- 1 immich www-data 0 Sep 4 18:56 imext
drwxr-x--- 1 immich www-data 0 Sep 4 18:56 imlib
Hm... You could inspect both containers and check if there are differences in how the bind mounts are done in docker internally to debug this further
For immich I specify in docker compose to mount individual disks
- "/mnt/imlib:/usr/src/app/upload/library"
- "/mnt/imext:/mnt/media:ro"
the AIO uses NEXTCLOUD_MOUNT: /mnt
and then it's already set up in External storage
Okay, this explains things. Since local disk mount seems to work, I am still pretty sure that this is a mounting issue with docker together with your rclone systemd mount. Probably best to open a thread in a rclone and/or docker forum for further help.
As a temporary solution, I have added a stop and start container to the cron (a few minutes after the backup). checked - it works.
Hi, did you already open a thread in a rclone and/or docker forum for further help on this?
Hello, didn't create on those forums. Still I have not had any problems with other containers in docker and rclone, only AIO and only when running a backup.
Since I added a task to the cron job to stop and start the container about 10 minutes after the scheduled backup (this happens when the cloud is not in use), the cloud operation is now unaffected by this backup.
10 22 * * * docker stop nextcloud-aio-nextcloud > /dev/null 2>&1 && docker start nextcloud-aio-nextcloud > /dev/null 2>&1
I saw that the backup container is also used by fuse (and rclone mounts with fuse).
"Devices": [
{
"PathOnHost": "/dev/fuse",
"PathInContainer": "/dev/fuse",
"CgroupPermissions": "rwm"
}
]
Come to think of it, maybe it has some effect and changes the permissions in the container
Lets try the following. Can you run the following and post the output here?
sudo docker run --name test -d --volume /mnt:/mnt:rw alpine sleep inf
sudo docker exec -it test ls -l /mnt
sudo docker stop test
sudo docker rm test
sudo docker run --name test -d --volume /mnt:/mnt:rw alpine sleep inf
sudo docker exec -it test ls -l /mnt
sudo docker restart test
sudo docker exec -it test ls -l /mnt
sudo docker stop test
sudo docker rm test
Hello, of course.
docker run --name test -d --volume /mnt:/mnt:rw alpine sleep inf
ca4c386ea0499ec846e026046d1eb8d3d26a999d4c76724f38be7d48c1297641
docker exec -it test ls -l /mnt
total 8
drwxr-x--- 1 33 33 0 Sep 11 09:42 hetzner
drwxr-x--- 1 1038 33 0 Sep 11 09:42 imext
drwxr-x--- 1 1038 33 0 Sep 11 06:16 imlib
drwxr-x--- 1 33 33 0 Sep 11 09:43 mega
drwxr-x--- 7 33 root 4096 Sep 10 19:10 ncdata
drwxr-x--- 3 33 33 4096 Sep 8 11:04 ncext
drwxr-x--- 1 33 33 0 Sep 11 09:43 ncgdrive
docker stop test
test
docker rm test
test
docker run --name test -d --volume /mnt:/mnt:rw alpine sleep inf
33bc00d232d0c72eb9b5bd2930064c6e4d183b8ea60aab062225b8c97c9c68f7
docker exec -it test ls -l /mnt
total 8
drwxr-x--- 1 33 33 0 Sep 11 09:42 hetzner
drwxr-x--- 1 1038 33 0 Sep 11 09:42 imext
drwxr-x--- 1 1038 33 0 Sep 11 06:16 imlib
drwxr-x--- 1 33 33 0 Sep 11 09:43 mega
drwxr-x--- 7 33 root 4096 Sep 10 19:10 ncdata
drwxr-x--- 3 33 33 4096 Sep 8 11:04 ncext
drwxr-x--- 1 33 33 0 Sep 11 09:43 ncgdrive
docker restart test
test
docker exec -it test ls -l /mnt
total 8
drwxr-x--- 1 33 33 0 Sep 11 09:42 hetzner
drwxr-x--- 1 1038 33 0 Sep 11 09:42 imext
drwxr-x--- 1 1038 33 0 Sep 11 06:16 imlib
drwxr-x--- 1 33 33 0 Sep 11 09:43 mega
drwxr-x--- 7 33 root 4096 Sep 10 19:10 ncdata
drwxr-x--- 3 33 33 4096 Sep 8 11:04 ncext
drwxr-x--- 1 33 33 0 Sep 11 09:43 ncgdrive
docker stop test
test
docker rm test
test
All right. Then another test.
sudo docker run --name test -d --volume /mnt:/mnt:rw alpine sleep inf
sudo docker exec -it test ls -l /mnt
sudo docker stop test
# The command below should start a backup via borg
sudo docker start nextcloud-aio-borgbackup
sudo docker rm test
sudo docker run --name test -d --volume /mnt:/mnt:rw alpine sleep inf
sudo docker exec -it test ls -l /mnt
sudo docker restart test
sudo docker exec -it test ls -l /mnt
sudo docker stop test
sudo docker rm test
docker run --name test -d --volume /mnt:/mnt:rw alpine sleep inf
b9f0f58e06ed5ac0770d63cf18ad6287e872abc9803409b0dc80e2db2a08da7d
docker exec -it test ls -l /mnt
total 8
drwxr-x--- 1 33 33 0 Sep 11 09:42 hetzner
drwxr-x--- 1 1038 33 0 Sep 11 09:42 imext
drwxr-x--- 1 1038 33 0 Sep 11 06:16 imlib
drwxr-x--- 1 33 33 0 Sep 11 10:21 mega
drwxr-x--- 7 33 root 4096 Sep 10 19:10 ncdata
drwxr-x--- 3 33 33 4096 Sep 8 11:04 ncext
drwxr-x--- 1 33 33 0 Sep 11 09:43 ncgdrive
docker stop test
test
docker start nextcloud-aio-borgbackup
nextcloud-aio-borgbackup
docker rm test
test
docker run --name test -d --volume /mnt:/mnt:rw alpine sleep inf
d2a5d5f582901589d58cee699f65c3301e7224946727ca49f06f85165ad69f34
docker exec -it test ls -l /mnt
total 8
drwxr-x--- 1 33 33 0 Sep 11 09:42 hetzner
drwxr-x--- 1 1038 33 0 Sep 11 09:42 imext
drwxr-x--- 1 1038 33 0 Sep 11 06:16 imlib
drwxr-x--- 1 33 33 0 Sep 11 10:21 mega
drwxr-x--- 7 33 root 4096 Sep 10 19:10 ncdata
drwxr-x--- 3 33 33 4096 Sep 8 11:04 ncext
drwxr-x--- 1 33 33 0 Sep 11 09:43 ncgdrive
docker restart test
test
docker exec -it test ls -l /mnt
total 8
drwxr-x--- 1 33 33 0 Sep 11 09:42 hetzner
drwxr-x--- 1 1038 33 0 Sep 11 09:42 imext
drwxr-x--- 1 1038 33 0 Sep 11 06:16 imlib
drwxr-x--- 1 33 33 0 Sep 11 10:21 mega
drwxr-x--- 7 33 root 4096 Sep 10 19:10 ncdata
drwxr-x--- 3 33 33 4096 Sep 8 11:04 ncext
drwxr-x--- 1 33 33 0 Sep 11 09:43 ncgdrive
docker stop test && docker rm test
test
test
Ah wait, you need to wait for the borgbackup container to stop before running the additonal commands. So simply add a sleep after starting the borgbackup container
How long do we have to wait? I can run it in the portainer, wait for it to stop and then run additional commands. How do you like this option?
docker run --name test -d --volume /mnt:/mnt:rw alpine sleep inf
7987987d044cb985b018b7bcba382b31eaead87c14e21fb2a520dbed274c301c
docker exec -it test ls -l /mnt
total 8
drwxr-x--- 1 33 33 0 Sep 11 11:43 hetzner
drwxr-x--- 1 1038 33 0 Sep 11 11:43 imext
drwxr-x--- 1 1038 33 0 Sep 11 11:36 imlib
drwxr-x--- 1 33 33 0 Sep 11 11:43 mega
drwxr-x--- 7 33 root 4096 Sep 10 19:10 ncdata
drwxr-x--- 3 33 33 4096 Sep 8 11:04 ncext
drwxr-x--- 1 33 33 0 Sep 11 11:43 ncgdrive
docker stop test
test
docker rm test
test
docker run --name test -d --volume /mnt:/mnt:rw alpine sleep inf
9d32aa0e63a699bbfe086a71a1a2065c6e9e1b0e4630cd649755a6bc3e8b096d
docker exec -it test ls -l /mnt
total 8
drwxr-x--- 1 33 33 0 Sep 11 11:43 hetzner
drwxr-x--- 1 1038 33 0 Sep 11 11:43 imext
drwxr-x--- 1 1038 33 0 Sep 11 11:36 imlib
drwxr-x--- 1 33 33 0 Sep 11 11:43 mega
drwxr-x--- 7 33 root 4096 Sep 10 19:10 ncdata
drwxr-x--- 3 33 33 4096 Sep 8 11:04 ncext
drwxr-x--- 1 33 33 0 Sep 11 11:43 ncgdrive
docker restart test
test
docker exec -it test ls -l /mnt
total 8
drwxr-x--- 1 33 33 0 Sep 11 11:43 hetzner
drwxr-x--- 1 1038 33 0 Sep 11 11:43 imext
drwxr-x--- 1 1038 33 0 Sep 11 11:36 imlib
drwxr-x--- 1 33 33 0 Sep 11 11:43 mega
drwxr-x--- 7 33 root 4096 Sep 10 19:10 ncdata
drwxr-x--- 3 33 33 4096 Sep 8 11:04 ncext
drwxr-x--- 1 33 33 0 Sep 11 11:43 ncgdrive
docker stop test && docker rm test
test
test
I fear I am for now out ideas how to debug this further and especially what could cause this.
Normal local does not disappear in nextcloud,
This still points towards the rclone mount not behaving correctly. So I still think it makes sense to open a thread there.
It seems to me that after a standard auto backup, nextcloud does not correctly determine the rights to folders and files on disks mounted using rclone.
But if after this you stop and start the nextcloud container, then the rights are determined correctly, and such disks work in nextcloud, as in other docker containers. During the time I added a task to stop and start the container to cron, there were no problems with disks. This is, of course, a "crutch", but it works.
About opening a report in rclone. Since such issues do not arise with several other containers, but only in nextcloud aio and then with auto backup, I just don’t know what to describe there, because the issue is specific only to nextcloud.
There may be a problem with the application External storage?
Steps to reproduce
Expected behavior
The disks should display and work normally in nextcloud
Actual behavior
Disk connection error in nextcloud
Other information
If you stop all containers and start them from the AIO interface, the disks show up normally again and everything works in nextcloud. If you manually stop the nextcloud-aio-nextcloud container, everything starts working correctly too. If you manually start backup creation from the AIO interface, everything works fine. The problem is only with scheduled backup.