szaimen
commented
1 year ago 2.
/ /files/001.jpg
Does this file has the correct file permissions? cp does not retain them IIRC...
Closed zakimano closed 1 year ago
szaimen
commented
1 year ago 2.
/ /files/001.jpg
Does this file has the correct file permissions? cp does not retain them IIRC...
zakimano
commented
1 year ago Same behavior (copied files still not recognized) with cp -p, and rsync -pog.
Though admittedly, I did overlook this at first.
joshtrichards
commented
1 year ago I'm unable to reproduce this in either v25.0.6/v25.0.5, but there are some differences in our test setups.
Anything unusual about your <nextcloud-data-mount-dir> (/mnt/ncdata I presume)? i.e. What's the underlying filesystem/mount type?
zakimano
commented
1 year ago Sorry for the delay, Here's my config:
nextcloud's data dir is mounted on a mergerFS filesystem @ /srv/mfs/mfs1/nc_data
fstab:
# <file system> <dir> <type> <options> <dump> <pass>
tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0
# /dev/sda2
UUID=91ffdb16-a201-4c55-a737-d5f2b1a2e723 / ext4 rw,relatime 0 1
# /dev/sda1
UUID=6DA7-880E /boot vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro 0 2
# /dev/sdb1
UUID=7ee2682a-3c9b-471c-964c-d1362f4fb3f6 /srv/scsi/Y57 ext4 defaults,rw,relatime 0 2
# /dev/sdc1
UUID=01d66e02-7f2b-48cc-ba7d-4d26118940da /srv/scsi/HUP ext4 defaults,rw,relatime 0 2
# /dev/sdd1
UUID=0435f595-067c-4c6d-8a93-bd80b499c495 /srv/scsi/5X1 ext4 defaults,rw,relatime 0 2
# /dev/sde1
UUID=333a1a4b-8185-48c0-9cfe-a6e6104ebd71 /srv/scsi/R1K ext4 defaults,rw,relatime 0 2
# MergerFS
/srv/scsi/5X1:/srv/scsi/HUP:/srv/scsi/Y57 /srv/mfs/mfs1 fuse.mergerfs defaults,fsname=mfs1,allow_other,use_ino,category.create=mfs,moveonenospc=mfs,minfreespace=100G 0 0I set the following in my .env config:
.env:
[...]
NEXTCLOUD_DATADIR="nextcloud_aio_nextcloud_data"
[...]
VOL_NC="./nc"
VOL_NC_DATA="/srv/mfs/mfs1/nc_data"In the docker compose file, I create named bound volumes, to keep all nextcloud files organized / in an easy to find place.
docker-compose.yml:
version: "3.9"
services:
nextcloud-aio-apache:
depends_on:
- nextcloud-aio-onlyoffice
- nextcloud-aio-collabora
- nextcloud-aio-talk
- nextcloud-aio-nextcloud
image: nextcloud/aio-apache:latest
#ports:
# - ${APACHE_IP_BINDING}:${APACHE_PORT}:${APACHE_PORT}/tcp
environment:
- NC_DOMAIN=${NC_DOMAIN}
- NEXTCLOUD_HOST=nextcloud-aio-nextcloud
- COLLABORA_HOST=nextcloud-aio-collabora
- TALK_HOST=nextcloud-aio-talk
- APACHE_PORT=${APACHE_PORT}
- ONLYOFFICE_HOST=nextcloud-aio-onlyoffice
- TZ=${TIMEZONE}
- APACHE_MAX_SIZE=${APACHE_MAX_SIZE}
- APACHE_MAX_TIME=${NEXTCLOUD_MAX_TIME}
volumes:
- nextcloud_aio_nextcloud:/var/www/html:ro
- nextcloud_aio_apache:/mnt/data:rw
restart: unless-stopped
networks:
- nextcloud-aio
- nc-net
nextcloud-aio-database:
image: nextcloud/aio-postgresql:latest
expose:
- "5432"
volumes:
- nextcloud_aio_database:/var/lib/postgresql/data:rw
- nextcloud_aio_database_dump:/mnt/data:rw
environment:
- POSTGRES_PASSWORD=${DATABASE_PASSWORD}
- POSTGRES_DB=nextcloud_database
- POSTGRES_USER=nextcloud
- TZ=${TIMEZONE}
- PGTZ=${TIMEZONE}
stop_grace_period: 1800s
restart: unless-stopped
networks:
- nextcloud-aio
shm_size: 268435456
nextcloud-aio-nextcloud:
depends_on:
- nextcloud-aio-database
- nextcloud-aio-redis
image: nextcloud/aio-nextcloud:latest
expose:
- "9000"
- "7867"
volumes:
- nextcloud_aio_nextcloud:/var/www/html:rw
- ./nc-config/user.config.php:/var/www/html/config/user.config.php
- ${NEXTCLOUD_DATADIR}:/mnt/ncdata:rw
- ${NEXTCLOUD_TRUSTED_CACERTS_DIR}:/usr/local/share/ca-certificates:ro
environment:
- POSTGRES_HOST=nextcloud-aio-database
- POSTGRES_PASSWORD=${DATABASE_PASSWORD}
- POSTGRES_DB=nextcloud_database
- POSTGRES_USER=nextcloud
- REDIS_HOST=nextcloud-aio-redis
- REDIS_HOST_PASSWORD=${REDIS_PASSWORD}
- AIO_TOKEN=${AIO_TOKEN}
- NC_DOMAIN=${NC_DOMAIN}
- ADMIN_USER=admin
- ADMIN_PASSWORD=${NEXTCLOUD_PASSWORD}
- NEXTCLOUD_DATA_DIR=/mnt/ncdata
- OVERWRITEHOST=${NC_DOMAIN}
- OVERWRITEPROTOCOL=https
- TURN_SECRET=${TURN_SECRET}
- SIGNALING_SECRET=${SIGNALING_SECRET}
- ONLYOFFICE_SECRET=${ONLYOFFICE_SECRET}
- AIO_URL=${AIO_URL}
- CLAMAV_ENABLED=${CLAMAV_ENABLED}
- CLAMAV_HOST=nextcloud-aio-clamav
- ONLYOFFICE_ENABLED=${ONLYOFFICE_ENABLED}
- COLLABORA_ENABLED=${COLLABORA_ENABLED}
- COLLABORA_HOST=nextcloud-aio-collabora
- TALK_ENABLED=${TALK_ENABLED}
- ONLYOFFICE_HOST=nextcloud-aio-onlyoffice
- UPDATE_NEXTCLOUD_APPS=${UPDATE_NEXTCLOUD_APPS}
- TZ=${TIMEZONE}
- TALK_PORT=${TALK_PORT}
- IMAGINARY_ENABLED=${IMAGINARY_ENABLED}
- IMAGINARY_HOST=nextcloud-aio-imaginary
- PHP_UPLOAD_LIMIT=${NEXTCLOUD_UPLOAD_LIMIT}
- PHP_MEMORY_LIMIT=${NEXTCLOUD_MEMORY_LIMIT}
- FULLTEXTSEARCH_ENABLED=${FULLTEXTSEARCH_ENABLED}
- FULLTEXTSEARCH_HOST=nextcloud-aio-fulltextsearch
- PHP_MAX_TIME=${NEXTCLOUD_MAX_TIME}
- TRUSTED_CACERTS_DIR=${NEXTCLOUD_TRUSTED_CACERTS_DIR}
- STARTUP_APPS=${NEXTCLOUD_STARTUP_APPS}
- ADDITIONAL_APKS=${NEXTCLOUD_ADDITIONAL_APKS}
- ADDITIONAL_PHP_EXTENSIONS=${NEXTCLOUD_ADDITIONAL_PHP_EXTENSIONS}
- INSTALL_LATEST_MAJOR=${INSTALL_LATEST_MAJOR}
restart: unless-stopped
networks:
- nextcloud-aio
nextcloud-aio-redis:
image: nextcloud/aio-redis:latest
expose:
- "6379"
environment:
- REDIS_HOST_PASSWORD=${REDIS_PASSWORD}
- TZ=${TIMEZONE}
volumes:
- nextcloud_aio_redis:/data:rw
restart: unless-stopped
networks:
- nextcloud-aio
profiles: ["fulltextsearch"]
image: nextcloud/aio-fulltextsearch:latest
expose:
- "9200"
environment:
- TZ=${TIMEZONE}
- discovery.type=single-node
- ES_JAVA_OPTS=-Xms1024M -Xmx1024M
- POSTGRES_HOST=nextcloud-aio-database
volumes:
- nextcloud_aio_elasticsearch:/usr/share/elasticsearch/data:rw
restart: unless-stopped
networks:
- nextcloud-aio
volumes:
nextcloud_aio_apache:
driver: local
driver_opts:
type: "none"
o: "bind"
device: "${VOL_NC}/apache"
nextcloud_aio_database:
driver: local
driver_opts:
type: "none"
o: "bind"
device: "${VOL_NC}/database"
nextcloud_aio_database_dump:
driver: local
driver_opts:
type: "none"
o: "bind"
device: "${VOL_NC}/database_dump"
nextcloud_aio_elasticsearch:
driver: local
driver_opts:
type: "none"
o: "bind"
device: "${VOL_NC}/elasticsearch"
nextcloud_aio_nextcloud:
driver: local
driver_opts:
type: "none"
o: "bind"
device: "${VOL_NC}/nextcloud"
nextcloud_aio_redis:
driver: local
driver_opts:
type: "none"
o: "bind"
device: "${VOL_NC}/redis"
nextcloud_aio_nextcloud_data:
driver: local
driver_opts:
type: "none"
o: "bind"
device: "${VOL_NC_DATA}"
networks:
nextcloud-aio:
nc-net:
driver: bridge
Since nextcloud-aio-nextcloud container binds ${NEXTCLOUD_DATADIR}:/mnt/ncdata:rw, then at the end I declare:
volumes:
[...]
nextcloud_aio_nextcloud_data:
driver: local
driver_opts:
type: "none"
o: "bind"
device: "${VOL_NC_DATA}"which corresponds to the
VOL_NC_DATA="/srv/mfs/mfs1/nc_data"line in the .env file (which also says that ${NEXTCLOUD_DATADIR} == nextcloud_aio_nextcloud_data), I have bound /srv/mfs/mfs1/nc_data on the host to /mnt/ncdata in the container.
And (somewhat to my surprise) it works really well! Except for this small feature which I really would like to have.
I'll do my best to help you if you have any ideas on what about this setup I should test out or modify.
joshtrichards
commented
1 year ago Okay yeah I'd say that's an unusual underlying filesystem. 🤣
I think NC mostly relies on stat() to grab metadata. Since you're using a fancy UnionFS my first guess is an interaction maybe with the overlays (e.g. the stat() call is getting data from one of them but not the one that necessarily has the new data on it) or something along those lines.
Check out the TIPS as well as KNOWN ISSUES sections for MergerFS here: https://github.com/trapexit/mergerfs You can see they hint at some interactions with things like Plex/Kodi doing filesystem scans to pick up new files
Looks like MergerFS supports an option called func.getattr=newest which is one possibility that may be help you get at the root of things and/or address your issue permanently.
Given that we can't reproduce this on normal filesystems, unless something ends up pointing back at a bug or oversight on the NC side I think it's going to turn out to be something like what I just mentioned.
I'm certainly curious about the outcome - let us know!
nextcloud-command
commented
1 year ago This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.
zakimano
commented
1 year ago Good reminder from the bot, thanks.
I could only do limited testing with the func.getattr=newest option so far, but it seems to have solved my problem.
Thanks for the pointer! I'll close this now.
⚠️ This issue respects the following points: ⚠️
Bug description
'filesystem_check_changes' => 1setting has no effect, when set in either/var/www/html/config/config.php/var/www/html/config/user.config.phpno error or warning observed.
Steps to reproduce
'filesystem_check_changes' => 1at the end of config.php or user.config.php like so:<nextcloud-data-mount-dir>/<username>/files/test_dir/001.jpg <nextcloud-data-mount-dir>/<username>/files/001.jpgExpected behavior
Copied file shows up at copy destination.
Installation method
Official All-in-One appliance
Nextcloud Server version
25
Operating system
Other
PHP engine version
PHP 8.0
Web server
Apache (supported)
Database engine version
PostgreSQL
Is this bug present after an update or on a fresh install?
Fresh Nextcloud Server install
Are you using the Nextcloud Server Encryption module?
Encryption is Disabled
What user-backends are you using?
Configuration report
List of activated Apps
Nextcloud Signing status
Nextcloud Logs
Additional info
Official AIO images used in a manual install, following the guide.
Observed before and after updating to 25.0.6 from 25.0.5. Configuration in
config.phpwas set viadocker exec -it <container id> bashand vi. Care was taken not to destroy the configured image. As an alternative,user.config.phpwas mounted as a local volume like so: