[Bug]: SMB - No update of data for User B when User A makes changes (notify)

thechonta commented 1 year ago

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

Nextcloud 27.0.0 Desktop client 3.9 Server OS Ubuntu 22.04

Users from Active Directory Integration of external SMB share (Share1) on Server 2012 Credentials stored in the database.

User A and User B have mounted Share1 (as a virtual file system).

User A creates files through the browser or uploads them using the desktop client. Both users have the desktop client running, but User B does not receive the new files. User B is unaware that data has been created.

Only when User B forces synchronization are the new files created. It is expected that when User A or User B creates a file on the share, all users who have access to the share will be notified. In both cases, the upload is done through the Nextcloud server, and it should inform all clients that a file has been created. However, the server does not push notifications to the clients, and the client itself does not check if there are any new files.

Finally, User B has the files and modifies one of the text files. The server sends the file to User B, who makes changes, and the modifications are transferred.

User A is not aware of these changes, so the timestamp for the modification date remains unchanged for User A. Currently, the file exists only in the cloud. User A tries to download the file, receives an error message, and cannot download it. Now User A also has the latest version of the file! User A makes a modification, and the current version is transferred to the server. However, User B is not notified of any changes made by User A. User B still has the local version and is not informed about the modification made by User A.

Even with a forced synchronization, there is no file comparison with the server. Worse still, User B can now modify and save their local file, which would be uploaded again and overwrite the file on the server!

This issue related to integrated Windows shares has been present since Nextcloud version 13 (possibly even 12). The communication between the desktop client and the server is insufficient, and changes are not propagated to all relevant users. I suspect this problem also affects other external shares that are not integrated within a fixed user context.

The solution, in theory, seems simple. The server transfers data from User A to User B, so it should also know which groups/users can access the share and trigger a push sync for all users when a file is modified. If User A modifies a file in the browser, their desktop client should immediately sync the changes!

Then, User B logs into the browser, and the file modification is immediately synced!

Now, User A makes a modification using the desktop client, and the changes are uploaded to the server. User B is still logged into the browser, but the timestamp in their view does not change!!! If User B now accesses the file in the browser, the server establishes a direct connection to the current version on the file server for User B, and the real version is loaded and then synced to the client.

This means that Nextcloud does not update the timestamps in its database for files located on external shares when modified by a user. Only the user who made the modification sees the change updated in the Nextcloud database, but nothing happens for all other users who have access to the same share. This behavior is a major problem. On the one hand, the browser display in terms of timestamps is unreliable, and on the other hand, working with the desktop client is dangerous! (Dangerous because users can overwrite each other's work without being aware that they are destroying or having their work destroyed by others.) When working in the browser, at least the current version is always loaded from the server at the time of access/download. The Nextcloud server is

not supposed to register changes on the Windows file server that were not initiated through the Nextcloud server. However, for all changes that occur through the Nextcloud server, the modifications should be propagated to all users and clients. The basic process actually works, otherwise User A's desktop client would not immediately sync data when User A modifies data through a browser.

Theoretically, there should be no need to check if the user is allowed to view the file on the file server; the file server will deny access if the user is not authorized. Since the valid credentials for the share are also stored in the Nextcloud database, and the upload and sync work through the desktop clients without being logged into the browser, the Nextcloud server could proactively check for changes in the shares whenever it detects that a user has made changes in the share.

Nextcloud is a great product, but it only works reliably when the data is stored locally on the Nextcloud server, preferably organized within group folders. If the reliability regarding external shares on Windows servers were improved, it would greatly enhance usability.

Steps to reproduce

Nextcloud 27 + LDAP + AD-Server + Windows File Server with shares, etc.
Windows share added as an External Share: Credentials stored in the database.
Desktop client installed and set up for Users A and B.
User A creates a file.
User B has to force a sync in order to see the new file.
User A modifies the file.
User B does not receive the file changes! (Even forcing a sync does not help.)

Expected behavior

It is expected that when using External Shares on a Windows server and storing login credentials in the database, changes made to files by User A will also be passed on to User B.

Both User A and User B access the same External Share (using their own login credentials). If User A makes a modification, User B should also be notified, meaning the data needs to be synced, and timestamps should be consistent across all users and applications.

Installation method

None

Nextcloud Server version

27

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

[ ] Default user-backend (database)
[X] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "testcloud.domäne.de",
            "testoffice.domäne.de",
            "192.168.0.1",
            "192.168.2.1",
            "127.0.0.1"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "skeletondirectory": "",
        "overwrite.cli.url": "https:\/\/testcloud.domäne.de:443",
        "dbtype": "pgsql",
        "version": "27.0.0.8",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "ldapIgnoreNamingRules": false,
        "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory",
        "maintenance": false,
        "theme": "",
        "loglevel": 2,
        "trashbin_retention_obligation": "7,14",
        "versions_retention_obligation": "7,14",
        "cache_chunk_gc_ttl": 86400,
        "integrity.check.disabled": true,
        "onlyoffice": {
            "verify_peer_off": true
        },
        "updater.release.channel": "stable",
        "app_install_overwrite": [
            "spreed",
            "occweb",
            "keeweb",
            "sharelisting",
            "polls",
            "files_clipboard",
            "breezedark",
            "issuetemplate",
            "groupfolders",
            "onlyoffice",
            "files_linkeditor"
        ],
        "encryption.legacy_format_support": false,
        "encryption.key_storage_migrated": false,
        "default_language": "de",
        "default_phone_region": "de",
        "default_locale": "de",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "updater.secret": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated Apps

Enabled:
  - activity: 2.19.0
  - admin_audit: 1.17.0
  - calendar: 4.4.2
  - circles: 27.0.0
  - cloud_federation_api: 1.10.0
  - comments: 1.17.0
  - contacts: 5.3.1
  - contactsinteraction: 1.8.0
  - dashboard: 7.7.0
  - dav: 1.27.0
  - deck: 1.10.0
  - federatedfilesharing: 1.17.0
  - federation: 1.17.0
  - files: 1.22.0
  - files_external: 1.19.0
  - files_markdown: 2.4.1
  - files_pdfviewer: 2.8.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - firstrunwizard: 2.16.0
  - groupfolders: 15.0.0
  - issuetemplate: 0.7.0
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - nextcloud_announcements: 1.16.0
  - notifications: 2.15.0
  - notify_push: 0.6.3
  - oauth2: 1.15.0
  - occweb: 0.1.0
  - password_policy: 1.17.0
  - photos: 2.3.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - recommendations: 1.6.0
  - related_resources: 1.2.0
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - sharelisting: 1.1.1
  - smb_test: 0.3.7
  - support: 1.10.0
  - survey_client: 1.15.0
  - systemtags: 1.17.0
  - text: 3.8.0
  - theming: 2.2.0
  - twofactor_backupcodes: 1.16.0
  - updatenotification: 1.17.0
  - user_ldap: 1.17.0
  - user_status: 1.7.0
  - viewer: 2.1.0
  - weather_status: 1.7.0
  - workflowengine: 2.9.0
Disabled:
  - apporder: 0.15.0 (installed 0.15.0)
  - bruteforcesettings: 2.7.0
  - documentserver_community: 0.1.13 (installed 0.1.13)
  - encryption: 2.15.0
  - event_update_notification: 2.2.0 (installed 2.2.0)
  - extract: 1.3.5 (installed 1.3.5)
  - files_clipboard: 0.7.1 (installed 0.7.1)
  - files_linkeditor: 1.1.14 (installed 1.1.14)
  - mail: 3.2.1 (installed 3.2.1)
  - metadata: 0.18.0 (installed 0.18.0)
  - onlyoffice: 8.0.0 (installed 8.0.0)
  - polls: 5.0.5 (installed 5.0.5)
  - spreed: 17.0.0 (installed 17.0.0)
  - suspicious_login: 5.0.0
  - tasks: 0.15.0 (installed 0.15.0)
  - twofactor_totp: 9.0.0
  - user_saml: 5.2.0 (installed 5.2.0)

Nextcloud Signing status

No response

Nextcloud Logs

No response

Additional info

No response

szaimen commented 1 year ago

Hi, please translate your report to english.

thechonta commented 1 year ago

Translatet with the help of chatgpt :-)

szaimen commented 1 year ago

please also adjust the title

kesselb commented 1 year ago

Sounds like https://github.com/nextcloud/server/issues/15033

thechonta commented 1 year ago

Probably, but the bug report of #15033 got kinda turned off, so I started with a new on. The only Assistence i can provide further is teamvewer access to the dev, so they can look for them self i dont talk bullshit and the Server really works like that.

I think it got even worse. Back a view years ago the sync started, if you was logged in into the webbrowser and entered the folder, but it seems to enter the folder in the webbrowser do not trigger an directory refresh on the file server and so the user never know it there are new data or not, till they download the file or upload stuff. Uploading Stiff sometimes triggers an refresh of the folder for new items for the user that uploades a new item.

thechonta commented 1 year ago

As in #15033 mentioned at the end.

/1 * php -f /var/www/nextcloud/occ files:scan --unscanned sudo -u www-data php /var/www/nextcloud/occ files_external:notify -u BenutzerB -p PasswortBenutzerB 6

Changed something in with user B and User A did not got any changes And it would also not be practiacal to have an user, that has access to every file in the fileshare and than the passwort is planetext stored on the linuxserver...

The only way to make it work right is how i described how it should work. Something how nextlouddeal with smb share seems to be changed from 11 to 12/13

Funny: If the PC is restartet and the User Logs in, the Folder ist aktualized. Force Sync does not help, kill nextcloud app does not help...

XTC-Desperado commented 1 year ago

Have a look here https://apps.nextcloud.com/apps/files_inotify and all attachments to it. For the use of a Nextcloud client on the PC, this is mandatory and also has restrictions depending on the number of folders/files. In the browser and on the smartphone app, inotifiy is not necessary because the session actively looks in live when it is called up. The Nextcloud client works differently, regardless of whether with virtual folders or in the offline cache - at the beginning it works, when using the "client" it always works because the client actively plays along. If a user B changes something on the external SMB share, the Nextcloud server doesn't notice - logical, that's why files_innotify. Or make a request to build a function into the Nextcloud client so that it does a "resync", but I think this is impractical for "end users", dangerous and takes too long depending on the amount of data. Here's a challenge for personal home drives "Servername; /home ;/$user ;MyDomain" https://github.com/nextcloud/documentation/commit/32241b66935d135e1540c7e7beccbfdb0ed92323

German: Schau mal hier rein https://apps.nextcloud.com/apps/files_inotify und alle Anhängigkeiten dazu. Für die Benutzung eines Nextcloud-Clients auf dem PC ist das zwingend notwendig und hat auch je nach Einschränkungen je nach Anzahl Ordner/Dateien. Im Browser und auf per Smartphone-App ist inotifiy nicht nötig, da die Sitzung beim Aufruf da aktiv live reinschaut. Der Nextcloud-Client funktioniert anders, egal ob mit virtuellen Ordnern oder im Offline-Cache - am Anfang klappts, bei der Nutzung des "Clients" geht´s auch immer, weil der Client aktiv mitspielt. Wenn ein User B auf dem externen SMB-Share etwas ändert, bekommt Nextcloud-Server davon nichts mit - logisch, deswegen files_innotify. Oder nen Request machen, ob man in Nextcloud-Client da eine Funktion einbaut, das der einen "Resync" macht, halt ich aber für "Endanwender" unpraktikabel, gefährlich und dauert je nach Datenmenge zu lange. Hier mal eine Herausforderung für persönliche Homelaufwerke "Servername; /home ;/$user ;MyDomain" https://github.com/nextcloud/documentation/commit/32241b66935d135e1540c7e7beccbfdb0ed92323

XTC-Desperado commented 1 year ago

_> As in #15033 mentioned at the end.

/1 * php -f /var/www/nextcloud/occ files:scan --unscanned sudo -u www-data php /var/www/nextcloud/occ filesexternal:notify -u BenutzerB -p PasswortBenutzerB 6 > Changed something in with user B and User A did not got any changes And it would also not be practiacal to have an user, that has access to every file in the fileshare and than the passwort is planetext stored on the linuxserver...

Try to create a user www-data in your domain witch hat reading abilities to the external SMB-share - same password for the www-data host. Don´t know if you can hide/hash the password in a script - else you will see the process with cleartext-pwd. Make a cron-job every hour - take care of remote SMB-Shares, quite slow. Even local SMB-Shares with alot of data results in endless loops.. A live test needs patience to see. Fire command, take a brake and get a coffee, come back an see if it works.

_>

The only way to make it work right is how i described how it should work. Something how nextlouddeal with smb share seems to be changed from 11 to 12/13

Funny: If the PC is restartet and the User Logs in, the Folder ist aktualized. Force Sync does not help, kill nextcloud app does not help..._

It uses a local db file for filechanges - hashes, whatever... Resync doesn´t help either. There is also a delay of sync, mostly 15min.

check also this one, if many files/folders are scanned https://github.com/awslabs/amazon-eks-ami/issues/1065

kesselb commented 1 year ago

Glad to hear files_inotify made it work for you :partying_face:

I assume you mounted the smb share in linux and added it to Nextcloud as local storage?

thechonta commented 1 year ago

@XTC-Desperado If i run: sudo -u www-data php /var/www/nextcloud/occ files_external:notify -v 6

 at /var/www/nextcloud/apps/files_external/lib/Lib/Backend/SMB.php:79
 OCA\Files_External\Lib\Backend\SMB->manipulateStorageConfig() at /var/www/nextcloud/apps/files_external/lib/Command/Notify.php:167
 OCA\Files_External\Command\Notify->execute() at /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php:298
 Symfony\Component\Console\Command\Command->run() at /var/www/nextcloud/core/Command/Base.php:177
 OC\Core\Command\Base->run() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:1040
 Symfony\Component\Console\Application->doRunCommand() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:301
 Symfony\Component\Console\Application->doRun() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:171
 Symfony\Component\Console\Application->run() at /var/www/nextcloud/lib/private/Console/Application.php:211
 OC\Console\Application->run() at /var/www/nextcloud/console.php:100
 require_once() at /var/www/nextcloud/occ:11

files_external:notify [-u|--user USER] [-p|--password PASSWORD] [--path PATH] [--no-self-check] [--dry-run] [--output [OUTPUT]] [--] <mount_id>

For me it is not an Option to ad even a sigle user with passwort in an cronjob on the Server. And if it would probably only sync the data of that user and not of User C, D, E and so on.

The only way to make it work for everyone and every external storrage is. The Desktopclinet of User A informs the Nextcloudserver about changes for the folder/file (all the transfered date are going trough the server anyway) and the Server informs all user, that have the right to access the external storrage about changes. Now the cleint of user B (C,D,....) check the version of ther local file and the file on the server. If there is anny changes, than the will adapt the changes, if not, or access forbidden, the do nothing.

No need for an notify tool that needs userlogin and an cronjob. The only way files_external:notify would be usable would be if it coult set for -alluser and take the databases stored login.

@kesselb My SMB Storage is conectet within the Nextcloud itself as external Storrage.

szaimen commented 1 year ago

For built-in smb, there is this: https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/external_storage/smb.html?highlight=notify#smb-update-notifications

thechonta commented 1 year ago

@szaimen Thank you for the Link. Even when using ‘Login credentials, save in database’ or ‘User entered, stored in database’ authentication the notify process can not use the credentials saved to attach to the smb shares because the notify process does not run in the context of a specific user in those cases you can provide the username and password using the --username and --password arguments.

this makes the notyfy still useless for me. Think of an envirroment with 120 users and monthly changes of 10 Users that that go ore come new. the notyfy wort, ich would have to permanently adrust cronjobs for every user and would have to know ther passwords....

The whol process should not relay on some notify subsystem that need interaction in the user context. Filechanges are stored in the psotgss/mariadb database of the nextcloud server, if realy not than this is one of the first thing that have to change to have an relaiable notifying solution.

User A changes a file > change is market in the server database > server inform all user that can access the external storrage about of change of file > the Desktopclint/app now check the specifc file/folder and confirm the changes.

thechonta commented 1 year ago

Push push :-) For me the behaviour is a bug and not relay practical. But is my problem also official recognized as bod or as work as designed?

If it official not considered as a bug, how to make an Feature Request for making Nextcloud work properly with an storagebacked on a Windows Fileserver.

thechonta commented 1 year ago

https://help.nextcloud.com/t/problem-in-design-of-fileindex-and-fulltextindex-and-kommunikation-between-client-and-server/168388

There i discripe the fundamental Problem with the usuage (and not usuage) of oc_filecache waht is probably the reason why the Problem described in thes topic exisit.

thechonta commented 1 year ago

push push

nextcloud / server