SMB Sync multiple users

[thechonta]

Server configuration

Operating system:Ubuntu 18.04 Web server:Apache 2.4.29 Database:mariadb Ver 15.1 PHP version:7.2.15 Nextcloud version:15.0.7 Desktopclient 2.5.0++ Client OS:Windows 10/ Windows 7 **SMB Server 2016

The issue you are facing:

Is this the first time you’ve seen this error? (Y/N): N

Hello together. Given is an nextcloud updated to 15.0.7 on an Ubuntu 18.04 up to date and smbclient Version of 4.9.5 because auf troble of Notification SMB in the 4.7.X version that comes with Ubuntu but the following problem is regardless of that.

Userbackend is LDAP (AD) on an Server 2016. It is used the external_store SMB/CIFS with logindata saved in Database. In the Store is Accessenumeration active, the share has a lot of sub folders.

All Users uses the Nextcloud Desktop client. User A changes an File on the Share on first Level the on the Desktop the File Changes will be submitted to User B as well. User A changes an File though web-browser (even txt) the changes only submitted to user A, user B has to open the file in the browser or to visit the folder of the file in the browser, than it will be synced to the computer.

User A changes an file or create an file in an subfolder, no changes will be noticed for user B. If user B browses to the file, than it will be synced.

The expected behavior is, that user A changes an file or creates something on the share (per client or browser), user B will sync the data to his desktop as well. A view minutes after i was upgrading from 15.0.4 to 15.0.7 it worked like this but than all of a sudden it stopped working. There no errors in the Logfiles of the nextcloud so far.

If the share set up with username and password saved in the nextcloud, than it works. But than the accessenumeration on the fileserver could not be used. Because the fact, that changes on the first level got noticed, as log as it was initiated per client are synced, shows that it should work also for subfolder as well. The nextcloud should know what file was changed and provide the clients the information so the clients can check the backend store.

Is there a workaround to force the clients to checkup the store for changes?

Thank you for help.

[whberkhof]

Hi, if have same issues

[thechonta]

Maby it is an Bug related to the Syncclient itself.

[thechonta]

Push push, still not working :-) If the server and the desktopclient would proerly share information about filechanges among all users Live would be much easyer. :-)

[thechonta]

Push

[kesselb]

Are you looking for https://docs.nextcloud.com/server/16/admin_manual/configuration_files/external_storage/smb.html?highlight=notify#smb-update-notifications?

[thechonta]

Tested Nextcloud Server 15.0.4.0 PHP: 7.2 Ubuntu 18.04.2 SMB Backend on Server 2012, Credential saved in Database. Authbackend: LDAP (AD) Clients: Nextcloud-2.2.3.4-, Nextcloud-2.2.4.2, Nextcloud-2.3.1.8, Nextcloud-2.3.2.1, Nextcloud-2.3.3.1, Nextcloud-2.5.1, Nextcloud-2.5.3.5526-daily-20190614 As long there is only one User that uses the Client to Sync date between Server and Client and the changes are only made by the Client the sync will work.

The moment there are two users that sync and changes the same data it will not work. User A changes, and the Changes are synced to the Server but User B will never get any changes as long he changes some file in the same Folder itself. In this moment magically the changes made by User A will also synced but user A will not get the Changes of user B….

Also as normal there are changes on the Fileserver itself, there will no sync between Server and User, as long there is some changes by the User….

Expected is there is a change on the User A User B will get the changes synced. B changes something A will also get the information. Also if something is changed on the external storage by User C, it should by synced to the Nextcloud user A and B.

The server should manage all changes of the users in the database and the clients should look the server database for changes off all data of all users they have access. And on the server should be possible to set a sync-user to check the external storage for changes, and store it in the database to inform all the users.

[thechonta]

@kesselb Thx for the Link. I read it before, waht the link does not say, how can this used to create realtimenotification of changes for each user? The administrator does not have the passwords, the users will not type in the line. Is it enough that i create an croncom in on user context an all users will have access to the information and by up to date?

I need something that helbs to keep all the date in sync between multiple users without useraction.

[kesselb]

cc @wiswedel @cowai any idea? I'm not familiar with smb.

[ariselseng]

There was a SMB notify reconnection fix 15.0.8 for when the DB connection times out. That could explain why @thechonta says it works for a little while. I have never used per user mounts. Only global smb mounts. Its very important to have the latest nextcloud for SMB as I did some PR to fix my use cases quite recently.

@thechonta Can you reproduce this in 15.0.8? Also, I would recommend latest 16 as that is what I tested my PRs on: https://github.com/nextcloud/server/pulls?utf8=%E2%9C%93&q=is%3Aclosed+is%3Apr+author%3Acowai+

[thechonta]

I will be able to go on the latest 16 of nextcloud on Thursday maby tomorrow. i will leave a comment :-) Thx for respond :-)

[thechonta]

https://help.nextcloud.com/t/mounting-hidden-share-as-external-cifs-storage-fails-after-nextcloud-16-upgrade/55273 After Update to 16.0.1 an Externale Share with $ (hitten share) on MS Server 2016 is marked red. But it is possible to access all folders if you know the path url / or set the subfolders as favorite.

[kesselb]

https://help.nextcloud.com/t/mounting-hidden-share-as-external-cifs-storage-fails-after-nextcloud-16-upgrade/55273 After Update to 16.0.1 an Externale Share with $ (hitten share) on MS Server 2016 is marked red. But it is possible to access all folders if you know the path url / or set the subfolders as favorite.

https://github.com/nextcloud/server/issues/15567 is the related issue.

[thechonta]

https://github.com/nextcloud/server/issues/15567

if (strpos(rtrim($option, '$'), '$') !== false) {
$result = false;}

Did the magic :-)

[kesselb]

Did the magic :-)

• Upgraded to Nextcloud 16 and run into the issue with the hidden shares
• Applied the patch and the hidden share is working again
• The sync issue is still happening (if not we could close this issue)

Right? It's quite hard to follow you. Please try to be precise about your problem. Thank you :+1:

[thechonta]

• Upgraded to Nextcloud 16 and run into the issue with the hidden shares yes

• Applied the patch and the hidden share is working again yes (lot of thanks to the one who found out :-) )

• The sync issue is still happening sadly yes the sync is still not working as expected. Even with the latest dayly Desktopclient it did not work (no matter if windows or Linux Client)

User B Change a File and User A will not Sync the changes till he create a file or does any Changes (one folder above the file that changes or in some folder beneth) It looks that it will only searched for changes if you do changes by yourself and not check the database on the server for just in case Even an force syncing has no effect.

The Sync issue is still there.

[ariselseng]

@thechonta just to confirm I understand your issue: It is only happening if you use access enumeration? More info about access enumeration here.

[ariselseng]

@icewind1991 is access enumeration with multiple credentials supported in files_external?

[thechonta]

@cowai Thx for Respond. On Shares without access enumeration it is the same.

[ariselseng]

@thechonta This is a long shot, but, could you try out desktop client version 2.3.3? Maybe this is related to this issue: https://github.com/nextcloud/desktop/issues/1000

[thechonta]

Allready tested with the following clients: Clients: Nextcloud-2.2.3.4-, Nextcloud-2.2.4.2, Nextcloud-2.3.1.8, Nextcloud-2.3.2.1, Nextcloud-2.3.3.1, Nextcloud-2.5.1, Nextcloud-2.5.3.5526-daily-20190614

[ariselseng]

@thechonta Just to be sure. Your issue is about the Nextcloud desktop client and the Nextcloud web interface. You are not talking about interactions directly to the SMB share? If you are not, can you replicate this on a folder that is not on SMB?

[ariselseng]

@thechonta If you are using the SMB share directly, you must find a way to use occ files_external:notify and then run php occ files:scan --all --unscanned like every 30 seconds (or wait for cron to do it).

[thechonta]

@cowai The Issue is between the Desktopaps itself. There is no directly writing to the share, all data only transfered via the Desktop ap. Each Userchange is transfered to the Server Fileshare but the Change that is made By User A is not synced back from the Server to the lokal PC of User B. Only User B create a filechange in the same Folderstructure (on level above the change or at the end) where User A made a change user B will sync the Changes of User A. But than Uer A will not get changes of User B.

Aslong there would only be one user, everything is in sync. the moment you have another user that musst sync and change data of the share you are lost.

The Access over browser leads always to the actual data but changes over browser from user A will not synced to user B.

If User A and B are loged in web in the same folder and make changes, each will get the changes synct to the client.

Also working. User B is loged in the webclient via browser. User A changes something in the same folder user B is loged in the browser, user B will sync the changes with the desktopclient. ################# How i can use occ files_external:notify whitout Userdata as a cronjob? Can i use for that an genereic account an all users will be get the changes?

occ files:scan --all --unscanned Does not work because if i run this it will check the localfilesystem and not the external share

the only way to bring this to work i see, is that all changes from an user will written to the nextcloud database (mqsyl, postgres wahtever) and the desktobclent ask ths database for changes and sync the data. to check the filesystem itself for changes seems only to work with local filesystems or with global usercredentials.

[ariselseng]

Alright so if you only use SMB via nextcloud and never directly, then occ files_external:notify should not be nessesary. But if you do read this: if you put in credentials that have access to the folders you can use occ files_external:notify. What the command does is get the change for the SMB server and then mark its parent folder as size with -1. the "--unscanned" parameter of occ files:scan just looks up all folders that have size -1.

How are the folders setup in nextcloud? As a global external storage or on each personal account?

[thechonta]

It is an global share, setup like in the picuture. Logincredentials stored in Database

Little update for the browser. To get the changes synced an other user make, you have at least to actualize the website https:///index.php/apps/files/

[ariselseng]

@thechonta I will try to reproduce this later.

If you just want this to work I would recommend just doing regular username/password for now.

[thechonta]

@cowai Sadly i need an working solution :-) without global/credentials The complete situation is to complex to make just a workaround. I have not test it, but i guess other external storage like nfs, amazon are also affected.

The goal is to set less as possible right management in the nextcloud and everything on the storage server itself. The fun fact is that the mechanic seams to work because the client sync missing data if i am log in to the browser and refresh https://nextcloudserver/index.php/apps/files/ Or if i change something else in the path my client will sync also the data that changed by other users.

Maby it would be enough if the desktopclient would set an request to https://nextcloudserver/index.php/apps/files/ like an webbrowser to get missing information about changes.

[ariselseng]

@thechontaI I think the reason why it works when you go to the root in the web ui is that it checks the storage smb modified time with the modified time stored in the database. And triggers a change. But I think that will not always work if you have deep nested levels, as NTFS will not update all modified timestamps up the tree.

I understand this is frustrating for you. I think if you got the files_external:notify to work and ran files:scan --unscanned --all or cron at quick intervals it could help getting the server to rescan.

[thechonta]

@cowai Ok how to run files_external:notify for 100 user without having their credentials and not get them and dont forget the credentials will change from time to time. files:scan --unscanned --all i will run this job every second if it would work bit if i run files:scan it check only the local filesystem. I have an cronjob already running but it will not check the external storage. And if it does will all user get the information?

[thechonta]

How cani pass an Password that includes ! to files_external:notify ?

[ariselseng]

Sorry, I think I was mistaken about how it worked. You would need to provide it with --username --password, but I think it filters out changes made from within nextcloud...

[ariselseng]

I have successfully replicated what your issue using authentication method: "user entered, store in database".

[ariselseng]

@thechonta If I run occ files:scan --unscanned --all, it makes the two desktop clients get the same data, without check the browser. Try this in the terminal as web user:

while true;do occ files:scan --unscanned --all -v;sleep 30;done

[thechonta]

But if i change it to that, the Share is unavailable for everyone till they enter username and password in the webfrontend of the share

Than the user has to put in his credentials with domain\username and to CLICK on save. On one User the sync broke the Desktopclientsync with http error. My Client able to sync with the server but will not sync changes are made from another user via Webbrowser. I have to login via webbrowser and open the file to get the changes syncet to my client.

sudo -u www-data php -f /var/www/nextcloud/occ files:scan --unscanned --all -v

Chek wo all of the local linuxfilesystemsmounts and found as expected nothing.

The storage is on an Windows Server 2016 Fileserver and only available through Nextcloud.

[ariselseng]

@icewind1991 Can you scan through this thread and see if you can think of anything?

[thechonta]

Some additional Info To prevent Data on the Linuxsystem all files folder of the user are dr-xr-xr-x

sudo -u www-data php -f /var/www/nextcloud/occ files:scan --all

wil give than an error for not writpermission in the files folder of the user, but this is how it should be because on the linux filesystem should prevent to save data.

sudo -u www-data php -f /var/www/nextcloud/occ files:scan --all --unscanned -v

Runs withoud error.

+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 0       | 0     | 00:00:00     |
+---------+-------+--------------+

Now i have add the write permission but there will not be found any changes or will be synced files. The only thing that has changed ist i get a list of files if i run sudo -u www-data php -f /var/www/nextcloud/occ files:scan username -v

of all the files, bit it does not help for sync the data properly.

smbclient -V
Version 4.9.8

[ariselseng]

Based on your information. There should be a bug to find in the code that gets requests from the client. Somewhere, parent folder of a change is not properly set size to -1.

[thechonta]

How is it possible to pass an passwort with an !

occ files_external:notify -u username -p passwort!#!99 <mountid>

i tried \ befor the ! and #, with and without " ", also ' ' or

Thank you a lot for your support

[ariselseng]

Make sure you use bash and not just sh. Not try to do it directly and not with sudo -u.

www-data $ occ files_external:notify -u username -p 'passwort!#!99' <mountid>

[thechonta]

Thank you. I have to set the user also

www-data $ occ files_external:notify -u 'domainname\username' -p 'passwort!#!99' <mountid>

but than comes

Protocol negotiation failed: NT_STATUS_CONNECTION_RESET
Error while running self-test, no changes detected

Befor i had an

In Parser.php line 104:

  Invalid login

if i abrot the working command than folowing output apears.

sudo -u www-data php occ files_external:notify --user 'domainname\username' --password 'passwort!#!99' 1 -v
Protocol negotiation failed: NT_STATUS_CONNECTION_RESET
Error while running self-test, no changes detected
^C
In Connection.php line 112:

  [Icewind\SMB\Exception\ConnectException]
  Unknown error (WARNING: The "syslog" option is deprecated
  directory_create_or_exist: mkdir failed on directory /var/run/samba: Permission denied
  Unable to initialize messaging context
  WARNING: The "syslog" option is deprecated)

Exception trace:
 () at /var/www/nextcloud/apps/files_external/3rdparty/icewind/smb/src/Wrapped/Connection.php:112
 Icewind\SMB\Wrapped\Connection->unknownError() at /var/www/nextcloud/apps/files_external/3rdparty/icewind/smb/src/Wrapped/Connection.php:75
 Icewind\SMB\Wrapped\Connection->read() at /var/www/nextcloud/apps/files_external/3rdparty/icewind/smb/src/Wrapped/NotifyHandler.php:77
 Icewind\SMB\Wrapped\NotifyHandler->listen() at /var/www/nextcloud/apps/files_external/lib/Lib/Notify/SMBNotifyHandler.php:74
 OCA\Files_External\Lib\Notify\SMBNotifyHandler->listen() at /var/www/nextcloud/apps/files_external/lib/Command/Notify.php:148
 OCA\Files_External\Command\Notify->execute() at /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php:255
 Symfony\Component\Console\Command\Command->run() at /var/www/nextcloud/core/Command/Base.php:166
 OC\Core\Command\Base->run() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:901
 Symfony\Component\Console\Application->doRunCommand() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:262
 Symfony\Component\Console\Application->doRun() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:145
 Symfony\Component\Console\Application->run() at /var/www/nextcloud/lib/private/Console/Application.php:213
 OC\Console\Application->run() at /var/www/nextcloud/console.php:97
 require_once() at /var/www/nextcloud/occ:11

[thechonta]

UPDATE:

if notyfy is running on the Bash with usercredentials, it will take some time but changes will be notified and synced as long the command runs. If i stop it, change data and start again, files that was changed during the downtime will not be transfred. And it helps only the user in wich context the job is running, all other users will not get any changes.

i would need to run this job for every user on the nextcloud with the credentials of every user... Not prakticable as long the job could get the credentials that is stored in the database and i ned only on job that creates for every user that running deamon. But i gues the load will explode if you have a lot of users.

[ariselseng]

The notify command does not update anything. It only marks a folder as "needs check". To actually make the clients get the changes, cron needs to run (or occ files:scan --unscanned). Are you sure you run cron between each change to confirm that its not working for all users?

[thechonta]

as www-data

*/15  *  *  *  * /usr/bin/php -f /var/www/nextcloud/cron.php
*/5  *  *  *  * /usr/bin/php -f /var/www/nextcloud/occ files:scan --all --unscanned

[ariselseng]

the "unscanned" triggers a background scan (only checks folders that are marked with a needs check). The same background scan is also ran in cron. I would have made it run like each minute. Its not a very expensive command.

[thechonta]

*/1  *  *  *  * /usr/bin/php -f /var/www/nextcloud/occ files:scan --all --unscanned

And if i run this every nanosecund it does noting

[ariselseng]

There is no files found if you do a change from a desktop client?

[thechonta]

There will be only found and synced data if the notyfyjob will run, otherwise nothing will happen. And only the user ther is an notifyjob running will get changed data that was changed by other user.

[kesselb]

I have successfully replicated what your issue using authentication method: "user entered, store in database".

Mind to share the steps?

[ariselseng]

@kesselb I am afraid I did not replicate all of his issues. Just that it took waiting until cron had run before changes were seen between desktop clients, which does not make sense to me why it should be like that. If I ran occ files:scan --unscanned --all like every 10 seconds, I saw the changes faster.

In his case this does not help so I think it must be something specific with using ldap authentication of some kind. @kesselb Do you know of cases where ldap authentication works along with smb mount using login credentials?

[thechonta]

Well well some new Information

sudo -u www-data  php -f /var/www/nextcloud/occ files:scan --all

Brings nothing at the moment. with or withoud --unscanned. It may list some Terrabytes but does not help atall.

 sudo -u www-data  php -f /var/www/nextcloud/occ files:scan --path="/username/files/FILESHARENAME/SUBFOLDER FILESHARE" -v

Does indeed help, but only the user this is running for. Sadly it is not possible to run this command for all user at once and if --unscanned is added, nothing will happen if files are changed or not. To get the sync i need a cronjob for all files ans all shares for each user and that on filshares with some TB of Data :-)

So possible something with this command in combination with external storage is broken.

I hav also tested to mount an smb share

sudo mount -t cifs //SERVER/Testfreigabe$ /mnt/cloud -o user=Administrator,uid=33,gid=33

Than this mount has to be setup in the Nextcloud as external_local share. The Sync works pretty fine, Versioning works also. Same as the other external_share i have to run:

 sudo -u www-data  php -f /var/www/nextcloud/occ files:scan --path="/username/files/FILESHARENAME/SUBFOLDER FILESHARE" -v

But here it is enough tu run it for one user, the rest will sync