Closed Amanoo closed 7 years ago
Yeah, direct modifications to SMB shares need a detection script to run. @icewind1991 should have more information on that.
SMB is actually the thing that seems to work best for me. Changes are detected just fine. It's just slow. Even though the SMB server and the Nextcloud server are on the same device, I can't get more than 10.5MB/s out of it, despite my network being capable of much higher speeds. That's my only problem with it.
It's FTP type External Storages and shared Local External Storages that refuse to adequately detect changes. The FTP type just screwed up, basically. Just blatantly doesn't want to detect any changes at all. I would assume that my problem with Local type storages would also occur with any other type of storage, including SMB, Dropbox, and whatever else, as long as you share it with other Nextcloud accounts. It's probably the act of sharing an External Storage that's a problem. However, I haven't tried it with other storage types, so I can't confirm my suspicions.
I am going to add a comment here because my issue is related to this but it may be separate. I have several instances of Nextcloud and Ownclould running. All of them use SMB External access for files. I am giving access to common and other shares that the user would have while in their office. I have had a terrible time keeping what is shown through Nextcloud to keep in sync with whats shown via Windows explorer in the office. I have had to start running the occ files:scan --all. Last weekend after trying to sort out a bunch of conflict files and running the scan to pick up new changes I noticed that every time the scan ran all the folders in the nextcloud web page would update and show that they had just been changed. The files kept their dates and times the same but all folders would update each time. I can't say for sure but I have a few users that get a number of conflict files each day. The conflicts happen with the desktop sync client running.
I can confirm. Seeing the same issue related to changes with shared external storage with Nextcloud 10. I'm using "Local" external storage. This issue was present on my previous OwnCloud install as well.
I would suggest adding a third option to the check for changes setting that includes when shares are accessed.
I am adding another issue to this thread. Again this may be a separate issue but it's related. I have now seen this on both Nextcloud 10.0.1 and Owncloud 9.1.0
I have shares connected to via SMB and the Check For Changes is set to Once every direct access I have at least two users accessing this share with the windows desktop sync client. Nextcloud server is matched with the Nextcloud desktop client and the Owncloud sever matched with the owncloud client. I am seeing buildups of connections to the samba server. They show up from the webserver but don't show that they are accessing any files. The problem is as these connections build so does the load on the web and samba servers. I have a 4 core machine and I caught the server load shown in Nextcloud at over 80 the other day. I went to the samba server and terminated all the connections and the server dropped back to under 1. If the sync client is turned off this ends but it also ends if I change the settings for Check For Changes to Never This may make sense that if there is scanning going on that there would be load. I don't mind some load and some bandwidth usage but this is bogging down the server too far. Has anyone else seen this and been able to solve the problem?
Same behaviour with NC 11. Files uploaded via ftp are not visible via NC web or client, even if Check for changes is set. Please change label to bug. Thanks
I noticed that every time the scan ran all the folders in the nextcloud web page would update and show that they had just been changed
Fix is here: https://github.com/nextcloud/server/pull/3447
I can't reproduce the problem of changes on a local external storage not being detected when shared
Files have to be down several levels in a folder structure. One or two levels work just fine. Go 8 or 9 levels down and the regular scan just doesn't make it that far. I have had to set up the occ scan for several of my clients. I don't know if the occ scan is still updating the files it scans with the current time. With a previous version it was as if the "touch" command was being used. I'll look at the folders and files and see if the occ scan is doing that currently on NC 11.0.1
I just checked one of my setups. The files being scanned every 15 minutes via occ scan are not being updated to the current date and time on each scan. I am running NC 11.0.1
I have had to set the option in the share [Check for Changes] to never or it will create hundreds of connections to the samba server bogging down the webserver. I have found it with a server load of 170+ on a four core machine. Normal operation is under 1. So the occ scan is they only way I can operate on a larger share. A little additional information that may or may not be pertinent to this situation: my connection to the samba server where the data is located is over a 5mbps connection.
installing php-smbclient will usually give an improvement in smb performance
Regarding the detection script: See https://github.com/nextcloud/server/issues/1016#issuecomment-279032265 this describes what to do to set it up.
This should solve this issue here.
Hate to necro a bug, but if what @MorrisJobke and others are referring to works for Samba hosted SMB share file change notifications, can we PLEASE have the documentation updated to say that it WORKS??? I'm talking about : https://docs.nextcloud.com/server/12/admin_manual/configuration_files/external_storage/smb.html
" Note
Due to limitations of linux based SMB servers, this feature only works reliably on Windows SMB servers. "
I've seen a few people in this issue thread alone say it works, but why do the docs say it doesn't? :S
All times I've tested it update notifications with linux servers are not recursive, you only get notifications for changes made directly in a folder, not in any subfolder.
@icewind1991 so what would it take to have that addressed?
It's something that can only be fixed properly in upstream samba
Has this been passed to the Samba devs like abartlet?
All times I've tested it update notifications with linux servers are not recursive, you only get notifications for changes made directly in a folder, not in any subfolder.
Testing this currently with Samba 4.15.13 and this issue seems to be fixed. Can anyone confirm?
The "Check for changes: Once every direct access" option doesn't seem to always work, and it's very frustrating. What I wanted to do is give users read access to certain folders, but not write access. Since I can't add Local Storages to an account as "read only" directly (which would be a nice feature, BTW), I have to be a bit creative. At first, I tried adding "local storage" to my own account, and then sharing it with the other users, giving them read only access in the process. Thing is, I actually need to access folders from my own account to detect changes made within them. If the user tries to access the folder without me being logged in, the system doesn't check for chances. Apparently, accessing a shared folder doesn't count as an access. As a result, any newly added files won't show up. I then tried to add an FTP type External Storage. But that one didn't detect any changes ever. It initially added the external folder just right, but the system doesn't check for any changes at all, no matter which account explores the FTP. Right now, I'm using an SMB type external storage. It works, the guest accounts can only access read only shares, so that's all good, but it's kind of slow. I get download speeds of about 10.8MB/s, whereas I can go up to 52.5MB/s if I access the same SMB share from desktop's file explorer. And since I'm using SMB to have Nextcloud access the localhost's hard drives, it is kind of redundant to use a protocol like that. The type Local should just work. The type Local is also much faster, so I think Nextcloud's SMB client is just slow. But Local doesn't just work, since it doesn't always check for changes in a folder, and I have to resort to methods like this. In short, my problems are as follows: the External Storage system doesn't always check for changes, even though it should, and the SMB client built into Nextcloud (or maybe built into External Storages, not sure how it is implemented exactly) is kind of slow.
Note that SMB server, VSFTPD (FTP server), and Nextcloud are all hosted on the same server.
Steps to reproduce problems with Local External Storage
Expected behaviour
New files should immediately be detected by the second account.
Actual behaviour
Changes don't actually show up on the secondary account, unless I login into the admin account again and browse to the shared folder. Only then can the secondary account see the changes.
Note: I'm not entirely sure if it happens every time. After logging out as admin, you might have to wait a bit, but I'm not entirely sure. However, changes will eventually cease to be detected until you login as admin again.
Steps to reproduce problems with FTP External Storage
Expected behaviour
New files should immediately be detected by the second account.
Actual behaviour
Changes don't show up.
Server configuration
Operating system: Debian Testing (Stretch) Web server: Nginx 1.10.1 Database: mysql 5.6.30 PHP version: 7.0.10 Nextcloud version: (see Nextcloud admin page) 10.0.0 Updated from an older Nextcloud/ownCloud or fresh install: Fresh Where did you install Nextcloud from: Downloaded the zip file from the Nextcloud website Signing status:
Signing status
No errors have been found.List of activated apps:
App list
Enabled: - activity: 2.3.2 - comments: 1.0.0 - dav: 1.0.0 - documents: 0.13.1 - federatedfilesharing: 1.0.1 - federation: 1.0.1 - files: 1.5.2 - files_accesscontrol: 1.1.2 - files_external: 1.0.2 - files_pdfviewer: 0.8.1 - files_sharing: 1.0.0 - files_texteditor: 2.1 - files_trashbin: 1.0.0 - files_versions: 1.3.0 - files_videoplayer: 0.9.8 - firstrunwizard: 1.1 - gallery: 15.0.0 - notifications: 0.3.0 - ownnote: 1.08 - password_policy: 1.0.0 - provisioning_api: 1.0.0 - serverinfo: 1.1.1 - survey_client: 0.1.5 - systemtags: 1.0.2 - theming: 1.0.1 - updatenotification: 1.0.1 - workflowengine: 1.0.1 Disabled: - admin_audit - encryption - external - files_automatedtagging - files_retention - templateeditor - user_external - user_ldap - user_samlThe content of config/config.php:
Config report
{ "system": { "passwordsalt": "**_REMOVED SENSITIVE VALUE**_", "secret": "**_REMOVED SENSITIVE VALUE**_", "trusted_domains": [ "localhost", "Are you using external storage, if yes which one: local/smb/sftp/... Local and SMB. Tried to use FTP, but didn't work as planned Are you using encryption: yes/no SSL Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/... No
Client configuration
Browser: Firefox 49.0b1 Operating system: Debian Testing (Stretch)
Logs
Nextcloud log (data/nextcloud.log)
Nextcloud log
{"reqId":"Phli3e\/a8gKiehC9v++y","remoteAddr":"","app":"mysql.setup","message":"Specific user creation failed: An exception occurred while executing 'SELECT user FROM mysql.user WHERE user=?' with params [\"oc_admin\"]:\n\nSQLSTATE[42000]: Syntax error or access violation: 1142 SELECT command denied to user 'nextuser'@'localhost' for table 'user'","level":3,"time":"2016-08-27T05:17:38+00:00","method":"--","url":"--","user":"--"} {"reqId":"Phli3e\/a8gKiehC9v++y","remoteAddr":"","app":"mysql.setup","message":"Database creation failed: An exception occurred while executing 'GRANT ALL PRIVILEGES ON `nextcloud` . \* TO 'nextuser'':\n\nSQLSTATE[42000]: Syntax error or access violation: 1044 Access denied for user 'nextuser'@'localhost' to database 'nextcloud'","level":3,"time":"2016-08-27T05:17:38+00:00","method":"--","url":"--","user":"--"} {"reqId":"Rx9nT2umUjccscTyiLr3","remoteAddr":"