file sync - external storage SMB / CIFS using OC login

[cdamken]

We have a problem with the syncing of files from external storage which is connected via SMB /CIFS using OC login. The files only seem to be synced if the user logs into the webinterface. After a login to the webinterface, the files also get synced to owncloud clients.

In the administation menu its set up as Cron.

I've set up two cronjobs for the apache user:

/5 * * * * php -f /var/www/html/owncloud/cron.php > /dev/null 2>&1 /5 * * * * php occ files:scan --all

I think the problem lies with the SMB / CIFS login. The filescanner cannot scan the files without the login data.

Is there an option to enable something like this:

-user starts his laptop with owncloud client installed -owncloud client connects to the server -server starts a file scan of the smb mount with the user login data -client pushes credentials in regular intervals to the server for further scans

I've selected Cron from the beginning, but somehow the Ajax scheduling method seems to remain active as well. I see in the cron logs that both owncloud cron jobs are executed every 5 minutes.

or what would be the correct setup to solve this problem?

Server Architecture:

ownCloud Enterprise Edition 8.0.2 (stable) running on RHEL 7 (Linux 3.10.0-229.1.2.el7.x86_64) libsmbclient.x86_64 4.1.12-21.el7_1 httpd.x86_64 2.4.6-31.el7 php.x86_64 5.4.16-23.el7_0.3 mariadb.x86_64 1:5.5.41-2.el7_0

@MorrisJobke

00003055

[RobinMcCorkell]

The best solution for this will be to store the user credentials (encrypted, of course) in mount.json, and update that file on every login to maintain a recent password. Then we don't need to resort to trickery at all. Doing this the Right Way™ is proving difficult however: see the various discussions at #8167, #12635, #15451

[MorrisJobke]

I've selected Cron from the beginning, but somehow the Ajax scheduling method seems to remain active as well. I see in the cron logs that both owncloud cron jobs are executed every 5 minutes.

That's weird. The ajax cron could be triggered (log entry in access log) if the web page isn't refreshed yet by someone and still the old JS code is present there, but it will not invoke any operation (just browse https://your.domain.com/cron.php to see, that cron mode is active.)

[MorrisJobke]

@icewind1991 Isn't the filesystem scanner also run on the webdav accesses? Could be a propagation of the scanner be the problem here?

[cdamken]

@Xenopathic When will be those pull requested done?

@MorrisJobke @icewind1991 If the scanner does not run over SMB, how can be checked the files?

[MorrisJobke]

@icewind1991 Isn't the filesystem scanner also run on the webdav accesses? Could be a propagation of the scanner be the problem here?

@icewind1991 ping

Maybe @PVince81 can also answer this question :)

[MorrisJobke]

The best solution for this will be to store the user credentials (encrypted, of course) in mount.json, and update that file on every login to maintain a recent password.

The reason for not storing the password could also be due to security aspects. Storing the password could also be the wrong way. cc @LukasReschke

[MorrisJobke]

I don't see a quick fix for this.

[icewind1991]

@icewind1991 Isn't the filesystem scanner also run on the webdav accesses? Could be a propagation of the scanner be the problem here?

Just like with the web interface, only the folder loaded over webdav is checked for updates, no deep check is done (to expensive to do during the request)

I have some ideas to better implement change detection but the problem with using oc login still exists

[MorrisJobke]

The reason for not storing the password could also be due to security aspects. Storing the password could also be the wrong way. cc @LukasReschke

@LukasReschke Pong :P

[cdamken]

@MorrisJobke @LukasReschke @icewind1991 Could you take a look please? Do you need extra information?

[cdamken]

A.countryitecture for ownCloud Support:

Server stack:

• Operating system (name and version number): Linux XXXX 3.10.0-229.7.2.el7.x86_64 #1 SMP Fri May 15 21:38:46 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

• Webserver (name and version number): Server version: Ap.countrye/2.4.6 (Red Hat Enterprise Linux) Server built: Dec 2 2014 08:09:42

• Ap.countrye modules enabled as declared in: ap.countryectl -M Loaded Modules:

core_module (static)
so_module (static)
http_module (static)
access_compat_module (shared)
actions_module (shared)
alias_module (shared)
allowmethods_module (shared)
auth_basic_module (shared)
auth_digest_module (shared)
authn_anon_module (shared)
authn_core_module (shared)
authn_dbd_module (shared)
authn_dbm_module (shared)
authn_file_module (shared)
authn_soc.countrye_module (shared)
authz_core_module (shared)
authz_dbd_module (shared)
authz_dbm_module (shared)
authz_groupfile_module (shared)
authz_host_module (shared)
authz_owner_module (shared)
authz_user_module (shared)
autoindex_module (shared)
c.countrye_module (shared)
c.countrye_disk_module (shared)
data_module (shared)
dbd_module (shared)
deflate_module (shared)
dir_module (shared)
dumpio_module (shared)
.countryo_module (shared)
env_module (shared)
expires_module (shared)
ext_filter_module (shared)
filter_module (shared)
headers_module (shared)
include_module (shared)
info_module (shared)
log_config_module (shared)
logio_module (shared)
mime_magic_module (shared)
mime_module (shared)
negotiation_module (shared)
remoteip_module (shared)
reqtimeout_module (shared)
rewrite_module (shared)
setenvif_module (shared)
slotmem_plain_module (shared)
slotmem_shm_module (shared)
soc.countrye_dbm_module (shared)
soc.countrye_memc.countrye_module (shared)
soc.countrye_shmcb_module (shared)
status_module (shared)
substitute_module (shared)
suexec_module (shared)
unique_id_module (shared)
unixd_module (shared)
userdir_module (shared)
version_module (shared)
vhost_alias_module (shared)
dav_module (shared)
dav_fs_module (shared)
dav_lock_module (shared)
lua_module (shared)
mpm_prefork_module (shared)
proxy_module (shared)
lbmethod_bybusyness_module (shared)
lbmethod_byrequests_module (shared)
lbmethod_bytraffic_module (shared)
lbmethod_heartbeat_module (shared)
proxy_ajp_module (shared)
proxy_balancer_module (shared)
proxy_connect_module (shared)
proxy_express_module (shared)
proxy_fcgi_module (shared)
proxy_fdpass_module (shared)
proxy_ftp_module (shared)
proxy_http_module (shared)
proxy_scgi_module (shared)
ssl_module (shared)
systemd_module (shared)
cgi_module (shared)
php5_module (shared)

• Database (name and version number): PostgreSQL 9.2.13 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit

• PHP (version number): php -v

PHP Warning: Module 'libsmbclient' already loaded in Unknown on line 0

PHP 5.4.16 (cli) (built: Jun 10 2015 04:34:17) Copyright (c) 1997-2013 The PHP Group Zend Engine v2.4.0, Copyright (c) 1998-2013 Zend T.countrynologies

maybe this warning has to do with the problem, alltough I doubt that double loading a library can have a deeper effect.

• ownCloud Enterprise Server Version {"installed":true,"maintenance":false,"version":"8.0.5.2","versionstring":"8.0.5","edition":"Enterprise"}

• Tell us your migration path of the ownCloud software I can't remember, we started around version 7, so we upgraded from 7 to 8 some when.

• Do you have records from your migration steps? no

• List of owncloud-apps enabled/ in use (see bold printed apps in: administration ? Apps):

Activity 1.2.1
Deleted files 0.6.3
Enterprise License Key 0.1.0
External storage support 0.2.3
File Locking
File Shared access logging app 0.5
First Run Wizard 1.1
LDAP user and group backend 0.5.1
Log audit info 0.6
Mail Template Editor 0.1
Provisioning API 0.2
Share Files 0.6.2
Text Editor 0.4
Versions 1.0.6

• The content of your config/config.php:

<?php
$CONFIG = array (
'appstoreenabled' =false,
'has_internet_connection' =false,
'supportedDatabases' =>
array (
0 ='mysql',
1 ='pgsql',
2 ='oci',
3 ='mssql',
),
'instanceid' ='oc73453e92cb',
'passwordsalt' ='XXXXXXXXXXXXXXXXXXXXXXXXX',
'trusted_domains' =>
array (
0 ='cloud.DOMAIN_NAME.country',
),
'datadirectory' ='/data/owncloud',
'overwrite.cli.url' ='http://cloud.DOMAIN_NAME.country/owncloud-enterprise',
'dbtype' ='pgsql',
'version' ='8.0.5.2',
'dbname' ='owncloud',
'dbhost' ='db.as34288.net',
'dbtableprefix' ='oc_',
'dbuser' ='owncloud',
'dbpassword' ='XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
'installed' =true,
'license-key' ='XXXXXXXXXXXXXXXXXXXXXXXXXXX',
'forcessl' =true,
'theme' ='',
'maintenance' =false,
'secret' ='XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
'ldapIgnoreNamingRules' =false,
'mail_from_address' ='cloud',
'mail_smtpmode' ='smtp',
'mail_domain' ='DOMAIN_NAME.country',
'mail_smtpsecure' ='tls',
'mail_smtphost' ='xxxxxxx',
'mail_smtpport' ='25',
'mail_smtpauthtype' ='LOGIN',
'mail_smtpauth' =1,
'mail_smtpname' ='notification',
'mail_smtppassword' ='XXXXXXXXXX',
'loglevel' =2,
);

• Storage type and setup The local ownCloud storage is an xfs partition. The external storage is a Windows 2012R2 Server in an AD Domain.

• LDAP AD Windows Server 2012R2

• LDAP configuration (delete this part if not used) sudo -u www-data php occ ldap:show-config

+------------------------------+...
| Configuration                | s01
+------------------------------+...
| hasMemberOfFilterSupport     | 1
| hasPagedResultSupport        |
| homeFolderNamingRule         |
| lastJpegPhotoLookup          | 0
| ldapAgentName                |CN=owncloud-service,OU=Service,OU=Accounts,DC=earth,DC=DOMAIN_NAME,DC.country
| ldapAgentPassword            | ***
| ldapAttributesForGroupSea.xx | 
| ldapAttributesForUserSea.yyy | 
| ldapBackupHost               | 
| ldapBackupPort               | 
| ldapBase                     | DC=earth,DC=DOMAIN_NAME,DC.country 
| ldapBaseGroups               |OU=Groups,OU=Accounts,DC=earth,DC=DOMAIN_NAME,DC.country
| ldapBaseUsers                |OU=Te.countryers,OU=Accounts,DC=earth,DC=DOMAIN_NAME,DC.country;OU=Staff,OU=Accounts,DC=earth,DC=DOMAIN_NAME,DC.country;OU=Students,OU=Accounts,DC=earth,DC=DOMAIN_NAME,DC.country;OU=Special,OU=Accounts,DC=earth,DC=DOMAIN_NAME,DC.country
| ldapC.countryeTTL            | 600
| ldapConfigurationActive      | 1
| ldapEmailAttribute           | mail
| ldapExperiencedAdmin         | 0
| ldapExpertUUIDGroupAttr      | cn
| ldapExpertUUIDUserAttr       | uid
| ldapExpertUsernameAttr       |
| ldapGroupDisplayName         | cn 
| ldapGroupFilter              | (&(|(objectclass=group)))|
| ldapGroupFilterGroups        | 
| ldapGroupFilterMode          | 1 
| ldapGroupFilterObjectclass   | group 
| ldapGroupMemberAssocAttr     | member 
| ldapHost                     | ldaps://vaduz.earth.DOMAIN_NAME.country
| ldapIgnoreNamingRules        | 
| ldapLoginFilter              |(&(|(objectclass=person)(objectclass=user))(uid=%uid)) 
| ldapLoginFilterAttributes    | 
| ldapLoginFilterEmail         | 0 
| ldapLoginFilterMode          | 1 
| ldapLoginFilterUsername      | 1 
| ldapNestedGroups             | 0 
| ldapNoCase                   | 1 
| ldapOverrideMainServer       | 0 
| ldapPagingSize               | 500 
| ldapPort                     | 636 
| ldapQuotaAttribute           | 
| ldapQuotaDefault             |  
| ldapTLS                      | 0 
| ldapUserDisplayName          | displayname 
| ldapUserFilter               |(&(|(objectclass=person)(objectclass=user))) 
| ldapUserFilterGroups         | 
| ldapUserFilterMode           | 1 
| ldapUserFilterObjectclass    | person;user 
| ldapUuidGroupAttribute       | auto 
| ldapUuidUserAttribute        | auto 
| turnOffCertCheck             | 0 
+-------------------.....

• Loadbalancer no load balancer

• High availability setup no HA

• Proxies no Proxy

• Customizations none

• In the Administration panel: att.countryed

• Additional a.countryitectural features (disaster recovery setup, etc.): nothing special

[RobinMcCorkell]

@LukasReschke When the fancy new backend stuff is complete, do we still want to support 'use credentials from session, not stored in mount.json' or do we want to require that the server always has the required information to use an external storage? I'll definitely be implementing the latter, the question is whether we want the former or not alongside it. Bearing in mind that using session credentials will still cause problems as described in this issue.

[LukasReschke]

@LukasReschke When the fancy new backend stuff is complete, do we still want to support 'use credentials from session, not stored in mount.json'

I think so.

cc @MTRichards

[MTRichards]

Yes please. The session credential store is for those very security minded individuals. While sharing is then turned off for all such authenticated external storage mounts, it is one use case that quite a few folks want to use.

[RobinMcCorkell]

In that case, we need to fix the root of this issue here. It seems like the filescanner isn't run correctly via a client login? cc @icewind1991

[cdamken]

@icewind1991 Is there any extra information from the customer that I can provide you?

[icewind1991]

Scanning for changes in external storages in the background is not supported and a complex problem to solve

[cdamken]

@icewind1991 What can we do? Is there a workaround?

[icewind1991]

Nothing we can do right now, the use case is unsupported atm

[MorrisJobke]

Nothing we can do right now, the use case is unsupported atm

So more a feature request?

[cdamken]

Would it work if is set 2 here?

/**
 * Specifies how often the filesystem is checked for changes made outside
 * ownCloud.
 *
 * 0 -> Never check the filesystem for outside changes, provides a performance
 * increase when it's certain that no changes are made directly to the
 * filesystem
 *
 * 1 -> Check each file or folder at most once per request, recommended for
 * general use if outside changes might happen.
 *
 * 2 -> Check every time the filesystem is used, causes a performance hit when
 * using external storages, not recommended for regular use.
 */
'filesystem_check_changes' => 1,

I know its not recommended, but sounds better than the cron job every 5 minutes.

[MorrisJobke]

I know its not recommended, but sounds better than the cron job every 5 minutes.

I remind that this now can be set per backend, right @Xenopathic ?

[PVince81]

Yes, but it won't do the same as the cron job. The setting "2" doesn't mean "full rescan" unfortunately as @icewind1991 already stated in other tickets.

[MorrisJobke]

The customer was informed, that it is currently not possible in that way. The SF was closed. I will close this ticket here then.

@cdamken @bboule FYI

[cdamken]

The customer asked about if we already found a solution, I will organize a call with him and see if it could be better to use the WND instead.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.