owncloud / core

:cloud: ownCloud web server core (Files, DAV, etc.)
https://owncloud.com
GNU Affero General Public License v3.0
8.4k stars 2.05k forks source link

Locks in oc_file_locks despite redis usage #30636

Closed pollopolea closed 6 years ago

pollopolea commented 6 years ago
I have two server working with owncloud and a database. I tried to configure a redis server to avoid some file locks (row contention) in the database. to test this scenario, I set up a new server with owncloud and a new server with remote redis (single master). Following the setup instructions, I changed the config. php file. I cleaned the "oc_file_locks"table, cleaned the redis server (flushdb) and restarted the web server. Unexpectedly, there were no changes in REDIS (command KEYS *) but there were new entries in the file_locks table of the database. In case there was a problem with the network server, I tested the communication with redis-cli, and everything was correct. Giving a step further I changed the config. php to use redis as memcache file again: Memcache. distributed' =>' Ooh, Memcache Redis', Memcache. local' =>' Oct' Memcache Redis, Now there was information on the Redis server (I assume it will be session information). But there were also new entries in the oc_file_locks table. My conclusions are that: my redis server works and communication is correct Redis is working to store session information. Redis is not being used for file blocking. ### Steps to reproduce 1. Configure remote Redis Server 2. Configure owncloud to use Redis filelocking [admin_manual](https://doc.owncloud.org/server/10.0/admin_manual/configuration/server/caching_configuration.htm) : "filelocking.enabled" => true, "memcache.locking" => "\OC\Memcache\Redis", "redis": { "host": "172.18.221.141", "port": 6379, "timeout": 0, "password": "***REMOVED SENSITIVE VALUE***" } 3. stop Webserver. 4. truncate table oc_filelocks 5. flush redis 6. Start Apache ### Expected behaviour Some locks or any Key in Redis server. ### Actual behaviour No data in redis Server. New entries in oc_filelocks. ### Server configuration **Operating system**: Centos 7 **Web server:** Apache 2.4 **PHP version:** 7.1.8 **ownCloud version:** (see ownCloud admin page) 10.0.6 **Updated from an older ownCloud or fresh install:** uldate from 10.0 **Where did you install ownCloud from:** tar file **Signing status (ownCloud 9.0 and above):** ``` Login as admin user into your ownCloud and access http://example.com/index.php/settings/integrity/failed paste the results into https://gist.github.com/ and puth the link here. ``` **The content of config/config.php:** { "system": { "instanceid": "ocbew9vgyenp", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "datadirectory": "\/var\/www\/html\/owncloud-data", "version": "10.0.6.1", "installed": true, "dbtype": "oci", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "dbtableprefix": "oc_", "dbtablespace": "USERS", "logtimezone": "Europe\/Madrid", "loglevel": 0, "log_type": "owncloud", "logfile": "\/var\/log\/owncloud.log", "mail_smtpmode": "smtp", "mail_smtpsecure": "ssl", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "***REMOVED SENSITIVE VALUE***", "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpauth": 1, "mail_smtpport": "465", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "filesystem_check_changes": 1, "trashbin_retention_obligation": "auto", "lost_password_link": "disabled", "ldapIgnoreNamingRules": false, "maintenance": false, "filelocking.enabled": true, "memcache.locking": "\\OC\\Memcache\\Redis", "memcache.distributed": "\\OC\\Memcache\\Redis", "memcache.local": "\\OC\\Memcache\\Redis", "redis": { "host": "172.18.221.141", "port": 6379, "timeout": 0, "password": "***REMOVED SENSITIVE VALUE***" } } } ``` Log in to the web-UI with an administrator account and click on 'admin' -> 'Generate Config Report' -> 'Download ownCloud config report' This report includes the config.php settings, the list of activated apps and other details in a well sanitized form. or If you have access to your command line run e.g.: sudo -u www-data php occ config:list system from within your ownCloud installation folder *ATTENTION:* Do not post your config.php file in public as is. Please use one of the above methods whenever possible. Both, the generated reports from the web-ui and from occ config:list consistently remove sensitive data. You still may want to review the report before sending. If done manually then it is critical for your own privacy to dilligently remove *all* host names, passwords, usernames, salts and other credentials before posting. You should assume that attackers find such information and will use them against your systems. ``` **List of activated apps:** ``` If you have access to your command line run e.g.: sudo -u www-data php occ app:list from within your ownCloud installation folder. ``` **Are you using external storage, if yes which one:** local/smb/sftp/... **Are you using encryption:** yes/no **Are you using an external user-backend, if yes which one:** LDAP/ActiveDirectory/Webdav/... #### LDAP configuration (delete this part if not used) ``` With access to your command line run e.g.: sudo -u www-data php occ ldap:show-config from within your ownCloud installation folder Without access to your command line download the data/owncloud.db to your local computer or access your SQL server remotely and run the select query: SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap'; Eventually replace sensitive data as the name/IP-address of your LDAP server or groups. ``` ### Client configuration **Browser:** **Operating system:** ### Logs #### Web server error log ``` Insert your webserver log here ``` #### ownCloud log (data/owncloud.log) ``` Insert your ownCloud log here ``` #### Browser log ``` Insert your browser log here, this could for example include: a) The javascript console log b) The network log c) ... ```
PVince81 commented 6 years ago

It is possible that you have old locks in the database.

Please put the server in maintenance mode and truncate the oc_file_locks table then try again. Your configuration looks ok.

After using the server there should be no new locks in the table.

pollopolea commented 6 years ago

As I said before, I stop web server, and truncate tables and after that i started sever again. Anyway, I just have done this steps again: 1) maintenance:mode --on 2) systemctl stop httpd24-httpd.service 3) truncate tables 4) systemctl start httpd24-httpd.service 5) maintenance:mode --off

And again there are new rows in the table "oc_file_locks"

pollopolea commented 6 years ago

0. Initial state:

Http server => stop sql tables (oc_file_locks)=> empty redis server (KEYS *) => empty

owncloud config.php

'filelocking.enabled' => true,
  'filelocking.ttl' => 3600,
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => '172.18.224.141',
    'port' => 6379,
    'timeout' => 0,
    'password' =>  **** OMMITED ****,
  ),

1. HTTP running :

Http server => running

sql tables (oc_file_locks)=> "id" "lock" "key" "ttl" 1285611 0 "files/b8babf424dc040e0028aef732c1a77b4" 1520239647 1285612 0 "files/5dcb9a052b69b48cfc03b6332df7d816" 1520239616 1285613 0 "files/04b5321444b4cc0a83e5f60354713419" 1520239616 1285614 0 "files/4d563c589303cd69713c443ba2dcf0ff" 1520239647 1285615 0 "files/6320d6eeca4a026ed9c9aad0192b8450" 1520239647

redis server (KEYS *) => empty

I stopped redis server while 1 client was syncronicing. This is the error

{"reqId":"Wpzvp50Sum@VFcHr9oW1EQAAAAE"
,"level":3
,"time":"2018-03-05T08:20:13+01:00"
,"remoteAddr": ****OMMITED***
,"user":"--"
,"app":"index"
,"method":"GET"
,"url":"\/"
,"message":"Exception: {\"Exception\":\"RedisException\"
,\"Message\":\"Redis server went away\"
,\"Code\":0
,\"Trace\":\"#0 /var/www/html/owncloud-10.0.6/lib/private/RedisFactory.php(85): Redis->auth('****OMMITED***')
#1 /var/www/html/owncloud-10.0.6/lib/private/RedisFactory.php(103): OC\RedisFactory->create()
#2 /var/www/html/owncloud-10.0.6/lib/private/Memcache/Redis.php(37): OC\RedisFactory->getInstance()
#3 /var/www/html/owncloud-10.0.6/lib/private/Memcache/Factory.php(132): OC\Memcache\Redis->__construct('dd9aabde8856943...')
#4 /var/www/html/owncloud-10.0.6/lib/private/Server.php(684): OC\Memcache\Factory->createLocking('lock')
#5 /var/www/html/owncloud-10.0.6/lib/composer/pimple/pimple/src/Pimple/Container.php(113): OC\Server->OC\{closure}(Object(OC\Server))
#6 /var/www/html/owncloud-10.0.6/lib/private/AppFramework/Utility/SimpleContainer.php(111): Pimple\Container->offsetGet('LockingProvider')
#7 /var/www/html/owncloud-10.0.6/lib/private/ServerContainer.php(87): OC\AppFramework\Utility\SimpleContainer->query('LockingProvider')
#8 /var/www/html/owncloud-10.0.6/lib/private/Server.php(1433): OC\ServerContainer->query('LockingProvider')
#9 /var/www/html/owncloud-10.0.6/lib/private/Files/View.php(118): OC\Server->getLockingProvider()
#10 /var/www/html/owncloud-10.0.6/lib/private/Server.php(163): OC\Files\View->__construct()
#11 /var/www/html/owncloud-10.0.6/lib/composer/pimple/pimple/src/Pimple/Container.php(113): OC\Server->OC\{closure}(Object(OC\Server))
#12 /var/www/html/owncloud-10.0.6/lib/private/AppFramework/Utility/SimpleContainer.php(111): Pimple\Container->offsetGet('EncryptionManag...')
#13 /var/www/html/owncloud-10.0.6/lib/private/ServerContainer.php(87): OC\AppFramework\Utility\SimpleContainer->query('EncryptionManag...')
#14 /var/www/html/owncloud-10.0.6/lib/private/Server.php(868): OC\ServerContainer->query('EncryptionManag...')
#15 /var/www/html/owncloud-10.0.6/lib/base.php(753): OC\Server->getEncryptionManager()
#16 /var/www/html/owncloud-10.0.6/lib/base.php(671): OC::registerEncryptionWrapper()
#17 /var/www/html/owncloud-10.0.6/lib/base.php(998): OC::init()
#18 /var/www/html/owncloud-10.0.6/index.php(54): require_once('/var/www/html/o...')
#19 {main}\"
,\"File\":\"/var/www/html/owncloud-10.0.6/lib/private/RedisFactory.php\"
,\"Line\":85}"}

So I understand redis server is being used to lock ("OC\Memcache\Factory->createLocking('lock')").

I truncate again tables sqltable and I started redis server again. Desktop client started syncronized again. " searching for changes"

redis server (KEYS *) => empty sql tables (oc_file_locks)=> => 89 locks

One time (and only one), there were locks in redis AND locks in the sql tables. But I'm unable to reproduce this case again.

PVince81 commented 6 years ago

I did a local test with 10.0.7 and redis 4.0.8 with PHP 7.1.14.

In my config I simply added 'memcache.locking' => '\\OC\\Memcache\\Redis', without config details. Syncing with the desktop client does not produce any entries in my oc_file_locks table. In redis I can see keys appearing and disappearing during the sync process by repeatedly running keys * during the syncing process.

Not sure what the problem is on your side. At least the current implementation seems to be working here.

Note that in oc_file_locks the lock entries are kept even when not locked, this is an optimization to avoid too many deletions/insertions in the locks table. The lock entries are reused. However with redis as soon as the lock is released the entry/key is removed, which might explain why you did not see any.

pollopolea commented 6 years ago

Maybe I am wrong, but I'm starting to think that owncloud is using DB table to lock something , and Redis to lock uploaded files.

Test 1:

Test 3: Close client, Open client. Force syncronization

Test 4: Running client, create 15 folder.

Test 5: Running client, create 15 folder, whith 2 files insides (1.8GB).

Test 7: (Cron disabled). No new operation with files or folder. Some minutes later:

Test 8: 500 small file (< 1MB), totalzise 378MB While uploading:

(Cron disabled). No new operation with foles or folder. 1h later:

Conclusion:

My question: ¿Why are there locks in the sq ltables? From my point of view , this is some kind of bug.

pollopolea commented 6 years ago

I suppose I can close this bugs, and maybe open another about filelocks in sql tables while using redis.

PVince81 commented 6 years ago

I've renamed this ticket, let's reuse it for the problem you observe.

When memcache (redis) is configured for locking, this is the code that decides whether to use DB or Memcache (Redis): https://github.com/owncloud/core/blob/v10.0.7/lib/private/Server.php#L685

and here the MemcacheFactory would set the $lockingCacheClass to the value set in config.php: https://github.com/owncloud/core/blob/v10.0.7/lib/private/Memcache/Factory.php#L115

The only way that would make it fall back to the database would be if somehow the config or class availability is instable.

Are you using a clustered environment with multiple nodes ?

pollopolea commented 6 years ago

In this case, I created a test environment with one web server node and another node running only redis. The only web server node is working as single-instance more (default mode in config.php)

tomneedham commented 6 years ago

@pollopolea can you try running occ config:system:get memcache.locking and let us know the output please

pollopolea commented 6 years ago

Hi, tomneedham this is the output:

$occ config:system:get memcache.locking
\OC\Memcache\Redis
pollopolea commented 6 years ago

Hi again, how can I get the path of files in the lock table? I tried some sql sentences without success:

SQL table:
id | lock | key | ttl
1502881 0   files/3872e0c32f4ab290b71be8c3256f832f  1521105952

REDIS:
3736505ae3f7ce31dff8eff686eb0f71/lockfiles/b600b18dc28d6c1e1af7a439254427b5

select * from oc_filecache where path_hash = '3872e0c32f4ab290b71be8c3256f832f'; 
select * from oc_filecache where checksum like '%3872e0c32f4ab290b71be8c3256f832f%'; -
pollopolea commented 6 years ago

After installing new server using sqlite, mysql and oci, y finally detect the problem. Redis and owncloud are working fine. The problem was my configuration, although the owncloud node was configured to use a different DB (in a different server, with different ip), in fact that DB is the same that is being used with other 2 nodes without redis.

The result is that the source of the locks are nodes without redis.

lock[bot] commented 5 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.