Multiple file uploading and deleting is very slow

[jerrac]

Summary

This is kinda long. To summarize, OwnCloud appears to have issues when dealing with large numbers of files. Both on upload and delete. Via both the sync client and webdav. After some testing, and reading the issue queues, I'm sure it probably has something to do with how many sql queries that have to run when uploading or deleting a file.

The slowness of uploading and deleting effectively makes OwnCloud unbearble to use.

https://github.com/owncloud/mirall/issues/331 is the issue I would normally post this to, but this is core problem. Not a mirall problem. It's also focussed on the sync client.

In the core queue, I found https://github.com/owncloud/core/issues/769 and https://github.com/owncloud/core/issues/2010 but both are Mac specific.

So I don't think I'm posting a duplicate, but I'll leave that decision up to someone else. Especially if I missed a better issue to post on when I searched...

Server/Client config

My Owncloud server is running 5.0.5 on Ubuntu 12.04 with Apache 2.2.22 and PHP 5.3.10.

My desktop is running Nautilus 3.4.2 on Ubuntu 12.04.

The Problem

When uploading a 95MB folder to OwnCloud via Nautilus, the fastest speed I saw was 600ish kB/sec. Most of the time it was under 100-150ish kB/sec. The folder had hundreds of files in it. It took at least 20 minutes to upload. All according to the Nautilus file copy dialog.

Uploading a zip of the same folder was basically instant. As was any single file upload.

Can't really tell you what kind of speeds I got with the client, but it did take just as long as with webdav.

When uploading or deleting I see the server cpu fluctuate up to 60+%.

My hardware is pretty powerful, on both client and server. So, seeing that high of a load with just one person uploading is not good.

Diagnostics

When uploading, I tailed the owncloud.log and saw a lot of messages like this:

{"app":"core","message":"Adding user backend instance of OCA\\user_ldap\\USER_LDAP.","level":0,"time":1366827798}
{"app":"core","message":"Adding user backend instance of OC_User_Database.","level":0,"time":1366827799}
{"app":"core","message":"Adding user backend instance of OCA\\user_ldap\\USER_LDAP.","level":0,"time":1366827799}
{"app":"core","message":"Adding user backend instance of OC_User_Database.","level":0,"time":1366827799}
{"app":"core","message":"Adding user backend instance of OCA\\user_ldap\\USER_LDAP.","level":0,"time":1366827799}
{"app":"core","message":"Adding user backend instance of OC_User_Database.","level":0,"time":1366827799}
{"app":"core","message":"Adding user backend instance of OCA\\user_ldap\\USER_LDAP.","level":0,"time":1366827799}
{"app":"core","message":"Adding user backend instance of OC_User_Database.","level":0,"time":1366827799}
{"app":"core","message":"Adding user backend instance of OCA\\user_ldap\\USER_LDAP.","level":0,"time":1366827799}
{"app":"core","message":"Adding user backend instance of OC_User_Database.","level":0,"time":1366827799}
{"app":"core","message":"Adding user backend instance of OCA\\user_ldap\\USER_LDAP.","level":0,"time":1366827799}

What exactly does that mean?

My user is an LDAP user. And all my potential users will be authenticating via LDAP.

I also tried moving the database to localhost instead of hosting on my normal mysql server. Now I see mysqld rise to the top of htop while uploading and deleting.

I increased the mysql query_cache_limit to 8M and query_cache_size to 64M. I think it sped up a bit, but I'm not sure it's not just my imagination.

I also turned on the slow query log. I get the following when uploading a single file, and when uploading lots of files, (and tailing the log) the log appears to do the same for every single file in the multi upload.

# Time: 130424 12:23:00
# User@Host: testfilehost[testfilehost] @ localhost []
# Query_time: 0.004940  Lock_time: 0.000032 Rows_sent: 0  Rows_examined: 11827
SET timestamp=1366831380;
SELECT `path`, `fileid` FROM `oc_filecache` WHERE `path` LIKE 'files/Testing/titan profile.png.part/%';
# User@Host: testfilehost[testfilehost] @ localhost []
# Query_time: 0.000380  Lock_time: 0.000061 Rows_sent: 0  Rows_examined: 5
SET timestamp=1366831380;
SELECT `oc_share`.`id`, `item_type`, `oc_share`.`parent`, `share_type`, `share_with`, `file_source`, `path`, `permissions`, `stime`, `expiration`, `token`, `storage` FROM `oc_share` INNER JOIN `oc_filecache` ON `file_source` = `oc_filecache`.`fileid` AND `uid_owner` = 'USERUID' AND `share_type` != '2' AND ( `file_source` = '52767' OR `item_type` IN ('folder'));
# User@Host: testfilehost[testfilehost] @ localhost []
# Query_time: 0.000297  Lock_time: 0.000044 Rows_sent: 0  Rows_examined: 5
SET timestamp=1366831380;
SELECT `oc_share`.`id`, `item_type`, `oc_share`.`parent`, `share_type`, `share_with`, `file_source`, `path`, `permissions`, `stime`, `expiration`, `token`, `storage` FROM `oc_share` INNER JOIN `oc_filecache` ON `file_source` = `oc_filecache`.`fileid` AND `uid_owner` = 'USERUID' AND `share_type` != '2' AND ( `file_source` = '39286' OR `item_type` IN ('folder'));
# User@Host: testfilehost[testfilehost] @ localhost []
# Query_time: 0.000240  Lock_time: 0.000083 Rows_sent: 0  Rows_examined: 5
SET timestamp=1366831380;
SELECT `oc_share`.`id`, `item_type`, `item_source`, `item_target`,
                            `oc_share`.`parent`, `share_type`, `share_with`, `uid_owner`,
                            `file_source`, `path`, `file_target`, `permissions`, `stime`, `expiration`, `token`, `storage` FROM `oc_share` INNER JOIN `oc_filecache` ON `file_source` = `oc_filecache`.`fileid` WHERE `file_target` IS NOT NULL AND `share_type` IN ('0','1','2') AND `share_with` IN ('USERUID') AND `uid_owner` != 'USERUID';

I get the same behavior uploading from a Windows XP VirtualBox VM.

I haven't tried myself, but others in my department have reported the Mac OSX is slow. Like in the issues I linked to earlier.

Deleting the uploaded folder is just as slow as uploading.

Same with the sync client.

While deleting, the slow query log shows a lot of queries like these:

# User@Host: testfilehost[testfilehost] @ localhost []
# Query_time: 0.000172  Lock_time: 0.000059 Rows_sent: 0  Rows_examined: 5
SET timestamp=1366837346;
SELECT `oc_share`.`id`, `item_type`, `oc_share`.`parent`, `share_type`, `share_with`, `file_source`, `path`, `permissions`, `stime`, `expiration`, `token`, `storage` FROM `oc_share` INNER JOIN `oc_filecache` ON `file_source` = `oc_filecache`.`fileid` AND `uid_owner` = 'USERUID' AND `share_type` != '2' AND ( `file_source` = '55519' OR `item_type` IN ('folder'));
# User@Host: testfilehost[testfilehost] @ localhost []
# Query_time: 0.000091  Lock_time: 0.000035 Rows_sent: 1  Rows_examined: 3
SET timestamp=1366837346;
SELECT `size` FROM `oc_files_trashsize` WHERE `user`='USERUID';
# User@Host: testfilehost[testfilehost] @ localhost []
# Query_time: 0.002972  Lock_time: 0.000027 Rows_sent: 4906  Rows_examined: 4911
SET timestamp=1366837346;
SELECT `location`,`type`,`id`,`timestamp` FROM `oc_files_trash` WHERE `user`='USERUID';
# User@Host: testfilehost[testfilehost] @ localhost []
# Query_time: 0.000158  Lock_time: 0.000057 Rows_sent: 0  Rows_examined: 3
SET timestamp=1366837346;
UPDATE `oc_files_trashsize` SET `size`='976451849' WHERE `user`='USERUID';
# User@Host: testfilehost[testfilehost] @ localhost []
# Query_time: 0.000204  Lock_time: 0.000069 Rows_sent: 0  Rows_examined: 5
SET timestamp=1366837346;
SELECT `oc_share`.`id`, `item_type`, `item_source`, `item_target`,
                            `oc_share`.`parent`, `share_type`, `share_with`, `uid_owner`,
                            `file_source`, `path`, `file_target`, `permissions`, `stime`, `expiration`, `token`, `storage` FROM `oc_share` INNER JOIN `oc_filecache` ON `file_source` = `oc_filecache`.`fileid` WHERE `file_target` IS NOT NULL AND `share_type` IN ('0','1','2') AND `share_with` IN ('USERUID') AND `uid_owner` != 'USERUID';

I tried increasing my php memory to 256M and saw no difference.

My last test before posting this was to try connecting to my local install on my desktop. No network, on an SSD, 8 core cpu, 16GB of RAM, no https. Still slow. It might be faster than over the network, but not enough for me to notice... OwnCloud was running code from the master branch, as of yesterday sometime. Not 5.0.5.

Ending Thoughts

Am I right in blaming the sql queries for this? Are there some other server settings I could try?

From what I can tell, the issue is that there is a LOT of overhead to uploading or deleting a file. Thus, when you need to do that more than a couple times, it's slow. How can the overhead be reduced?

Are there some better ways I can test this than just uploading a random folder with a few thousand files in it? Ways that would provide more useful data?

[MorrisJobke]

cc @icewind1991 @bartv2 dup of #2762?

[dlangh]

I wanted to post a note to say that I too am having this problem. My problem is with the client sync app. The performance is utterly TERRIBLE, I mean beyond describing. I feel like I've transported back to the days of a 14.4k modem.

I am running Owncloud Server 5.0.5. I was previously running 4.5 and I had no problems using the sync client. Also FYI when I tried to upgrade to 5.0 it stalled and never finished upgrading (I let it run for days!). I finally gave up on that and blew away 4.5 and installed 5.0.5 from scratch, so this problem would have nothing to do with a problematic upgrade.

My problem is specifically with the owncloud client on Mac. I also upgraded this morning to 1.2.5 which I see is recently out, that did not help. It transfers at speeds in the kilobytes. Just to prove how bad this is. I was testing Bittorrent Sync this morning and while Owncloud transfered < 1MB of data, the Bittorrent Sync client transfered > 500MB. This is on the same client and server. In fact Bittorrent Sync transfered 1GB in a few minutes which is pretty much on par for my network speed.

I really want to be of a positive mind-set but this is a MAJOR problem and one I see many others are reporting. Whatever it is this should be priority one as it renders owncloud useless. I think this product has great potential, but right now it's basically of no value to me and I'm forced to find alternatives. I cannot say how dissapointed I was when I saw 1.2.5 was released and a fix for this was not part of it. anything else you guys work on should take a back seat to this until it's resolved.

[munialabs]

i second that opinion, i would like to stress that owncloud is excellent work-- but my 70000 files needed more than a week to upload in my internal network. as i only need dropbox-like syncing, i switched to seafile, which, thanks to git, shows much more performance (about 30 minutes).

again, THANK YOU for all your work!

but this bug is a show-stopper and should be handled with upmost priority.

[MorrisJobke]

What database did you use?

[munialabs]

i used sqlite. and if i remember correctly i also used mysql on the raspberry pi. but no postgresql.

[MorrisJobke]

In my case sqlite was the show-stopper. Using mysql, uploads are pretty fast.

[dlangh]

I don't remember choosing a database, so I have to assume that means sqlite. Having said that since someone else mentioned raspberry pi, where the hardware could possibly be the bottleneck (I have 4 pi's and I love them but when I tested owncloud 4.5 on one of them it was just too slow for me. In that case it was web access. My owncloud server is running on a 2.2 Ghz Mac Mini with 4GB ram, so performance should not be an issue.

Having said this I will give mysql a try and report back here my findings.

Thanks!

Don

[munialabs]

as a sidenote: i also had owncloud running on some quad-core amd, x4 631 i think, under linux mint, and the client was just the laptop next to it, connected via a 100mbit switch. the performance was not different to my pogoplug and also not different to my raspberry pi. on both the amd and the pogoplug i'm quite confident that i have chosen sqlite.

[MorrisJobke]

If just one php module is installed(sqlite, mysql, psql), owncloud automatically chooses this and no choices are listed;)

[jerrac]

I haven't tried anything but mysql.

[dlangh]

Ok, I performed the following:

1. Installed owncloud server on ubuntu server, setup apache with SSL the works. This was a completely NEW installation, no legacy files, no upgrade, no migration. Brand new Owncloud server 5.0.5. This time I used Mysql for the database backend, NOT sqlite as before. This box really isn't doing much at the moment, no other activity and it's an internal box of mine.
2. I confirmed owncloud web interface loads and works via https://
3. Opened my client app on Mac version 1.2.5. I deleted all sync folders in the client so I had none anymore. The old ones were pointing to my original owncloud server on my mac mini, so I figured best to wipe them and start clean with client as well.
4. I created a new sync client definition. it connected without issue and let me create new sync folder.
5. Sync started running.
6. In ssh terminal on ubuntu server, cd into the owncloud data folder. This folder is btw not located in the owncloud webroot, I have an external drive I am using, I don't expect that is the cause but trying to provide as much info as I can. After going into the files folder for my user I can see the new folder I synced, if I go into that I can see it's folders/files being populated.
7. Running du -c from the files folder tells me that in fact files are continuing to be populated, but again the speed is horrific. If I wait a few seconds and rerun the command, I might get 10-25K increase. It is still painfully slow. After about 10 mins and maybe 2 MB transfered, I finally aborted and gave up.

That is my experience, for me mysql was no help.

[MorrisJobke]

@dlangh Thanks for your detailed report. Here at my (virtual) machine with ubuntu 12.04 and oc 5.0.5 (upgraded from fresh 5.0.0) my 2GB documents folder (< 20k files) was uploaded in 20-30 minutes. And for me sqlite was the bottleneck.

Maybe @icewind1991 has a glue.

[dermase]

Having the same problem here. My system: Windows 7 x64 SP1 using current downloadable version of xamp I used mysql as database

Content was about 2 GB mainly pdf-documents in several sub directories. Some files were bigger. Upload to server was something around 5 files in one minute and approximately 10MB in total. Server was within local network, 100MBit connection, running also with windows 7 x64 SP1 Any more info I'll be happy to submit

[tfelice]

My experience is similar, and I'm using PostgreSQL with ownCloud 5.0.5. Uploads of a single file seem pretty normal, but whenever there are large amounts of files, it's so slow as to be unusable. I started to sync Windows server (using client 1.2.5) that had maybe 200,000 files totaling about 180 GB, to an ownCloud server to which the client machine had a gigabit Ethernet path. Even across gigabit Ethernet, the progress was so slow that it would have taken ten days to complete had I not interrupted it after two days.

The server is Ubuntu 12.04 64-bit, running PHP 5.3.10-1ubuntu3.6 and PostgreSQL 9.1+129ubuntu1.

We're interesting in providing a private backup to our customers and are interested in using ownCloud as a service provider, but obviously we won't go any further in this direction until we see our test install work well.

[ArjenR]

On my system with owncloud 5.0.5 on opensuse 12.3 with sqlite as db I notice the same issue. I use nginx with php-fpm

Since I can hear the disk activity, I have focussed in that venue to trace the issue.

I have noticed that on every PROPFIND the owncloud.db and owncloud.db-journal is openen multiple times with NOEXIST as a result given. Surprising since owncloud.db exists.

owncloud.db-journal is being created and deleted repeatedly.

opened the owncloud.db with sqlite3 and issued a PRAGMA journal_mode = WAL

This has mitigated the issue greatly, but I still notice a lot of file open commands on the owncloud data dir.

This leads me to believe that the connection to the database is opened and closed repetitively. in de lib/db.php I do notice some code which tests access to the database. I have not tersely researched the code. Well I can read it, but only with half a clue... ;-) But I think that connections to the database are not persistent and are opened and closed about 3 times for each query (on PROPFIND).

Expensive in sqlite and perhaps also on other (remote) db backends.

I hope this can help in resolving the issue.

BB, Arjen

[MorrisJobke]

@ArjenR Thanks to your great research!

[ArjenR]

@kabum

You are welcome :-)

But looking again, I am unsure on the fact that it is related to PROPFIND. (I did not look into the code) I look at log files and the strace and correlate them. So I think it is at PUT that all the repetitive database open seem to happen. As is mentioned elsewhere.

[tfelice]

I don't know if this helps at all, but I did a tcpdump on the server, which uses a local copy of PostgreSQL, to see the traffic on localhost:5432, and I see the following curious output (incorrect checksums). This is while a couple of clients are actively syncing, and I see the file count incrementing on the server, so the slow file transfers are still in progress. Doesn't seem to be any filesystem problems on the server; when I use scp I get transfer speed close to 300x what I'm getting with the ownCloud client.

IP (tos 0x0, ttl 64, id 778, offset 0, flags [DF], proto TCP (6), length 126) 127.0.0.1.41236 > 127.0.0.1.5432: Flags [P.], cksum 0xfe72 (incorrect -> 0x3830), seq 721585049:721585123, ack 4005059246, win 768, options [nop,nop,TS val 17694106 ecr 17694104], length 74 IP (tos 0x0, ttl 64, id 14240, offset 0, flags [DF], proto TCP (6), length 126) 127.0.0.1.5432 > 127.0.0.1.41236: Flags [P.], cksum 0xfe72 (incorrect -> 0x52d3), seq 1:75, ack 74, win 613, options [nop,nop,TS val 17694106 ecr 17694106], length 74 IP (tos 0x0, ttl 64, id 779, offset 0, flags [DF], proto TCP (6), length 158) 127.0.0.1.41236 > 127.0.0.1.5432: Flags [P.], cksum 0xfe92 (incorrect -> 0xc084), seq 74:180, ack 75, win 768, options [nop,nop,TS val 17694106 ecr 17694106], length 106 IP (tos 0x0, ttl 64, id 14241, offset 0, flags [DF], proto TCP (6), length 174) 127.0.0.1.5432 > 127.0.0.1.41236: Flags [P.], cksum 0xfea2 (incorrect -> 0x9ee1), seq 75:197, ack 180, win 613, options [nop,nop,TS val 17694106 ecr 17694106], length 122 IP (tos 0x0, ttl 64, id 780, offset 0, flags [DF], proto TCP (6), length 142) 127.0.0.1.41236 > 127.0.0.1.5432: Flags [P.], cksum 0xfe82 (incorrect -> 0x4d4b), seq 180:270, ack 197, win 768, options [nop,nop,TS val 17694106 ecr 17694106], length 90 IP (tos 0x0, ttl 64, id 14242, offset 0, flags [DF], proto TCP (6), length 142) 127.0.0.1.5432 > 127.0.0.1.41236: Flags [P.], cksum 0xfe82 (incorrect -> 0xc627), seq 197:287, ack 270, win 613, options [nop,nop,TS val 17694106 ecr 17694106], length 90 IP (tos 0x0, ttl 64, id 781, offset 0, flags [DF], proto TCP (6), length 158) 127.0.0.1.41236 > 127.0.0.1.5432: Flags [P.], cksum 0xfe92 (incorrect -> 0xdaba), seq 270:376, ack 287, win 768, options [nop,nop,TS val 17694106 ecr 17694106], length 106 IP (tos 0x0, ttl 64, id 14243, offset 0, flags [DF], proto TCP (6), length 126) 127.0.0.1.5432 > 127.0.0.1.41236: Flags [P.], cksum 0xfe72 (incorrect -> 0x1d7f), seq 287:361, ack 376, win 613, options [nop,nop,TS val 17694106 ecr 17694106], length 74 IP (tos 0x0, ttl 64, id 782, offset 0, flags [DF], proto TCP (6), length 126) 127.0.0.1.41236 > 127.0.0.1.5432: Flags [P.], cksum 0xfe72 (incorrect -> 0xbfb2), seq 376:450, ack 361, win 768, options [nop,nop,TS val 17694108 ecr 17694106], length 74 IP (tos 0x0, ttl 64, id 14244, offset 0, flags [DF], proto TCP (6), length 126) 127.0.0.1.5432 > 127.0.0.1.41236: Flags [P.], cksum 0xfe72 (incorrect -> 0x815a), seq 361:435, ack 450, win 613, options [nop,nop,TS val 17694108 ecr 17694108], length 74 IP (tos 0x0, ttl 64, id 783, offset 0, flags [DF], proto TCP (6), length 158) 127.0.0.1.41236 > 127.0.0.1.5432: Flags [P.], cksum 0xfe92 (incorrect -> 0x1dfc), seq 450:556, ack 435, win 768, options [nop,nop,TS val 17694108 ecr 17694108], length 106 IP (tos 0x0, ttl 64, id 14245, offset 0, flags [DF], proto TCP (6), length 174) 127.0.0.1.5432 > 127.0.0.1.41236: Flags [P.], cksum 0xfea2 (incorrect -> 0x79d8), seq 435:557, ack 556, win 613, options [nop,nop,TS val 17694108 ecr 17694108], length 122

[ArjenR]

On further investigating, my change to WAL journal for sqlite3 made little change.

the strace is still running on php-fpm. It now points me to stufft happeninging in updater.php and mimetypes.lst.php should warrant further investigation. Each time these are opened an open on owncloud datadir happens and the process stocks for a second or two.

updater.php has a function correctFolder which recursively updates some metadata on all parenfolders. This is good in code, but I guess it results in multiple selects and updates on the database. And, perhaps is also the cause for the frequent database connects/opens I observed. Since there is also an array mentioned in the code this perhaps points to code iterating over a recordset where an SQL command resolving all this in one go, is faster.

mimetypes.lst.php just defines an array of mimetypes. So I think whatever calls/includes it is another culprit.

Oh well, lot's of other stuff to do..

@tfelice imho this should not be related to the issue at hand, since this Oh well, shows osi layer 3 / kernel stuff and not on the application level?

[dlangh]

I wanted to follow up to say how I am addressing this issue for the time being in the hopes it might help others. My problem with owncloud at this point is limited to the client only. If I manually place files in my owncloud folder on my server it's working fine. I can also upload files via my phone ( I use camscanner to scan receipts and then drop them into owncloud).

So my hangup was the owncloud client and how slow it is communicating with my owncloud server. I decided to use BitTorrent Sync to sync the few folders I needed synced. I run btsync on my laptop as well as in linux where my owncloud server is. So far this has worked for me. I can now access owncloud on any device and it gets updated automatically for those folders I need to have that feature. I may even just keep it this way, since I don't need yet another cloud client running on my laptop...

[tfelice]

Not knowing the internals of the server, I had assumed that if you placed files in the data store using an out-of-band method, the database would not know about them and the server would not replicate them. So when I read dlangh's comment, I tested, and to my surprise, files I copy to the server are not only accessible by the web interface but are also replicated down to clients. The client's speed problems are still a show-stopper for us, because we need it's ability to watch folders on the client machine and propagate additions, deletions and changes to the server in a timely manner.

Can anyone enlighten me as to what other implications there may be of dropping files into the server's data store without use of the client? Obviously versioning and retention of deleted files involve database functions. What else would be broken?

[dlangh]

I have "dropped" a LOT of files at one time and it's worked pretty well. The server does not have any daemon running, so the next time you nagivate to that folder owncloud will spider the folder and index its contents. For larger data this can take a while, but even with a lot of files (one folder of mine was 3GB and definately several thousand files) it took I want to say 5 minutes or so before it finished.

I can't speak to versioning, it may well work, for me that's not an issue so I never bothered to even check...

[HugoPoi]

I want to help with this problem but i don't have enough card in my hand to resolve it. I read a lot of open issues (client and core) and I raise 2 performance problems : 1) The current release execute too many sql request when a file is added. 2) The client doesn't upload multiple files at the same time or together

We have to implement a completely new method to handle sync on server and client side. Add a piece system to handle large file. Add an other method to join small file together -> make packet and disassemble them on the server side.

[trousev]

Yes, I've faced this problem too. I'm using apache+php5+postgresql as database.

I've found a dirty solution:

in file lib/db.php delete any information about host connection (if You have localhost database). This will speed up postgresql connections.

                            case 'pgsql':
                                    if($port) {
                                            $dsn='pgsql:dbname='.$name.';host='.$host.';port='.$port;
                                    }else{
                                            //$dsn='pgsql:dbname='.$name.';host='.$host;
                                            $dsn='pgsql:dbname='.$name;
                                    }

[tytgatlieven]

Hello guys.

My observations:

1) around 600 queries are needed for a single file upload (put, proppatch and propfind), which seems a bit much imo. (I'm certainly no expert in PHP + MYSQL, so I might be wrong)

2) When I issue a few of the queries myself, I noticed that the SELECT queries on oc_filecache take around 0.5 - 1ms each, so relatively fast. However, the UPDATE statements to oc_filecache take a lot longer, namely 45ms. As there are 80 UPDATE statements during a single file upload, then this already takes around 3.6s, which is a major part of the total processing time I have noticed. Add to this value the 500 SELECT queries taking 0.75ms average and we have a MYSQL processing time of around 4s out of a total of 4.7s on my system. The remainder seems to be PHP, networking and client overhead, but is at this moment negligible imo.

I implemented a check current value before update in the update function In cache.php. as follows:

public function update($id, array $data) {
list($queryParts, $params) = $this->buildParts($data);
$params[] = $id;
$updateneededquery = \OC_DB::prepare('SELECT ' . implode(', ', $queryParts) . ' FROM `*PREFIX*filecache` WHERE `fileid` = ?');
$currentvalues = $updateneededquery->execute(array($id));

if ($row = $currentvalues->fetchRow()) {
  foreach ($data as $name => $value) {
//In buildParts they calculate the md5 to store it in path_hash and change the textual mimetype into an number
     if ($name === 'path') {
       $value = md5($value);
     } elseif ($name === 'mimetype') {
       $value = $this->getMimetypeId($value);
     }
//If a value is changed, commit the update
     if ($row[$name] != $value) {
       $query = \OC_DB::prepare('UPDATE `*PREFIX*filecache` SET ' . implode(' = ?, ', $queryParts) . '=?'
    . ' WHERE fileid = ?');
       $query->execute($params);
       break;
     }
  }
}

I checked on my setup with a flat folder containing 1000 files. test executed:

• pause client sync
• add 1000 files in a single dir (bash: for i in {1..1000}; do touch $i; done)
• resume client sync
• when sync finished, remove files
• wait until sync finished

The results are: adding the files: without checking db contents first: 1.16s / file with checking db contents: 0.837s / file

removing the files: without checking db contents first: 0.28s / file with checking db contents: 0.2s / file

or about a 40% improvement with a simple fix!

[jerrac]

As far as I can tell, no core developers have commented on this issue. @icewind1991 and @bartv2 were tagged, but never replied. So I assume this doesn't qualify as a duplicate.

We have several people willing to troubleshoot this, but no direction on what we should report that would help the core developers solve this issue. Anyone know who on https://github.com/owncloud?tab=members we could tag that would be able to give us some direction?

Would some output from xdebug help?

[tfelice]

The silence on this issue from the developers is puzzling. OwnCloud is clearly unusable for my current projects, so I've had to move on for now. I'm investigating solutions like Double-Take Availability from Vision Solutions and PeerSync from Peer Software. My current projects happen to require replicating data from Windows servers, but I continue to be interested in cross-platform solutions. I realize that replicating sources with very large numbers of files is a good trick. Even a low-end solution like BitTorrent Sync works nicely when you don't have a huge number of files.

I'm still hopeful that OwnCloud will find its way. It really has the makings of a great solution. I certainly would be open to licensing the Enterprise or perhaps the Service Provider edition once it's ready for prime time, but not when there's still a show-stopping problem in the core product. I'll provide appropriate other solutions to my customers now, and just keep checking on the progress of Owncloud every six months or so.

[trousev]

600 (!) queries!?

Yes, this DEFINETELY very much.

By the way, ownCloud supports webdav as access protocol. Is anyone know -- can it be used for sync purposes?

[jerrac]

@trousev As far as I can tell, the speed issues apply across the board. Most of the testing in my original post was over webdav. The sync client is slow, but I'm not sure how you'd upload 1000's of files through the web interface...

Haven't tried uploading a bunch of files to the owncloud data directory via rsync or sftp, but I bet it would take a while for Owncloud to find them.

@ the core developers, please don't read any of this as whining. Most of us are willing to work with you to get this issue fixed. We just don't know the code like you do, and can't do it on our own.

I really really want to use OwnCloud. It's open source, cross platform, and easy to deploy. I haven't found anything else like that.

[trousev]

@jerrac Well, I've two cases: sync for desktop (via client) and sync for mobile phone (via webdav). I'll try to make some performance comparison here.

[MTGap]

I'll look into it tonight.

[MTGap]

@tytgatlieven Open a pull request please and we'll review it.

[MTGap]

I've found some performance problems in the WebDAV implementation, no fixes yet.

Join me in #owncloud-dev tomorrow if you want to help. Logs of queries from the server would be nice. ('log_query' => true in config.php)

[tytgatlieven]

This is my first pull request ever, so please give feedback in case something is not as it is supposed to be :-)

[jerrac]

Something I just found interesting while testing Plesk. They have a webdav feature as well, and uploading lots of small files is just as slow on Plesk as it is on Owncloud.

So, could the problem be with webdav itself?

Didn't we run into speed issues with the sync clients as well?

[nc0re]

Same problem here...

I'm running the server on a BeagleBone (ARM) + ArchLinux + Apache + MariaDB (mySQL) + PHP5.4 + OC5 (all plugins disabled) + external HDD + local network.

When uploading several files via the web interface (drag and drop) things go reasonably fast, but when using the client its unusable SLOW.

I've noticed that when uploading via the web interface the server runs lots of httpd threads, but when uploading with the client only one thread is used taking up over 80% of the CPU. Is this behaviour normal (only one thread)?

I've also commented out the part Trousev mentioned in file lib/db.php, which gave a tiny performance boost, yet OC is still practically unusable.

Also I've noticed that the web interface is also slow, It takes quite a lot to enter the administrator tab, or to list my current files.

[delijati]

I have the same problem on ubuntu 13.04 and ownclound behind nginx.

[jerrac]

Anyone know how to read xdebug files? I have a few you could look at. Though, how do you attach files on github? Or at least explain what the columns on the left side bar of the attached screenshot mean. My google-fu was weak today...

[icewind1991]

• Incl.: Time spend in the function including time spend in functions called
• Self: Time spend in the function excluding time spend in functions called

Seems like the main thing to take away from that trace is the 12% time spend doing DB queries

[jerrac]

So, 49% of the time was spent calling Saber_DAV_Server->invokeMethod and all the functions that method calls. But, since self is .08, most of that time was spent in other functions. Right?

[dlangh]

I wanted to post an update on my situation. I gave up on owncloud a while back and today I decided to give it another try. With 5.0.10 released, I think I was using something like 5.0.6 when I moved on before, I thought I would give it another go. This time I completely erased my old owncloud setup, the web folder, mysql database, everything and started from scratch. I also renamed my owncloud folder on my mac client. So suffice to say I was performing a completely NEW installation.

I downloaded owncloud onto my server, setup a new empty database/user on my mysql server (which is running on the same box that is serving owncloud webpages). and did the initial setup of the webserver. No problems so far. Everything worked, I created admin acct and a user acct. I then re-connected my IOS owncloud app to my new server and performed a test upload of a PDF file. I didn't clock the speed, but it seemed fine, took about as long as I would expect. I then setup my Mac client and after it successfully connected to my new owncloud server, I dragged a folder into my local owncloud folder on my mac. The folder is 82MB in size and just over 2000 files (it's an archive file of old programming projects, so lots of small files). As I write this after 30 minutes, it's copied over just 5Mb. I don't mean to be dis-resptful, because I realize this is a free product, but can something like this seriously be ignored like this? Especially considering the debugging people have already tried to do here.

I also do not understand how more people aren't experiencing this, the developers even. What did I do wrong? It was a basic installation, I changed no default setting, just installed, added users, added clients and run into this brick wall???

Very frustrated that something this bad is being ignored like this

[nc0re]

@dlangh What hardware is your server running on? Maybe it has something to with it. I use a beaglebone as server ;) But I also tried it on a Intel Core Duo (3.4GHz) on a gigabit network and experienced the same problem. It was just frustating because I couldnt fix it whatever I did. I dont know if we are the only experiencing this problem or if the rest just simply dont mind it :(

[dlangh]

It is a cubieboard currently, though I tried it on a Mac Mini earlier on with the same result. Again, This issue is DEBILITATING and is a show-stopper. The fact as someone else mentioned that none of the core developers even responded to this makes me feel it's time to move on. Going to try seafile and see how that works, if not I'll just pay for more dropbox storage and give up on this, though I REALLY would prefer to have my files on my own servers and not on dropbox servers...

[MTGap]

2 core developers have already responded here. And no one responded about our changes to the scanner.

Don't accuse us of ignoring this.

[jerrac]

@MTGap Two things make it seem like the issue is being ignored. First, not everyone knows who the core developers are. Second, there are few posts from them in this issue, none that give us any clues on how to do further testing, or what kind of test results we could post that would be useful.

What changes to the scanner are you referring to? Would they have been present when I ran the tests a couple weeks ago on the latest code from the stable5 branch?

If not, then which version of owncloud should I test? And how should I gather useful data?

[MTGap]

Some of the changes have been backported, but not all. So, it would still be best to test against master.

You need to provide back traces, and not just screenshots of them. The query logs are also helpful.

[jerrac]

By back traces, you mean the xdebug output? I did offer them, but no one seemed interested.

How should I go about making sure that the output contains useful data? Where should I post them? They get pretty big, and I'm not sure what limits gists and pastebins have...

As for query logs, are you talking about the mysql database query logs?

Oh, and have you been able to reproduce the problem?

[MTGap]

By back traces, you mean the xdebug output? I did offer them, but no one seemed interested.

Yes. And just put them on gist.

Oh, and have you been able to reproduce the problem?

There are areas that need improvement, but nothing as far as I'm aware that should be causing the use of the word "DEBILITATING" by other users.

[jerrac]

https://gist.github.com/jerrac/2ac832fb5a06251d30c1

xdebug cachegrind from uploading 404 files in one folder, about 4.3MB, to OwnCloud. OwnCloud master branch was installed on a local Virtualbox vm running Ubuntu 12.04. DB was mysql 5.5 on the VM. It took 1-2 minutes to upload, and Nautilus reported speeds between 20-55 kB/sec. Hopefullyt his is the cachegrind file that resulted from that.

[MTGap]

That was a propfind, not an upload. And nothing seems to be taking too long. Did you notice any improvements?

Was this the sync client or a webdav client? Not that it makes much difference, but some will do more propfinds than others.