Upload of many Files really slow

[myzinsky]

Expected behaviour

Upload of a lot of MP3 Files (4GB) should be fast.

Actual behaviour

Upload Only 6 Files (ca 40mb) in 4 hours!

Server configuration

Operating system: Debian Stable

Web server: Apache2

Database: Mysql

PHP version: PHP Version 5.3.3-7+squeeze14

ownCloud version: 4.5.6

Client configuration

Client version: 1.2

Operating system: Debian Unstable

OS language: German

Installation path of client: /usr/bin/owncloud

Logs

ownclouds://xxx/Blue Moon 02-12 14:18:20:533 oc_module: owncloud_stat ownclouds://xxx/Blue Moon called 02-12 14:18:20:678 oc_module: Simple propfind result code 207. 02-12 14:18:20:678 oc_module: Server Date from HTTP header value: Tue, 12 Feb 2013 13:18:20 GMT 02-12 14:18:20:679 oc_module: Ok: Time delta remained (almost) the same: 0. 02-12 14:18:20:679 oc_module: => Errno after fetch resource list for ownclouds://xxx/Blue Moon: 0 02-12 14:18:20:679 oc_module: Working on file Blue Moon 02-12 14:18:20:679 oc_module: :> Subtracting 0 from modtime 1360675100 02-12 14:18:20:679 oc_module: STAT result from propfind: Blue Moon, mtime: 1360675100 02-12 14:18:20:679 oc_module: Directory of file to open exists. 02-12 14:18:20:679 oc_module: PUT request on //xxx/Blue%20Moon/Robben%20Ford%20-%20My%20Everything.mp3! 02-12 14:18:20:679 oc_module: Sendfile handling request type PUT. 02-12 14:18:20:679 oc_module: Put file size: 6858752, variable sizeof: 8

This is an example, it waits a really loooong time :( (Paths shortened with xxx)

Web server error log

Deprecated: Directive 'register_long_arrays' is deprecated in PHP 5.3 and greater in Unknown on line 0

There is a lot of this stuff, but i'm not shure if this is realted to owncloud

ownCloud log (data/owncloud.log)

{"app":"media","message":"error reading title tag in '\/var\/www\/owncloud\/xxx.mp3'","level":2,"time":1360575380}### Expected behaviour Upload of a lot of MP3 Files (4GB) should be fast.

[danimo]

Please provide details about your network setup (speed/hubs/switches/wifi).

[myzinsky]

I tried it from my home, where i have a standard dsl connection. And from University where I'm directly connected to DFN (http://www.dfn.de/fileadmin/1Dienstleistungen/XWIN/Netzentwurf_X-WiN.pdf)

At home I have a Fritzbox as router and I use Wifi there. At university im behind at least two switches and a firewall and I use a cable connection.

[danimo]

Are you seeing the same data rate regardless of the connection?

[myzinsky]

From home it was actually a little bit faster. bit still very slow... someone in the IRC channel suggested that this behavior could be due to a php time-out bug in 4.5.x.

[dragotin]

On 12.02.2013 21:51, myzinsky wrote:

From home it was actually a little bit faster. bit still very slow... someone in the IRC channel suggested that this behavior could be due to a php time-out bug in 4.5.x.

Hmm. I am not sure.

I think we could do two things:

1. We could use the new upload progress API in csync to write a specifc performance log file that gives us concrete data.
2. based on that we need to investigate where speed loss happens, ie. in client, on network transport or on the server side.

The client does a lot of requests in a row. Is it be possible that this is missunderstood as DOS attack by any part of the network infrastructure and because of that speed limited? I don't know.

[myzinsky]

sounds like a plan, what to do?

[dragotin]

@myzinsky: Maybe you want to try to capture some http traffic that can be investigated. http://justniffer.sourceforge.net/ might be an interesting tool for that.

[myzinsky]

i tried the tool, and i tried wireshark! But I dont see anything because its ssl encrypted!

[myzinsky]

Every 20s one data packet is transmitted, and sometimes i got this:

8230 365.037659000 172.xx.xx.xx 62.xx.xx.xx ICMP 590 Destination unreachable (Port unreachable)

Maybe that is the problem

[myzinsky]

So I think the problem is not realted to mirral, it is something with my connection or the apache server. If i upload files from KDE's dolphin to ownlcoud its also very very slow!

[myzinsky]

i tried uploading 200 mbs with my mac fom the university, its running very very fast only some seconds! So the problem is not the server, its mit linux laptop! Is mirral using the same library for webdav access as kde's dolphin?

[danimo]

@myzinsky No, different implementations. Port unreachable is odd. Is it possible that apache runs out of available workers or that there is some kind of rate-limiting firewall in place? It could be that those behave different depending on the implementation.

[mkudro]

I guys, i will jump into conversation since i do have similar issues as myzinsky. If i try to upload large amounts of files..well..it's just impossible. I have tried to sync my pictures directory +/- 30Gb large, containing some 20000 photos. Sync-client (linux version 1.2) couldn't handle it - it seemed that sync was in progress, but progress was excruciatingly slow, some 2Gigs in 8 hours. And since my ISP was having some network quality issues for awhile (stability, millisecond drops...i think they are heavily using deep package inspections on their clients :) ) I naturally assumed it was network problem. Next day i went to my father house where my server is stationed (he has fiberglass 100MB up/down connection) and tried to sync files through LAN - same effect. It seems impossible for client to sync large amounts of files at once. I did synced documents directory (some 100MB +/- 300 files) and it was done within a minute or so, i think less. And i do not think it's a apache issue either, when i rsynced (photos) files to server, then refreshed my owncloud "files" page whole pictures directory (30GB) was parsed by apache within couple of minutes. Their are one other things could be relevant to this. Whenever i start sync client on my desktop http daemon throws following errors in the endless (and constant) loop: PROPFIND /files/webdav.php/my_sync_folder HTTP/1.1" 401 291 PROPFIND /remote.php/webdav/my_sync_folder HTTP/1.1" 207 13797 Which is very strange, considering that authentication went well - sync is started, i see files showing up - very slowly but it shouldn't through 401 errors at me. And i know that version 1.2 (which i'm using) should talk to server through remote.php and not webdav. Same errors are generated when akonadi owncloud service is configured to sync calendar and contacts with KDE desktop.

PS some version facts of applications i'm using: Linux (Fedora 17) with kernel 3.7.9-101 KDE 4.9.5 Sync-client/mirall 1.2 csync 0.7

[Deltachaos]

Same for me

[tytgatlieven]

Hi guys

I'm having the same issue. The owncloud client log looks like this:

03-14 13:10:20:519 oc_module: MKdir on /owncloud/remote.php/webdav/Documents/PhD/documentation/
03-14 13:10:21:055 oc_module: => open called for owncloud://tv/owncloud/remote.php/webdav/Documents/PhD/documentation/Modeling Cognitive Radio to Improve the Performance of Communications Part3.pdf
03-14 13:10:21:055 oc_module: Stating directory owncloud://tv/owncloud/remote.php/webdav/Documents/PhD/documentation
03-14 13:10:21:055 oc_module: owncloud_stat owncloud://tv/owncloud/remote.php/webdav/Documents/PhD/documentation called
03-14 13:10:21:236 oc_module: Simple propfind result code 207.
03-14 13:10:21:236 oc_module: Server Date from HTTP header value: Thu, 14 Mar 2013 12:10:21 GMT
03-14 13:10:21:236 oc_module: Ok: Time delta remained (almost) the same: 0.
03-14 13:10:21:236 oc_module: => Errno after fetch resource list for owncloud://tv/owncloud/remote.php/webdav/Documents/PhD/documentation: 0
03-14 13:10:21:236 oc_module: Working on file documentation
03-14 13:10:21:236 oc_module:   :> Subtracting 0 from modtime 1363263020
03-14 13:10:21:236 oc_module: STAT result from propfind: documentation, mtime: 1363263020
03-14 13:10:21:236 oc_module: Directory of file to open exists.
03-14 13:10:21:236 oc_module: PUT request on /owncloud/remote.php/webdav/Documents/PhD/documentation/Modeling%20Cognitive%20Radio%20to%20Improve%20the%20Performance%20of%20Communications%20Part3.pdf!
03-14 13:10:21:236 oc_module: Sendfile handling request type PUT.
03-14 13:10:21:236 oc_module: Put file size: 158574, variable sizeof: 8
03-14 13:10:22:861 oc_module: http request all cool, result code 201
03-14 13:10:22:862 oc_module: Add a time delta to modtime 1269854080: 0
03-14 13:10:22:862 oc_module: Setting LastModified of /owncloud/remote.php/webdav/Documents/PhD/documentation/Modeling%20Cognitive%20Radio%20to%20Improve%20the%20Performance%20of%20Communications%20Part3.pdf to 1269854080
03-14 13:10:24:512 oc_module: owncloud_stat owncloud://tv/owncloud/remote.php/webdav/Documents/PhD/documentation/Modeling Cognitive Radio to Improve the Performance of Communications Part3.pdf called
03-14 13:10:25:283 oc_module: Simple propfind result code 207.
03-14 13:10:25:283 oc_module: Server Date from HTTP header value: Thu, 14 Mar 2013 12:10:24 GMT
03-14 13:10:25:283 oc_module: Ok: Time delta remained (almost) the same: -1.
03-14 13:10:25:283 oc_module: => Errno after fetch resource list for owncloud://tv/owncloud/remote.php/webdav/Documents/PhD/documentation/Modeling Cognitive Radio to Improve the Performance of Communications Part3.pdf: 0
03-14 13:10:25:283 oc_module: Working on file Modeling Cognitive Radio to Improve the Performance of Communications Part3.pdf
03-14 13:10:25:283 oc_module:   :> Subtracting -1 from modtime 1269854080
03-14 13:10:25:283 oc_module: STAT result from propfind: Modeling Cognitive Radio to Improve the Performance of Communications Part3.pdf, mtime: 1269854081
03-14 13:10:25:283 oc_module: Get file ID for owncloud://tv/owncloud/remote.php/webdav/Documents/PhD/documentation/Modeling Cognitive Radio to Improve the Performance of Communications Part3.pdf: 5141be301cd00
03-14 13:10:25:283 _get_md5: MD5 for owncloud://tv/owncloud/remote.php/webdav/Documents/PhD/documentation/Modeling Cognitive Radio to Improve the Performance of Communications Part3.pdf: 5141be301cd00
03-14 13:10:25:283 _csync_push_file: FINAL MD5: 5141be301cd00
03-14 13:10:25:283 _csync_push_file: PUSHED  file: owncloud://tv/owncloud/remote.php/webdav/Documents/PhD/documentation/Modeling Cognitive Radio to Improve the Performance of Communications Part3.pdf
03-14 13:10:25:283 _store_id_update: SYNCED remember  dir: PhD/documentation/Modeling Cognitive Radio to Improve the Performance of Communications Part3.pdf

So basically the slow syncing originates from: 1) The PUT of the file: 158K in 1.6s 2) Setting LastModified: 1.65s 3) Owncloud_stat: 0.7s

Thus a total processing time of around 4s.

My setup: 100Mbps link between server and pc. No DNS resolution problems (0.4m average ping time both sides) server: dual core amd 5050e, ubuntu raring amd64, MYSQL owncloud backend client: owncloud 1.2.1

I have noticed a relatively high load on the mysql process on the server. Perhaps it is a slow query for the lastmodified and owncloud_stat calls?

kind regards

Lieven

[MTRichards]

@tytgatlieven Have you tried to create an index on the database table oc_properties? Reports of performance increases when this is done.

alter table oc_properties add index(userid); alter table oc_properties add index(propertypath); alter table oc_properties add index(propertyname);

[tytgatlieven]

Hi MTRichards

I have executed your queries. The log:

03-14 13:41:48:545 oc_module: PUT request on /owncloud/remote.php/webdav/Documents/CaTy/code/sensor/codesourcery/share/doc/arm-arm-none-eabi/html/gccint/Fragments.html!
03-14 13:41:48:545 oc_module: Sendfile handling request type PUT.
03-14 13:41:48:545 oc_module: Put file size: 4336, variable sizeof: 8
03-14 13:41:51:702 oc_module: http request all cool, result code 201
03-14 13:41:51:702 oc_module: Add a time delta to modtime 1313994618: 0
03-14 13:41:51:702 oc_module: Setting LastModified of /owncloud/remote.php/webdav/Documents/CaTy/code/sensor/codesourcery/share/doc/arm-arm-none-eabi/html/gccint/Fragments.html to 1313994618
03-14 13:41:55:344 oc_module: owncloud_stat owncloud://tv/owncloud/remote.php/webdav/Documents/CaTy/code/sensor/codesourcery/share/doc/arm-arm-none-eabi/html/gccint/Fragments.html called
03-14 13:41:57:565 oc_module: Simple propfind result code 207.
03-14 13:41:57:565 oc_module: Server Date from HTTP header value: Thu, 14 Mar 2013 12:41:55 GMT
03-14 13:41:57:565 oc_module: Ok: Time delta remained (almost) the same: -2.

So it seems this still takes long. Is the add index executed immediately, or should I wait for the index to be built in background and then review the values?

Forgot to mention: I'm on owncloud 5.0 rc2

[MTRichards]

Hi! How big is your installation? I have been a lot faster than you seem to be, I think - do you have a lot of files or folders in your root ownCloud folder on the desktop?

@bartv2 which tables are queried in an eTag request? Any other tables touched when the server is asked for an eTag?

[tytgatlieven]

Hi

I'm currently syncing some 75GB of files in something around 350000 files

Owncloud web interface mentions atm: 825.7 MB

Server has 8GB memory, about 1GB used, almost solely running apache and mysql atm.

[bartv2]

@MTRichards for etag the filecache table is queried, this has been changed in OC5. @icewind1991 do you have more information?

[tytgatlieven]

Hi all

I've been doing som performance verification, and these are my conclusions/thoughts up to now:

1) The mysql queries themselves, called from core: lib/files/cache/cache.php are all pretty fast (around 1ms for the get and getFolder calls), so this does not explain the large delays

2) I assume the client side does not introduce large delays.

3) So the delay should be inside the php code itself

So I looked into speedup of php, and found some info here: http://docs.joomla.org/How_do_you_tune_for_speed_with_PHP5_and_MySQL5%3F

I did step 1: php cache thorugh APC, howto can be found here: http://www.howtoforge.com/apc-php5-apache2-debian-etch

measured the delays again, and noticed significant improvements:

1) The PUT of the file: avg around 4s, independent of file size 2) Setting LastModified: 0.5s 3) Owncloud_stat: 60ms

The load on apache has dropped significantly due to the caching mechanism. However, the put files request is still very long, and I assume this can be due to the mysql queries, in which I did not yet check the time needed for a file put.

BTW: I currently have indexes as follows: oc_filecache: path_hash, etag, fileid oc_permissions: fileid

So I would at least recommend to enable APC, since this already gives a nice performance boost.

Hope this helps you guys forwards somewhat

[tytgatlieven]

Hi all

Did some more logging / analyzing of what is going on. You can find the mysql log and the client log here: https://gist.github.com/tytgatlieven/5183718

My observations:

1) around 600 queries are needed for a single file upload (put, proppatch and propfind), which seems a bit much imo. (I'm certainly no expert in PHP + MYSQL, so I might be wrong)

2) When I issue a few of the queries myself, I noticed that the SELECT queries on oc_filecache take around 0.5 - 1ms each, so relatively fast. However, the UPDATE statements to oc_filecache take a lot longer, namely 45ms. As there are 80 UPDATE statements during a single file upload, then this already takes around 3.6s, which is a major part of the total processing time I have noticed. Add to this value the 500 SELECT queries taking 0.75ms average and we have a MYSQL processing time of around 4s out of a total of 4.7s on my system. The remainder seems to be PHP, networking and client overhead, but is at this moment negligible imo.

So my advise for increased owncloud client performance is the following (on top of enabling APC): 1) Reduce the amount of UPDATE statements issued, or significantly improve their performance. (Indexing reduces the performance of these statements I believe)

2) Reduce the amount of SELECT statements only when 1) is improved

I won't have as much time in the coming weeks as I have put in in the last few days, but I'm certainly willing to test updated code or give additional information.

@bartv2, @MTRichards, @icewind1991 : I believe you guys know a lot more about this than I do. Especially since this turned out to be a core problem, and not a client problem. I looked into the core code somewhat, but it takes me way to long to grasp everything since I'm no php expert. I hope someone of you could bring this to the attention of the core developers. I'm certainly willing to help testing, but I'm not able to make sufficient time to write code myself.

Hope this is of help

Lieven

[mkudro]

Lieven, thnx for you detailed explanation and your time you have put in this ticket...makes things bit clearer now. Hopefully devs look in to this soon, it's hard to believe that we are the only ones having problem with this. Are we only ones who trying upload/sync large amount of files with owncloud server? I really wondering what are experiences of others who try to use owncloud to full extend.

[tafinho]

Hi,

You're not certainly the only one seeing this behavior. On my test case, there is around 200.000 files weighting around 300GB. The last time I tested, it took around 3 full days to synchronize. Although this is far from optimal, I'm more concerned with possible data loss than performance at this point. There still exist some data loss bugs withstanding. After those are solved, people can take care of performance, IMHO.

No dia Segunda-feira, 18 de Março de 2013, mkudronotifications@github.comescreveu:

Lieven, thnx for you detailed explanation and your time you have put in this ticket...makes things bit clearer now. Hopefully devs look in to this soon, it's hard to believe that we are the only ones having problem with this. Are we only ones who trying upload/sync large amount of files with owncloud server? I really wondering what are experiences of others who try to use owncloud to full extend.

— Reply to this email directly or view it on GitHubhttps://github.com/owncloud/mirall/issues/331#issuecomment-15043207 .

[happyreacer]

I use OC on a NAS. How I like to monitor the harware and see the following when copying a large file. In a WEBDAV connection in a directory on the NAS and connecting via the OC diret remoute.php LAN is busy. When copying many small files, I see a big difference. Directly over the LAN utilization WEBDAV is also at maximum but the OC remoute.php the NAS processor goes to its maximum and it will be 4-6 apache services launched parallel. The transmission is then in the b / s range. No difference whether Mirall and dolphin. I think it's a problem of remote.php?!

ownCloud 5.0.3 PHP 5.3.14 MySQL 5.1.36 apache 2 only LAN-connection More questions?

[munialabs]

ok, i had also extremely slow speed with my setup, tested with three different computers on two different local networks (i only do local stuff). all configurations were running on a debian/mint server, and when switchin to archlinux, the problem was gone. so in case you have a raspberry pi, try loading the debian version (current one mid march i think), install apache via apt-get, d/l owncloud and test speeds, i believe its reproducible. the issue creator also had debian installed, thats maybe a hint. i think there is something in the debian/mint apache default configuration that makes things incredible slow (10kb/s max). as said, configuring archlinux with owncloud on the very same network, very same clients, very same hardware did produce normal results.

[munialabs]

sorry, i think i have to take that last comment back: it is true that bigger files (images of 2-3 mb each) did sync very well, but as soon as i come back to my usual data (7507 directories, 92250 files in roughly 11 GB) it seems to be slow again. is it the raspberry pi not providing enough performance??

[tytgatlieven]

@munialabs : It's indeed the case. In essence it comes down to this: The file upload itself is relatively fast, even on a slower device. However, There is a lot of processing overhead per file. Thus a 1M or a 10M file take about equal time to be processed in a local network.

@happyreacer: Its indeed due to the php code, but not in remote.php. I only did a quick check, but I believe it is due to the underlying sabredav. Again, not the transfer of the file takes time, it is the updating of the file information in the SQL database which takes time. So it's most likely not sabredav itself, but the way the link is made to update the state information. It does to many requests for identical queries to the database. (See my earlier comment including the mysql logfile for a single file upload by a client)

[bartv2]

@tytgatlieven you have a couple of good points, i looked at your sql, owncloud/core#2747 will reduce the select queries quiet a bit, and you're right reducing the update queries will also make it faster. Could you create issues with your suggestions in the core repo? And also for the documentation?

[that0n3guy]

An easy way to duplicate this:

• download http://ftp.drupal.org/files/projects/drupal-7.22.zip (its an opensource CMS drupal 7.22 from http://drupal.org/project/drupal)
  • extract that zip in your sync folder on your desktop
  • it will take forever to upload since that zip has a ton of files in it.... but is less than 20mb.

Compare it to any 15mb single file (the extracted folder will be 15-ish mb I think) upload. The 15 mb file is way way faster.

On my 100mb network, the 15 mb file takes <1 second... the extracted folder (from the above zip) takes over 30 min.

[Narkhelek]

Awesome work tracking down where the performance hits are. This issue is probably my biggest pain point with OC. I can sync gigabyte files very quick but trying to sync folders with large number of files slows syncing down to under 50kBs.

[wontolla]

I'd find similar error, on 5.0.4 and 1.2.4 client, 2 folders sync:

1- 780 MB, only a few folders, took around 10 mins over FTTH 2- 900 MB, lots of small files, thumbs, more than 4 days later still data missing.

I'd found the same error over 4.5.5, 5.0.3 and 1.2.1 & 1.2.2

[munialabs]

now i am puzzled-- is there any ownCloud version not with this bug? i know i did trashed some old version of ownCloud because of that, back then i thought my pogoplug hardware was simply not good enough. but seeing that this problem is more like a software bug and still persists is very bad :(

[wontolla]

I wish I know, it really pisses me and is stopping for use owncloud at work... man we are being drained with dropbox!!!!

[dragotin]

@wontolla, @munialabs we now all have understood that its bad and how much you dislike that.

How about sitting down and analyzing what the problem is? I can hint you: Set up a mysql test installation of ownCloud and analyze the database activity that go through the server for an ordinary PUT. Do that for shared and not shared folder, and come back with results.

THAT would indeed help very much.

[wontolla]

All can see is the client is very slow handling the files one by one, I could understand it if it was synchronizing files but it was the very first!! it should copy them real fast, the server folder is empty!!! the logs simply show the typical handle file, I'm not MySQL expert so I'll need some help there.

[butonic]

In case you activated the full text search, please deactivate the app and see if that helps.

[wontolla]

I'll try tonight, I don't enable it maybe is enabled by default.

[saulojg]

I have the same performance problems using the default Ubuntu 12.10 install which seems to use sqlite3. Maybe the problem isn't MySQL specific and has to do more with the amount of sql statements triggered by each PUT request, as shown in this post https://github.com/owncloud/mirall/issues/331#issuecomment-15032047. Looks like some sort of caching mechanism could be implemented in order to avoid identical queries to the database, perhaps something like the Cake PHP implementation http://book.cakephp.org/2.0/en/core-libraries/caching.html#Cache::read.

Is anyone working on a fix? If not I'd gladly attempt to provide one, sometime next week

On 18/04/2013 01:04 p.m., wontolla wrote:

I'll try tonight, I don't enable it maybe is enabled by default.

— Reply to this email directly or view it on GitHub https://github.com/owncloud/mirall/issues/331#issuecomment-16585817.

[dragotin]

@mand1nga no, I did not mean its mysql specific. I agree with what you say.

I don't think somebody is focused working on that atm, so if you take that up, great! But I would recommend to first analyze a bit and than send a proposal for a solution to the mailinglist to get peoples buy in.

[Narkhelek]

Deactivating the full text search didn't make any difference for me. I get the same transfer times regardless if it's enabled or not.

[wontolla]

I'd deactivated full text search and lot of apps more, mostly viewers, and still ultra-slow. Any idea?.

[icewind1991]

@wontolla @Narkhelek @mand1nga can you try applying both https://github.com/owncloud/core/pull/3021 and https://github.com/owncloud/core/pull/3025 and see if it makes a difference

[Narkhelek]

owncloud/core#3025 does NOT make any difference performance wise syncing from client to server. I was nervous to try owncloud/core#3021 as the comments seemed to indicate that it could cause issues with the DB.

[wontolla]

Updated to 5.0.5 still slow, does anybody know if commercial version has the same bug?.

[ghost]

hi guys,

i have a raspberry pi with debian wheezy installed 3.6.11 and owncloud 5.5,

using lighttpd 1.4.3 as webserver (also tried apache2), just switched to mysql to try if it makes the difference.

still it is faaar to slow to be usable,

i am synching a 19 mb folder with savegames right now, but it takes like 1 hour to sync ... which makes it unusable .. i also synched 2 albums, which was actually faster -.- (like 5 minutes or so)

anybody fixed it yet?

I tested it with mirall client on arch and with the windows client, no difference thouh, really seems to be a core problem. anybody else tried with arch? the problem seems to be gone with arch, which is weird ...

[wontolla]

I really wish it has been fixed, I'm stuck for more than a month in a very basic trouble. I'll start to search for another software soon if it keeps unsolved.

[Deltachaos]

@wontolla seefile.com could be another software ;)

[wontolla]

Thanx I'll try that and BitTorrent Sync

[dragotin]

@wontolla, @Deltachaos I would appreciate if you could move your "ownCloud alternatives" discussion to another forum, as it annoys me to read that when I read through bug reports to actually fix bugs. Thanks!