Open arkascha opened 6 years ago
Hi,
I'm a new Nextcloud user, i did my first install this morning, and i imported 99GB of photos into it (by using occ files:scan
on files i manually moved into the correct directory on my server). My first entrance in the Gallery app was a pain of course (all CPUs to 100% for like 10 minutes and spinners everywhere), so i came here to post a feature request about an occ
command to generate thumbnails. But i saw this issue, and it got me worried: would every usage of this app be that painful?
And the answer is: no :relieved:
My config:
I can see the cached thumbnails under Nextcloud data directory, in the appdata/preview
subfolder. Maybe you should check if you have enough disk space in there.
I hope my feedback can help.
@David-Guillot Thanks for the comment and suggestion, unfortunately that does not address the issues I reported. There is enough disk space to store thumbnails and previews. And I also do see the files getting created. However as already said that does not help. The speed up impression I also get is client side caching. The moment I try to access the same images from another system or using another browser I again have a painfully slow gallery, unusable.
Ah sorry i forgot to tell you that i actually tested with multiple browsers, private sessions etc. The thumbnails are cached on the server, i'm 100% positive about it.
I absolutely agree with @arkascha . A couple of days ago I tried to show some holiday pics to my family and thought using nextcloud gallery would be the easiest thing. But what a disapointment! Spinner covered the screen about 50% of the time - who is willing to follow such a lousy presentation?
I tried to dig into the code and found that nextcloud is doing most of the things right:
Finally I deleted all the previews from the disc in /nextcloud_data_dir/app_serverid/preview and started the preview generator over again. This seems to work now and loading of images takes only a 500ms compared to ~10s before.
Only it takes several days to generate all the previews for my 100GB of images. I tried to find out why this is so slow. As far as I could dig into the structure function "imagecopyresampled" is called.That should be pretty standard and in a single script it takes 2s on my server to resample a DSLR camera image. That should be fast enough for the preview-spinner never to appear in the first place, even without any pre-generation. But the gallery is generating 22 different sizes of preview images. So it will keep my server busy for 6 days.
To make things worse, those preview need 7.4MB of disc space. The original image has 10MB. So in the end: 2x disc space and days of preparation to speed up a preview that could run fast enough...
Seriously???
@Gatak Thanks for you comment, let me add a few remarks to that:
It is possible to limit what sizes that should be pre-generated. < If I read the changelog, then the size strategy of the preview generation has specifically been altered such that this should NOT be required any more it is possible to set a maximum preview size. I have mine at 1024 pixels. < I did that, even down to smaller values. Sure it gets a bit faster, but still not really usable. it is possible to set a lower jpeg quality setting to save space. < I already limit myself to upload only seriously downgraded sizes and qualities mysql/mariadb is seriously I/O bound. Use a SSD disk. use a data journalling filsystem with double write off use php-fpm with large memory cache and many children. increase caching in mysql a lot! < you ARE aware that most users have to stick with what some hosting provider offers?
In general I expect from such an app to be more or less usable in a typical standard environment. Because that is what it targets and what is usually available. If the gallery app can only be used when served from a high performance system that has been specifically crafted for it - then it has failed in my eyes.
Nevertheless: I certainly will go through my setups once more considering your detailed explanations. So thanks for that, your input it very welcome!
I just found out that a huge amount of file locks (12.362 in my case) is created in the database, when one slideshow image is requested. Can anybody with deeper nextcloud knowledge help/comment on that?
I'm also having trouble with this... I have never been able to see all the pics in my gallery collection, as opening a 3000 pics directory in an android phone or tablet, either with firefox or chrome, is impossible. Both chrome and firefox crash after the gallery app "loaded" just a few hundred pics. And the tablet is a shield k1, so it's not that bad... there must be something wrong with the app itself. Using my laptop it just takes so long to load the pics that eventually I just give up. Thank you for your work.
I noticed too that the nextcloud gallery app is too slow on my older qnap. Each preview icon, for example, takes around >1053 milliseconds (thus > 1 sec) to load.
I tried to find out, where the time is spend. My idea was, maybe, there is a single step, which takes too long, and which can be optimized alone, and then everything is fast again. This turned out, not to be true.
Below is my analysis of the calls to mariadb.
<tl;dnr> Result: Serving files from a precalualted cache should be fast. The whole preparation steps (see below), auth checking, loading all the php files, in order to send out a 1kb preview image, is too much overhead.
Every mariadb call, in its self, is fast, but there are too much of them, and as a sum, it become too slow.
Setup for Benchmark The preview icons are precalculated, with the previewgenerator app. Thus the test only measures the time to send a static image in nextcloud.
If I put the static image (32x32 pixel) into the apache root I can download it with wget in
a) 51 millisecond (apache, static image) b) 1053 milliseconds (over php-fpm/nextcloud framework)
Thus a folder with 20 preview icons takes here 20 seconds to load with the gallery/nextcloud, compared to 1 second in the fastest case, with 20 "apache static images".
Where is all this time spend, in nextcloud?
Client Setup
I grabbed with firefox and the developer console the request header for an arbitrary preview image.("50789.JPG") with the size 32x32.(x=32&y=32). This request header is used from command line to re-read the same image again and again. (The request header is necessary for the session cookies.)
while true ; do
time netcat
Server Setup
On the server side I used strace and attached it to the php-fpm process. I divided the request on the server side in three steps:
The strace starts with the strace GET line, where the image ist send to the php-fpm process:
strace -T -o /tmp/a.dbg -s 2024 -tt -p pid
The idea of these three steps is: A cache should be a single lookup key(path+jpg)->value(jpeg+path) Maybe some access right checking for the key before (preparation). After that, the cached value is send out.(response)
Here is the results from the strace:
1 Preparation (takes 1011ms of overall 1053ms)
time | syscall 03.088579 read(3, "...GET ...50789.JPG...")
(Detailed mariadb calls for "preparation" are below) The next time, the image name appears, is in a maradb lookup, where a "path hash" is passed to mariadb and the filename for the previe image is returned. The "process" step starts at:
2 Process (starts after 1011ms, takes 42 ms)
04.099692 sendto(5, "SELECT fileid,storage, path); 04.099818 recvfrom(5,"oc_filecache.."
After that, the filename does not appear anymore in the trace. Now some mariadb lookups happen
04.112715 sendto(5,"SELECT id, numeric_id, available, last_checked)" 04.113697 sendto(5,"SELECTfileid,storage," 04.115236 sendto(5,"SELECTfileid,storage,path 04.116613 sendto(5,"SELECT fileid, storage 04.119943 sendto(5,"SELECT fileid, storage, path" 04.121042 sendto(5,"SELECT fileid,
At this time a mariadb sends back the path to the preview images starting with the bigger previews "2048-1536" and a last mariadb call, which return the 32-32-crop.jpg.
04.125552 sendto(5,"SELECT `fileid" 04.125677 recvfrom(5, "...preview/1186/32-32-crop.jpg
The HTTP response is send:
3 Response (send after 1053ms)
04.141209 write(3,"...X-Powered-By:PHP 7.2.5...filename=32-32-crop.jpg")
Preparing and sending out a preview image takes 1053ms in the setup. Most of the time is spend in the "preparation" step (1011ms), reading php files, check authorisation, lstat calls. The actual mariadb calls, which maps from the image name to the preview image, only takes 42ms.
The preparation step performs too many SQL statements and take with 1011ms more than 20 times the time of the process step. Every mariadb SQL statement is fast, but there are too much of them. (Additionally it reads many .php files, maybe too many for a cache lookup.)
Detailed mariadb calls in the preparation step START with: 03.088579 read(3, "...GET ...50789.JPG..."
mariadb:
03.244490 sendto(5 "password" 03.244603 recvfrom
03.260686 sendto(5,"SET SESSION AUTOCOMMIT" 03.260805 recvfrom(5
03.261192 sendto(5, SET SESSION TRANSACTION ISOLATION LEVEL 03.261316 recvfrom(5
03.309892 sendto(5,"SELECT * FROM `oc_appconfig" 03.310009 recvfrom(5
03.534898 sendto(5,"SELECT `uid 03.535018 recvfrom(5,
03.540644 sendto(5, SELECT appid, `configkey 03.540770 recvfrom(5,
03.548464 sendto(5,"SELECT id, uid 03.548592 recvfrom(5,
03.828441 sendto(5,"SELECT gid FROM `oc_group_user " 03.828564 recvfrom(5,
03.991060 sendto(5,"SELECT remote, `share_token" 03.991196 recvfrom(5,
04.002342 sendto(5,"SELECT s.*, f.`fileid" 04.002474 recvfrom(5,
04.004843 sendto(5,"SELECT s.*, f." 04.004975 recvfrom(5,
04.017029 sendto(5,"SELECT id," 04.017152 recvfrom(5,
04.023501 sendto(5,"SELECT `fileid" 04.023622 recvfrom(5,
04.027512 sendto(5,"SELECT storage_id, root_id 04.027634 recvfrom(5,
04.066119 sendto(5,"SELECT uid FROM oc_group_user WHERE (gid = 'admin') 04.066262 recvfrom(5,
04.091103 sendto(5, "SELECT fileid, `storage 04.091247 recvfrom(5,
04.092276 sendto(5,"SELECT id, mimetype FROM `oc_mimetypes 04.092397 recvfrom(5,
End of "preparation" the filename lookup happens:
04.099692 sendto(5, "SELECT `fileid 04.099818 recvfrom(5, "....50789.JPG"
@mvogt1 Independently I did similar analysis. I never described it in such a detailed way (thanks!), but I wrote a proof of concept code to speed things up considerably. Please have a look at my PoC here: https://github.com/tomasz-grobelny/server/tree/reactphp_poc and a partial production-ready pull request here: https://github.com/nextcloud/server/pull/14953
There is also another issue. Is that thumbnails are loading sequentially.
This is a typical problem when streaming data thru PHP, because of the session object.
You can only have one php thread per session.
What i do in my webapps, to avoid having this kind of issue, is closing the session just after doing all the security checks . That way the session object is closed and another php thread can start whilst the webserver is sending the data to the browser. Increasing image loading time drastically, and reducing the loading waterfall.
I just wanted to applaud and thank everyone pitching in to analyze this issue and provide real data in a positive, factual way. I have this problem as well and it's going to make me stop using Nextcloud, which I otherwise love, because the gallery functionality is the single most important one for my workflows.
Hi, preview generation is very long on my i3-4020Y CPU @ 1.50GHz 4gb ram. Loading 20pictures from local network take up to 3min, most of the time is spent in loading all 20 previews, consuming 100% CPU on server side. I've been running some tests to find the root cause.
OC\Core\Controller\PreviewController::getPreviewByFileId()
but it didn't change anything. Hopefully, session is automatically closed in SessionMiddleware
:
public function beforeController($controller, $methodName) {
$useSession = $this->reflector->hasAnnotation('UseSession');
if (!$useSession) {
$this->session->close();
}
}
In the end, cache and session seems to be correct, and the issue seems to rely in preview generation that is way too slow : 10s for a 8MB jpeg photo
When previews are not yet cached, making, say, 20 HTTP request to load 20 previews will simply bring your server to its knees...
Related issue : #437 Maybe related issues : #449 and https://github.com/nextcloud/server/issues/7269
And some interesting reading: https://ownyourbits.com/2019/06/29/understanding-and-improving-nextcloud-previews/ and a PR here: https://github.com/nextcloud/server/pull/16158
ok, both previews are generated in OC\Preview\Generator::getPreview()
:
/**
* Returns a preview of a file
*
* The cache is searched first and if nothing usable was found then a preview is
* generated by one of the providers
* ...
*/
public function getPreview(File $file, $width = -1, $height = -1, $crop = false, $mode = IPreview::MODE_FILL, $mimeType = null) {
...
// Get the max preview and infer the max preview sizes from that
$maxPreview = $this->getMaxPreview($previewFolder, $file, $mimeType);
...
// Try to get a cached preview. Else generate (and store) one
$preview = $this->generatePreview($previewFolder, $maxPreview, $width, $height, $crop, $maxWidth, $maxHeight);
I really don't get it why the max preview is generated first. As far as I understand, it seems to be used for size computation, and then the small preview is based on the max preview... Does any one have an explanation for this?
I really don't get it why the max preview is generated first.
It's faster to generate thumbnails from an image of reasonable size (defined in the config). Also, the original image could be a RAW image or a photoshop file.
This is something like a compilation of issues. I know, having issues separated is important for developers, but since all those issues have already been reported numerous times but did not get resolved over the last years there is little point in doing that again. One could argue that there also is no point in this entry. But I would reply that apparently the app's authors are not aware of the fact that the internet is full of reports like this. The issue is real, present and not handled or fixed. Every few weeks or month some new version comes out with exactly the same major issues. So apparently the issue has to be brought to the maintainers awareness much more. Sorry for that.
The issue in general: nextcloud's / owncloud's gallery app is nice if you have only very few pics, it is useless for a reasonable collection:
I know development is not easy and an ever improving progress. But one should at least see progress over the years or declare the app as "dead". The issue has been around for years now. No real change, despite all the entries in the changelog files pointing out various improvements.
Steps to reproduce
Expected behaviour
A usable gallery that does not recreate all previews and the like again and again and that does not take 20 seconds to download a small sized picture and fail even upon such task now and then. A gallery that is able to prepare pictures for viewing without me having to manually interfere on the server via the command line.
Actual behaviour
The gallery is more or less useless. For each picture the CPU load goes up to 50% php and 50% redis or 50% mysql, sometimes this, sometimes that. Loading a preview takes forever every single time, except if the pics have been client side cached Viewing the slide show is more or less impossible, each picture takes 20-30 secs to get loaded unless it is already cached on the client side. And yes, that is on a fast internet connection. Manual generation of previews and so on sometimes help a bit (maybe 10% improvement), but that cannot be the strategy to go! A gallery app really should be able to prepare some 50 uploaded pictures for viewing in a reasonable amount of time (maybe a few minutes).
Server configuration
A VPS but not a small one, it has enough performance and memory for everything else.
Operating system: Linux
Web server: Apache 2.4 with php as module or fpm
Database: mysql
PHP version: php 7
Nextcloud configuration
Nextcloud version: (see admin page or version.php)
Updated from an older installation or fresh install:
updated a few times, but the issue is present in all versions
List of activated apps:
Are you using external storage, if yes which one: local folder, smb share, sftp, etc. no
Are you using encryption: yes/no no
Are you using custom gallery.cnf config files: yes/no no
Web server error log
nothing related
Nextcloud log
nothing related Linux or MS-Windows
Browser log
a) The javascript console log nothing b) The network log requests are shown and taking ages (for nothing)