search

ariselseng / camerarawpreviews

Camera Raw Previews app for Nextcloud
GNU Affero General Public License v3.0
53 stars 11 forks source link

High RAM/CPU usage when generating large amounts of previews #31

Closed Eeems closed 5 years ago

Eeems commented 5 years ago

When opening a folder with lots of RAW images for the first time and scrolling to the bottom, I noticed that my server's fans went into overdrive. Upon further inspection it was because this addon was attempting to run every single preview generation at the same time (~400 images). This was maxing out the CPU and using over 20GB of RAM (All available and a good chunk of swap).

I would highly recommend adding some sort of configurable pooling/locking mechanism to limit how many processes can run at once to avoid having servers fall over.

ariselseng commented 5 years ago

What platform are you on? Nextcloud or owncloud? At least in nextcloud there is a certified amount of images that's retrieved in the same time. That limit is certainly not as high as 400. More like 8. This is probably also too high. It's not feasible to do this in this app I am afraid, because this is only a preview provider for *cloud. It would be the other apps like Files that handles retrieving of previews. I recommend you try out the preview generator to batch this up one by one during the night.

ariselseng commented 5 years ago

Come to think of it, it may also be a database issue. Maybe a non-optimized query to the the filescache. Are you able to put on a slow query log on you platform?

Eeems commented 5 years ago

I'm running Nextcloud 15.0.6. From what I was seeing it was spinning up well over 8 threads for rendering the images. Unfortunately I have the slow query log turned off, but from what I could tell it wasn't the database that was causing either problem. Most of the CPU time and RAM usage was coming from elsewhere.

As for the preview generator, I have encryption turned on, so it will only generate thumbnails the first time you open the folder.

While it may be that the Files app should ensure that it's not hammering the server with too many requests, I would still urge you to build in some sort of mechanism to enforce that only a certain number of renders can happen at a time just in case a calling application is misbehaving.

ariselseng commented 5 years ago

@Eeems I hear you. Let me check if this is doable without redoing the whole app. I have a method I can respond too that is "isAvailable()" that I can return false. But that will just stop it from generating previews. Limiting the amount would be NC core territory.

I bet this mostly comes from big files and the process of decrypting them + many threads. For small jpegs there isn't much decrypting needed, but for bigger files like CR2 there is. It makes sense. Maybe you could test with some very big tiff or jpeg files? Like 30MB+. I bet you would see the same problem as with raw.

Probably when files are encrypted the amount of previews requests by the Files app, should be limiting not only to a certain amount of images per request, but also limited to a maximum size.

ariselseng commented 5 years ago

@Eeems Another observation here is that the Files app in thumbnail mode shows way more images than if using the Gallery app in the same folder. Does it help for you in the Gallery app?

Eeems commented 5 years ago

I was actually in list view in the files app and scrolled to the bottom.

I can replicate the high CPU usage when viewing one of the directories. As you can see this is just due all the queries going on. I'm sure there is something that can be done to optimize that. image

Based on the number of threads being opened when I do this though, I'm assuming that the Files app is not limiting to 8 requests at once. My website is sitting behind a sslh-select instance that forks for every request coming in. The httpd threads also spike when doing this as there is normally only 11 threads sitting open: image Same with the mysqld instance, it has a ton of threads spin up. Likely all you could do is setup some sort of background service that limits the number of threads that can work on an image and just have your provider queue up jobs on the thread pool and wait for results. It would still potentially have performance problems and really the place to fix this would be in the Files app. That or in the *cloud handoff to preview providers.

ariselseng commented 5 years ago

@Eeems Thank you for your screenshots. Fixing this in a simple clean way is just not possible due to several things. I need to conform to how a nextcloud app should behave. It's PHP so it's inherently request based. Scrolling to the bottom in the details view of Files app with hundreds of image files that have not produced a preview yet will certain be slow on any kind of format. Not just with raw files. On top of that you have encryption too! I totally agree that this is a problem, it's just not here.

Fixing it here will only help for raw files, and would be a very hacky fix. Let's see if we could fix in this in Files instead.

Eeems commented 5 years ago

Thanks for the help @cowai

phpbg commented 5 years ago

[moved comment to new issue]