search

nextcloud / previewgenerator

Nextcloud app to do preview generation in the background.
https://apps.nextcloud.com/apps/previewgenerator
GNU Affero General Public License v3.0
455 stars 57 forks source link

Preview:pre-generate intermittently stalls with 100% CPU #315

Closed invario closed 2 years ago

invario commented 2 years ago

I'm not sure when precisely this started occurring but I recently updated some stuff on my NextCloud server and have noticed that the cronjob for "occ preview:pre-generate" is hanging on occasion which cascades into multiple cron instances of it hanging along with NextCloud's cron.php also.

I have my NextCloud debugging level set to 0 and have tried to trigger it by repeatedly running pre-generate manually and like I said, it is very intermittent.

Sometimes I will run it 3x in a row and it will trigger the lockup. Other times I have to run it as many as 30x before triggering the issue.

The worst part is that the nextcloud log shows nothing other than this warning message: {"reqId":"Bo2ycNdc64hepUv0yIN1","level":0,"time":"2022-10-05T01:24:36+00:00","remoteAddr":"","user":"--","app":"metadata","method":"","url":"--","message":"/appinfo/app.php is deprecated, use \\OCP\\AppFramework\\Bootstrap\\IBootstrap on the application class instead.","userAgent":"--","version":"24.0.5.1","data":{"app":"metadata"}}

I tried disabling the metadata app on the chance that it was causing some weird conflict but that didn't do anything except completely make my nextcloud log have nothing from running pre-generate.

If the problem doesn't happen, pre-generate runs and completes in about 1-2 seconds. Very fast.

If it hangs, it will sit there forever until I kill it.

I left it alone for about 24 hours and it got to the point where between cron.php and pre-generate processes, there were 7 running at 100% CPU each. I have a 6 core AMD processor so it was fully utilizing 70% of my CPU as a result.

For now I've removed the crontab entry for pre-generate. I should also note that I recently wiped all previews (the entire folder) and ran a "occ files:scan-app-data" to reset everything, and then a "previews:generate-all" on somewhere around 450GB of images/videos. It took a long time but completed. Afterwards, I ran another files:scan-app-data and files:scan just for good measure.

Any ideas? Anyone else have this issue? Thank you.

Currently on: TrueNAS Core 13 IOCage jail running FreeBSD 13.1-RELEASE-p2 NextCloud 24.0.5 PHP 7.4.30 MySQL v8.0.29

invario commented 2 years ago

I've been troubleshooting this for the last few hours with very little progress made except that I believe this might be an issue with NextCloud server and not Preview Generator.

I've taken the following steps:

After all that, I created a script to loop executing NextCloud's crontab.php discovering that it also has a problem with intermittently stalling at 100% CPU.

Very puzzled... I'm certain this didn't use to happen.

charleypaulus commented 2 years ago

I had a similar issue, described in this post. I ran some tests to understand the interaction between Nextcloud server and Preview Generator.

invario commented 2 years ago

Update: after a crazy amount of troubleshooting, , removing plugins, reinstalling NC, destroying jails and creating them from scratch, I've determined this problem NOT to be related to PG.

For now, it appears to have been PHP related to the point where the PHP binary may be the culprit. I think I've narrowed it down to a custom compiled version of the imagick plugin.