search

nextcloud / nextcloudpi

📦 Build code for NextcloudPi: Raspberry Pi, Odroid, Rock64, curl installer...
https://nextcloudpi.com
2.53k stars 296 forks source link

nc_previews : Could not create folder #1260

Open dsmic opened 3 years ago

dsmic commented 3 years ago

I tracked down this problem to a race condition in /usr/local/bin/nc-previews. So you can not reproduce it quite simple, but I am quite sure it is that problem, as I get in /var/log/ncp.log four times the same file logged for preview if the error is thrown.

I commented the cause in /usr/local/bin/nc-previews:

echo -e "\n[ nc-previews-auto ]" >> /var/log/ncp.log
(
#    for i in $(seq 1 $(nproc)); do
      ionice -c3 nice -n20 /usr/local/bin/ncc preview:pre-generate -n -vvv &
#    done
    wait
) 2>&1 >>/var/log/ncp.log &

PID=$!
[[ "0" != 0 ]] && {
  for i in $(seq 1 1 0); do
    sleep 1
    kill -0 "$PID" &>/dev/null || break
  done
  pkill -f preview:pre-generate
  pkill -f preview:generate-all
}
wait "$PID"

This is of cause not a nice solution, as it stops using four processes. But from time to time the cron job throws this exception otherwise resulting in an e-mail to the root user...

2021-02-10T21:28:37+00:00 2021-02-10T21:28:37+00:00 In Folder.php line 167: 2021-02-10T21:28:37+00:00 2021-02-10T21:28:37+00:00 [OCP\Files\NotPermittedException] 2021-02-10T21:28:37+00:00 Could not create folder/appdata_oc0iazbq1rtc/preview/b/3/9/f/5/d/3/487265 2021-02-10T21:28:37+00:00 2021-02-10T21:28:37+00:00 2021-02-10T21:28:37+00:00 Exception trace: 2021-02-10T21:28:37+00:00 at /var/www/nextcloud/lib/private/Files/Node/Folder.php:167 2021-02-10T21:28:37+00:00 OC\Files\Node\Folder->newFolder() at /var/www/nextcloud/lib/private/Files/AppData/AppData.php:157 2021-02-10T21:28:37+00:00 OC\Files\AppData\AppData->newFolder() at /var/www/nextcloud/lib/private/Preview/Storage/Root.php:75 2021-02-10T21:28:37+00:00 OC\Preview\Storage\Root->newFolder() at /var/www/nextcloud/lib/private/Preview/Generator.php:471 2021-02-10T21:28:37+00:00 OC\Preview\Generator->getPreviewFolder() at /var/www/nextcloud/lib/private/Preview/Generator.php:132 2021-02-10T21:28:37+00:00 OC\Preview\Generator->generatePreviews() at /var/www/nextcloud/lib/private/PreviewManager.php:205 2021-02-10T21:28:37+00:00 OC\PreviewManager->generatePreviews() at /var/www/nextcloud/apps/previewgenerator/lib/Command/PreGenerate.php:217 2021-02-10T21:28:37+00:00 OCA\PreviewGenerator\Command\PreGenerate->processFile() at /var/www/nextcloud/apps/previewgenerator/lib/Command/PreGenerate.php:195 2021-02-10T21:28:37+00:00 OCA\PreviewGenerator\Command\PreGenerate->processRow() at /var/www/nextcloud/apps/previewgenerator/lib/Command/PreGenerate.php:163 2021-02-10T21:28:37+00:00 OCA\PreviewGenerator\Command\PreGenerate->startProcessing() at /var/www/nextcloud/apps/previewgenerator/lib/Command/PreGenerate.php:131 2021-02-10T21:28:37+00:00 OCA\PreviewGenerator\Command\PreGenerate->execute() at /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php:255 2021-02-10T21:28:37+00:00 Symfony\Component\Console\Command\Command->run() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:1000 2021-02-10T21:28:37+00:00 Symfony\Component\Console\Application->doRunCommand() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:271 2021-02-10T21:28:37+00:00 Symfony\Component\Console\Application->doRun() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:147 2021-02-10T21:28:37+00:00 Symfony\Component\Console\Application->run() at /var/www/nextcloud/lib/private/Console/Application.php:215 2021-02-10T21:28:37+00:00 OC\Console\Application->run() at /var/www/nextcloud/console.php:100 2021-02-10T21:28:37+00:00 require_once() at /var/www/nextcloud/occ:11 2021-02-10T21:28:37+00:00 2021-02-10T21:28:37+00:00 preview:pre-generate 2021-02-10T21:28:37+00:00 2021-02-10T21:28:37+00:00 2021-02-10T21:28:37+00:00 In Folder.php line 167: 2021-02-10T21:28:37+00:00 2021-02-10T21:28:37+00:00 [OCP\Files\NotPermittedException] 2021-02-10T21:28:37+00:00 Could not create folder/appdata_oc0iazbq1rtc/preview/b/3/9/f/5/d/3/487265 2021-02-10T21:28:37+00:00 2021-02-10T21:28:37+00:00 2021-02-10T21:28:37+00:00 Exception trace: 2021-02-10T21:28:37+00:00 at /var/www/nextcloud/lib/private/Files/Node/Folder.php:167 2021-02-10T21:28:37+00:00 OC\Files\Node\Folder->newFolder() at /var/www/nextcloud/lib/private/Files/AppData/AppData.php:157 2021-02-10T21:28:37+00:00 OC\Files\AppData\AppData->newFolder() at /var/www/nextcloud/lib/private/Preview/Storage/Root.php:75 2021-02-10T21:28:37+00:00 OC\Preview\Storage\Root->newFolder() at /var/www/nextcloud/lib/private/Preview/Generator.php:471 2021-02-10T21:28:37+00:00 OC\Preview\Generator->getPreviewFolder() at /var/www/nextcloud/lib/private/Preview/Generator.php:132 2021-02-10T21:28:37+00:00 OC\Preview\Generator->generatePreviews() at /var/www/nextcloud/lib/private/PreviewManager.php:205 2021-02-10T21:28:37+00:00 OC\PreviewManager->generatePreviews() at /var/www/nextcloud/apps/previewgenerator/lib/Command/PreGenerate.php:217 2021-02-10T21:28:37+00:00 OCA\PreviewGenerator\Command\PreGenerate->processFile() at /var/www/nextcloud/apps/previewgenerator/lib/Command/PreGenerate.php:195 2021-02-10T21:28:37+00:00 OCA\PreviewGenerator\Command\PreGenerate->processRow() at /var/www/nextcloud/apps/previewgenerator/lib/Command/PreGenerate.php:163 2021-02-10T21:28:37+00:00 OCA\PreviewGenerator\Command\PreGenerate->startProcessing() at /var/www/nextcloud/apps/previewgenerator/lib/Command/PreGenerate.php:131 2021-02-10T21:28:37+00:00 OCA\PreviewGenerator\Command\PreGenerate->execute() at /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php:255 2021-02-10T21:28:37+00:00 Symfony\Component\Console\Command\Command->run() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:1000 2021-02-10T21:28:37+00:00 Symfony\Component\Console\Application->doRunCommand() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:271 2021-02-10T21:28:37+00:00 Symfony\Component\Console\Application->doRun() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:147 2021-02-10T21:28:37+00:00 Symfony\Component\Console\Application->run() at /var/www/nextcloud/lib/private/Console/Application.php:215 2021-02-10T21:28:37+00:00 OC\Console\Application->run() at /var/www/nextcloud/console.php:100 2021-02-10T21:28:37+00:00 require_once() at /var/www/nextcloud/occ:11 2021-02-10T21:28:37+00:00 2021-02-10T21:28:37+00:00 preview:pre-generate

feutl commented 3 years ago

I have the same problem, just want to back the issue.

dsmic commented 3 years ago

Thx, I have my fix running since I reported the issue without any problems. But as nc-preview is not a file in the git repository I do not know how to create a pull request.

nc-preview seems to be generated in some kind of install procedure from other files….

nachoparker commented 3 years ago

I think this is fixed now in the latest version, please reopen if this is not the case

mwip commented 2 years ago

Even though the messages indeed seem to be less often, I still get them once or twice a week. (This is as of NCP 1.42.2)

From: root@hostname (Cron Daemon)
To: root@hostname
Subject: Cron <root@hostname>  /usr/local/bin/nc-previews
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
Message-Id: <id.id@hostname>
Date: Tue, 26 Oct 2021 02:00:06 +0200 (CEST)

2021-10-26T00:00:06+00:00
2021-10-26T00:00:06+00:00
2021-10-26T00:00:06+00:00 In Folder.php line 176:
2021-10-26T00:00:06+00:00 In Folder.php line 176:
2021-10-26T00:00:06+00:00
2021-10-26T00:00:06+00:00   [OCP\Files\NotPermittedException]
2021-10-26T00:00:06+00:00
2021-10-26T00:00:06+00:00   Could not create folder
2021-10-26T00:00:06+00:00
2021-10-26T00:00:06+00:00
2021-10-26T00:00:06+00:00   [OCP\Files\NotPermittedException]
2021-10-26T00:00:06+00:00 Exception trace:
2021-10-26T00:00:06+00:00   Could not create folder
2021-10-26T00:00:06+00:00
2021-10-26T00:00:06+00:00   at /var/www/nextcloud/lib/private/Files/Node/Folder.php:176
2021-10-26T00:00:06+00:00
2021-10-26T00:00:06+00:00  OC\Files\Node\Folder->newFolder() at /var/www/nextcloud/lib/private/Files/App
Data/AppData.php:157
2021-10-26T00:00:06+00:00 Exception trace:
2021-10-26T00:00:06+00:00  OC\Files\AppData\AppData->newFolder() at /var/www/nextcloud/lib/private/Previ
ew/Storage/Root.php:75
2021-10-26T00:00:06+00:00  OC\Preview\Storage\Root->newFolder() at /var/www/nextcloud/lib/private/Previe
w/Generator.php:471
2021-10-26T00:00:06+00:00   at /var/www/nextcloud/lib/private/Files/Node/Folder.php:176
2021-10-26T00:00:06+00:00  OC\Preview\Generator->getPreviewFolder() at /var/www/nextcloud/lib/private/Pr
eview/Generator.php:132
2021-10-26T00:00:06+00:00  OC\Files\Node\Folder->newFolder() at /var/www/nextcloud/lib/private/Files/App
Data/AppData.php:157
2021-10-26T00:00:06+00:00  OC\Preview\Generator->generatePreviews() at /var/www/nextcloud/lib/private/Pr
eviewManager.php:205
2021-10-26T00:00:06+00:00  OC\Files\AppData\AppData->newFolder() at /var/www/nextcloud/lib/private/Previ
ew/Storage/Root.php:75
2021-10-26T00:00:06+00:00  OC\PreviewManager->generatePreviews() at /var/www/ncp-previewgenerator/ncp-pr
eviewgenerator-nc21/lib/Command/PreGenerate.php:209
2021-10-26T00:00:06+00:00  OC\Preview\Storage\Root->newFolder() at /var/www/nextcloud/lib/private/Previe
w/Generator.php:471
2021-10-26T00:00:06+00:00  OC\Preview\Generator->getPreviewFolder() at /var/www/nextcloud/lib/private/Pr
eview/Generator.php:132
2021-10-26T00:00:06+00:00  OCA\PreviewGenerator\Command\PreGenerate->processFile() at /var/www/ncp-previ
ewgenerator/ncp-previewgenerator-nc21/lib/Command/PreGenerate.php:187
2021-10-26T00:00:06+00:00  OCA\PreviewGenerator\Command\PreGenerate->processRow() at /var/www/ncp-previe
wgenerator/ncp-previewgenerator-nc21/lib/Command/PreGenerate.php:151
2021-10-26T00:00:06+00:00  OC\Preview\Generator->generatePreviews() at /var/www/nextcloud/lib/private/Pr
eviewManager.php:205
2021-10-26T00:00:06+00:00  OCA\PreviewGenerator\Command\PreGenerate->startProcessing() at /var/www/ncp-p
reviewgenerator/ncp-previewgenerator-nc21/lib/Command/PreGenerate.php:124
2021-10-26T00:00:06+00:00  OC\PreviewManager->generatePreviews() at /var/www/ncp-previewgenerator/ncp-pr
eviewgenerator-nc21/lib/Command/PreGenerate.php:209
2021-10-26T00:00:06+00:00  OCA\PreviewGenerator\Command\PreGenerate->execute() at /var/www/nextcloud/3rd
party/symfony/console/Command/Command.php:255
2021-10-26T00:00:06+00:00  Symfony\Component\Console\Command\Command->run() at /var/www/nextcloud/3rdpar
ty/symfony/console/Application.php:1009
2021-10-26T00:00:06+00:00  OCA\PreviewGenerator\Command\PreGenerate->processFile() at /var/www/ncp-previ
ewgenerator/ncp-previewgenerator-nc21/lib/Command/PreGenerate.php:187
2021-10-26T00:00:06+00:00  Symfony\Component\Console\Application->doRunCommand() at /var/www/nextcloud/3
rdparty/symfony/console/Application.php:273
2021-10-26T00:00:06+00:00  Symfony\Component\Console\Application->doRun() at /var/www/nextcloud/3rdparty
/symfony/console/Application.php:149
2021-10-26T00:00:06+00:00  OCA\PreviewGenerator\Command\PreGenerate->processRow() at /var/www/ncp-previe
wgenerator/ncp-previewgenerator-nc21/lib/Command/PreGenerate.php:151
2021-10-26T00:00:06+00:00  Symfony\Component\Console\Application->run() at /var/www/nextcloud/lib/privat
e/Console/Application.php:215
2021-10-26T00:00:06+00:00  OCA\PreviewGenerator\Command\PreGenerate->startProcessing() at /var/www/ncp-p
reviewgenerator/ncp-previewgenerator-nc21/lib/Command/PreGenerate.php:124
2021-10-26T00:00:06+00:00  OC\Console\Application->run() at /var/www/nextcloud/console.php:100
2021-10-26T00:00:06+00:00  OCA\PreviewGenerator\Command\PreGenerate->execute() at /var/www/nextcloud/3rd
party/symfony/console/Command/Command.php:255
2021-10-26T00:00:06+00:00  require_once() at /var/www/nextcloud/occ:11
2021-10-26T00:00:06+00:00
2021-10-26T00:00:06+00:00  Symfony\Component\Console\Command\Command->run() at /var/www/nextcloud/3rdpar
ty/symfony/console/Application.php:1009
2021-10-26T00:00:06+00:00 preview:pre-generate
2021-10-26T00:00:06+00:00  Symfony\Component\Console\Application->doRunCommand() at /var/www/nextcloud/3
rdparty/symfony/console/Application.php:273
2021-10-26T00:00:06+00:00
2021-10-26T00:00:06+00:00  Symfony\Component\Console\Application->doRun() at /var/www/nextcloud/3rdparty
/symfony/console/Application.php:149
2021-10-26T00:00:06+00:00  Symfony\Component\Console\Application->run() at /var/www/nextcloud/lib/privat
e/Console/Application.php:215
2021-10-26T00:00:06+00:00  OC\Console\Application->run() at /var/www/nextcloud/console.php:100
2021-10-26T00:00:06+00:00  require_once() at /var/www/nextcloud/occ:11
2021-10-26T00:00:06+00:00
2021-10-26T00:00:06+00:00 preview:pre-generate
2021-10-26T00:00:06+00:00
feutl commented 2 years ago

The same for me, I was wondering why, but on an irregular basis I get the same messages. Irregular means, not every week on the same day, but at least once a week. @nachoparker can you reopen the ticket to figure out whats going on ?

nachoparker commented 2 years ago

ok, so the race condition is still there then

dsmic commented 2 years ago

nc-previews-auto.sh still runs multiple processes… I have the fix from my first post running for one year now without problems….

ADadlDudlDa commented 1 year ago

I think I have got the same issue. I will change /usr/local/bin/nc-previews in the same way, as @dsmic done it, and try whether it helps.