search

nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
26.76k stars 4k forks source link

[Bug]: Server gets stuck with dozens of MariaDB queries running (most oc_filecache related) (no deadlock reported) #42344

Open major-mayer opened 9 months ago

major-mayer commented 9 months ago

⚠️ This issue respects the following points: ⚠️

Bug description

I just noticed today in the morning that our Nextcloud instance is not reachable any more. It is used by about 10 users (concurrently max. 3 I would say) and uses Docker containers for MariaDB, Redis and Nextcloud itself. I can't tell what was going wrong, but what I noticed is that there were about over a hundred of PHP worker threads running, all seemingly waiting for MariaDB to execute certain SQL queries.
MariaDB itself was creating significant CPU load with the execution of those queries.

The "fix" was simply to restart both the PHP server and MariaDB, but I still wanted to report this as a bug, since it happened one month ago as well.

I tried to capture a bit of debug information. The first thing I copied was the output of mysqladmin -u root -p -i 1 processlist

mysqladmin -u root -p -i 1 processlist
Enter password: 
+-----+---------+------------------+-----------+---------+-------+--------------+------------------------------------------------------------------------------------------------------+----------+
| Id  | User    | Host             | db        | Command | Time  | State        | Info                                                                                                 | Progress |
+-----+---------+------------------+-----------+---------+-------+--------------+------------------------------------------------------------------------------------------------------+----------+
| 212 | nextadm | 172.19.0.8:33222 | nextcloud | Query   | 62027 | Sending data | SELECT COUNT(*) FROM `oc_calendarobjects` WHERE (`calendarid` = '7') AND (`uid` = '29780378216658282 | 0.000    |
| 234 | nextadm | 172.19.0.8:42142 | nextcloud | Query   | 60240 | Sending data | SELECT `filecache`.`fileid`, `storage`, `path`, `path_hash`, `filecache`.`parent`, `filecache`.`name | 0.000    |
| 394 | nextadm | 172.19.0.8:36764 | nextcloud | Query   | 44281 | Update       | INSERT INTO `oc_activity` (`app`, `subject`, `subjectparams`, `message`, `messageparams`, `file`, `l | 0.000    |
| 412 | nextadm | 172.19.0.8:43282 | nextcloud | Query   | 44161 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 413 | nextadm | 172.19.0.8:43276 | nextcloud | Query   | 44176 | Updating     | UPDATE `oc_filecache` SET `mtime` = GREATEST(`mtime`, 1702824045), `etag` = '657f086d3e10a', `size`  | 0.000    |
| 414 | nextadm | 172.19.0.8:43292 | nextcloud | Query   | 44167 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 415 | nextadm | 172.19.0.8:43298 | nextcloud | Query   | 44174 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 416 | nextadm | 172.19.0.8:43308 | nextcloud | Query   | 44176 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 417 | nextadm | 172.19.0.8:43324 | nextcloud | Query   | 44172 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 418 | nextadm | 172.19.0.8:43330 | nextcloud | Query   | 44158 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 419 | nextadm | 172.19.0.8:43340 | nextcloud | Query   | 44158 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 420 | nextadm | 172.19.0.8:41668 | nextcloud | Query   | 44158 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 421 | nextadm | 172.19.0.8:41674 | nextcloud | Query   | 44172 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 425 | nextadm | 172.19.0.8:50232 | nextcloud | Query   | 44141 | Statistics   | SELECT `s`.*, `f`.`fileid`, `f`.`path`, `f`.`permissions` as `f_permissions`, `f`.`storage`, `f`.`pa | 0.000    |
| 442 | nextadm | 172.19.0.8:33146 | nextcloud | Query   | 44120 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 443 | nextadm | 172.19.0.8:33154 | nextcloud | Query   | 44108 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 444 | nextadm | 172.19.0.8:33166 | nextcloud | Query   | 44120 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 445 | nextadm | 172.19.0.8:33172 | nextcloud | Query   | 44109 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 465 | nextadm | 172.19.0.8:59736 | nextcloud | Query   | 44081 | Statistics   | SELECT `s`.*, `f`.`fileid`, `f`.`path`, `f`.`permissions` as `f_permissions`, `f`.`storage`, `f`.`pa | 0.000    |
| 480 | nextadm | 172.19.0.8:44428 | nextcloud | Query   | 44037 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 481 | nextadm | 172.19.0.8:44442 | nextcloud | Query   | 44037 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 482 | nextadm | 172.19.0.8:44456 | nextcloud | Query   | 44027 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 483 | nextadm | 172.19.0.8:44464 | nextcloud | Query   | 44026 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 484 | nextadm | 172.19.0.8:44480 | nextcloud | Query   | 44032 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 485 | nextadm | 172.19.0.8:44494 | nextcloud | Query   | 44037 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 486 | nextadm | 172.19.0.8:44510 | nextcloud | Query   | 44035 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 487 | nextadm | 172.19.0.8:44518 | nextcloud | Query   | 44026 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 492 | nextadm | 172.19.0.8:49032 | nextcloud | Query   | 44020 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 504 | nextadm | 172.19.0.8:50734 | nextcloud | Query   | 42575 | Update       | INSERT INTO `oc_filecache` (`mimepart`, `mimetype`, `mtime`, `size`, `etag`, `storage_mtime`, `permi | 0.000    |
| 517 | nextadm | 172.19.0.8:36380 | nextcloud | Query   | 42571 | Statistics   | SELECT `s`.*, `f`.`fileid`, `f`.`path`, `f`.`permissions` as `f_permissions`, `f`.`storage`, `f`.`pa | 0.000    |
| 526 | nextadm | 172.19.0.8:47402 | nextcloud | Query   | 42541 | Statistics   | SELECT `s`.*, `f`.`fileid`, `f`.`path`, `f`.`permissions` as `f_permissions`, `f`.`storage`, `f`.`pa | 0.000    |
| 533 | nextadm | 172.19.0.8:36112 | nextcloud | Query   | 41579 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 535 | nextadm | 172.19.0.8:59378 | nextcloud | Query   | 41338 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 536 | nextadm | 172.19.0.8:40760 | nextcloud | Query   | 41134 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 537 | nextadm | 172.19.0.8:40768 | nextcloud | Query   | 41133 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 538 | nextadm | 172.19.0.8:40778 | nextcloud | Query   | 41131 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 539 | nextadm | 172.19.0.8:40794 | nextcloud | Query   | 41130 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 540 | nextadm | 172.19.0.8:40810 | nextcloud | Query   | 41128 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 541 | nextadm | 172.19.0.8:40822 | nextcloud | Query   | 41127 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 542 | nextadm | 172.19.0.8:37852 | nextcloud | Query   | 41117 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 544 | nextadm | 172.19.0.8:50022 | nextcloud | Query   | 41098 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 547 | nextadm | 172.19.0.8:56428 | nextcloud | Query   | 41014 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 548 | nextadm | 172.19.0.8:36222 | nextcloud | Query   | 41009 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 549 | nextadm | 172.19.0.8:36234 | nextcloud | Query   | 41007 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 550 | nextadm | 172.19.0.8:36250 | nextcloud | Query   | 41006 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 551 | nextadm | 172.19.0.8:54990 | nextcloud | Query   | 41005 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 552 | nextadm | 172.19.0.8:54998 | nextcloud | Query   | 41004 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 553 | nextadm | 172.19.0.8:55004 | nextcloud | Query   | 41003 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 554 | nextadm | 172.19.0.8:40436 | nextcloud | Query   | 40991 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 555 | nextadm | 172.19.0.8:57428 | nextcloud | Query   | 40974 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 556 | nextadm | 172.19.0.8:57442 | nextcloud | Query   | 40966 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 557 | nextadm | 172.19.0.8:35898 | nextcloud | Query   | 40965 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 558 | nextadm | 172.19.0.8:35912 | nextcloud | Query   | 40964 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 559 | nextadm | 172.19.0.8:35920 | nextcloud | Query   | 40962 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 560 | nextadm | 172.19.0.8:35924 | nextcloud | Query   | 40961 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 561 | nextadm | 172.19.0.8:35934 | nextcloud | Query   | 40960 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 562 | nextadm | 172.19.0.8:35948 | nextcloud | Query   | 40959 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 563 | nextadm | 172.19.0.8:35960 | nextcloud | Query   | 40958 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 564 | nextadm | 172.19.0.8:54368 | nextcloud | Query   | 40955 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 565 | nextadm | 172.19.0.8:54378 | nextcloud | Query   | 40952 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 566 | nextadm | 172.19.0.8:49922 | nextcloud | Query   | 40915 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 568 | nextadm | 172.19.0.8:60282 | nextcloud | Query   | 40856 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 569 | nextadm | 172.19.0.8:60292 | nextcloud | Query   | 40856 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 570 | nextadm | 172.19.0.8:43224 | nextcloud | Query   | 40853 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 571 | nextadm | 172.19.0.8:43234 | nextcloud | Query   | 40853 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 572 | nextadm | 172.19.0.8:43242 | nextcloud | Query   | 40851 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 573 | nextadm | 172.19.0.8:43248 | nextcloud | Query   | 40850 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 574 | nextadm | 172.19.0.8:43264 | nextcloud | Query   | 40848 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 575 | nextadm | 172.19.0.8:43268 | nextcloud | Query   | 40847 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 576 | nextadm | 172.19.0.8:50402 | nextcloud | Query   | 40791 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 577 | nextadm | 172.19.0.8:58896 | nextcloud | Query   | 40729 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 578 | nextadm | 172.19.0.8:47052 | nextcloud | Query   | 40665 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 579 | nextadm | 172.19.0.8:47188 | nextcloud | Query   | 40605 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 580 | nextadm | 172.19.0.8:47314 | nextcloud | Query   | 40542 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 581 | nextadm | 172.19.0.8:44202 | nextcloud | Query   | 40481 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 588 | nextadm | 172.19.0.8:37748 | nextcloud | Query   | 40418 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 589 | nextadm | 172.19.0.8:39852 | nextcloud | Query   | 40356 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 590 | nextadm | 172.19.0.8:39760 | nextcloud | Query   | 40295 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 591 | nextadm | 172.19.0.8:47856 | nextcloud | Query   | 40233 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 593 | nextadm | 172.19.0.8:47032 | nextcloud | Query   | 40218 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 31) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 594 | nextadm | 172.19.0.8:55482 | nextcloud | Query   | 40171 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 595 | nextadm | 172.19.0.8:60646 | nextcloud | Query   | 40108 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 596 | nextadm | 172.19.0.8:59714 | nextcloud | Query   | 40070 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 2) AND (`path_hash` IN ('9d96bab0f8958719e389dd | 0.000    |
| 597 | nextadm | 172.19.0.8:33882 | nextcloud | Query   | 40046 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 598 | nextadm | 172.19.0.8:59828 | nextcloud | Query   | 40027 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 2) AND (`path_hash` IN ('9d96bab0f8958719e389dd | 0.000    |
| 606 | nextadm | 172.19.0.8:59898 | nextcloud | Query   | 40027 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 2) AND (`path_hash` IN ('9d96bab0f8958719e389dd | 0.000    |
| 607 | nextadm | 172.19.0.8:49924 | nextcloud | Query   | 39985 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 608 | nextadm | 172.19.0.8:49932 | nextcloud | Query   | 39976 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 2) AND (`path_hash` IN ('9d96bab0f8958719e389dd | 0.000    |
| 609 | nextadm | 172.19.0.8:59806 | nextcloud | Query   | 39950 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 2) AND (`path_hash` IN ('9d96bab0f8958719e389dd | 0.000    |
| 610 | nextadm | 172.19.0.8:57178 | nextcloud | Query   | 39933 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 2) AND (`path_hash` IN ('9d96bab0f8958719e389dd | 0.000    |
| 611 | nextadm | 172.19.0.8:46904 | nextcloud | Query   | 39923 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 612 | nextadm | 172.19.0.8:55638 | nextcloud | Query   | 39882 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 2) AND (`path_hash` IN ('9d96bab0f8958719e389dd | 0.000    |
| 613 | nextadm | 172.19.0.8:45936 | nextcloud | Query   | 39861 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 616 | nextadm | 172.19.0.8:56374 | nextcloud | Query   | 39799 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 617 | nextadm | 172.19.0.8:44066 | nextcloud | Query   | 39735 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 618 | nextadm | 172.19.0.8:34464 | nextcloud | Query   | 39674 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 2) AND (`path_hash` IN ('9d96bab0f8958719e389dd | 0.000    |
| 619 | nextadm | 172.19.0.8:34480 | nextcloud | Query   | 39675 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 620 | nextadm | 172.19.0.8:51222 | nextcloud | Query   | 39613 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 621 | nextadm | 172.19.0.8:50032 | nextcloud | Query   | 39551 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 622 | nextadm | 172.19.0.8:50046 | nextcloud | Query   | 39549 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 2) AND (`path_hash` IN ('9d96bab0f8958719e389dd | 0.000    |
| 623 | nextadm | 172.19.0.8:43332 | nextcloud | Query   | 39489 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 624 | nextadm | 172.19.0.8:41440 | nextcloud | Query   | 39427 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 2) AND (`path_hash` IN ('9d96bab0f8958719e389dd | 0.000    |
| 625 | nextadm | 172.19.0.8:41448 | nextcloud | Query   | 39426 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 626 | nextadm | 172.19.0.8:38028 | nextcloud | Query   | 39365 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 627 | nextadm | 172.19.0.8:59884 | nextcloud | Query   | 39302 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 628 | nextadm | 172.19.0.8:48200 | nextcloud | Query   | 39241 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 631 | nextadm | 172.19.0.8:55150 | nextcloud | Query   | 39179 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 632 | nextadm | 172.19.0.8:40124 | nextcloud | Query   | 39152 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 633 | nextadm | 172.19.0.8:43582 | nextcloud | Query   | 39116 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 636 | nextadm | 172.19.0.8:49502 | nextcloud | Query   | 39055 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 640 | nextadm | 172.19.0.8:58540 | nextcloud | Query   | 38993 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 641 | nextadm | 172.19.0.8:51592 | nextcloud | Query   | 38931 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 642 | nextadm | 172.19.0.8:58718 | nextcloud | Query   | 38869 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 643 | nextadm | 172.19.0.8:36838 | nextcloud | Query   | 38805 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 645 | nextadm | 172.19.0.8:34506 | nextcloud | Query   | 38745 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 646 | nextadm | 172.19.0.8:40476 | nextcloud | Query   | 38683 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 649 | nextadm | 172.19.0.8:57530 | nextcloud | Query   | 38621 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 650 | nextadm | 172.19.0.8:60544 | nextcloud | Query   | 38559 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 651 | nextadm | 172.19.0.8:54100 | nextcloud | Query   | 38496 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 652 | nextadm | 172.19.0.8:35352 | nextcloud | Query   | 38435 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 653 | nextadm | 172.19.0.8:54900 | nextcloud | Query   | 38373 | Statistics   | SELECT `path` FROM `oc_filecache` WHERE (`storage` = 43) AND (`path_hash` IN ('9d96bab0f8958719e389d | 0.000    |
| 672 | root    | localhost        |           | Query   | 0     | starting     | show processlist                                                                                     | 0.000    |
+-----+---------+------------------+-----------+---------+-------+--------------+------------------------------------------------------------------------------------------------------+----------+

The next is the output of SHOW ENGINE INNODB STATUS: https://pastebin.com/7muWuFiH

Of course, I needed to strip certain parts of the logs for privacy reasons, I hope it can still be valuable information for debugging what is going on. If there are some critical logs missing, that would be helpful, feel free to tell me. I will add this when this issue occurs the next time.

At first I thought this issue might be related to this one, but my oc_filecache table is roundabout 900mb large, which sounds reasonable for an instance of 1.53TB of storage size.

Steps to reproduce

Unfortunately, I can't tell how to reproduce.

Expected behavior

Nextcloud should not get stuck (i.e. actually return something when I try to request the page, not just timeout).

Installation method

None

Nextcloud Server version

26

Operating system

None

PHP engine version

None

Web server

None

Database engine version

None

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

Configuration report

{
    "system": {
        "maintenance": false,
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "nc.example.com",
            "nctest.example.com"
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/nc.example.com",
        "overwriteprotocol": "https",
        "dbtype": "mysql",
        "version": "27.1.4.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauth": 1,
        "mail_smtpauthtype": "PLAIN",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "enable_previews": true,
        "preview_max_memory": 512,
        "enabledPreviewProviders": [
            "OC\\Preview\\PNG",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\GIF",
            "OC\\Preview\\BMP",
            "OC\\Preview\\XBitmap",
            "OC\\Preview\\MP3",
            "OC\\Preview\\TXT",
            "OC\\Preview\\MarkDown",
            "OC\\Preview\\OpenDocument",
            "OC\\Preview\\Krita",
            "OC\\Preview\\Imaginary",
            "OC\\Preview\\HEIC",
            "OC\\Preview\\PDF",
            "OC\\Preview\\SVG",
            "OC\\Preview\\Movie"
        ],
        "preview_imaginary_url": "http:\/\/imaginary:9000",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "loglevel": 2,
        "theme": "",
        "onlyoffice": {
            "verify_peer_off": true
        },
        "activity_use_cached_mountpoints": true,
        "defaultapp": "files",
        "simpleSignUpLink.shown": false,
        "default_phone_region": "DE",
        "app_install_overwrite": [
            "imageconverter",
            "previewgenerator"
        ],
        "jpeg_quality": "60",
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "logfile": "\/dev\/stdout",
        "memories.vod.ffmpeg": "\/usr\/bin\/ffmpeg",
        "memories.vod.ffprobe": "\/usr\/bin\/ffprobe",
        "memories.vod.disable": false,
        "memories.vod.external": true,
        "memories.vod.bind": "",
        "memories.vod.connect": "go-vod:47788",
        "memories.vod.path": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/go-vod-amd64",
        "memories.exiftool_no_local": true,
        "memories.exiftool": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/exiftool-amd64-glibc",
        "memories.gis_type": 1,
        "memories.vod.qf": 30
    }
}

List of activated Apps

Enabled:
  - activity: 2.19.0
  - admin_audit: 1.17.0
  - calendar: 4.6.0
  - circles: 27.0.1
  - cloud_federation_api: 1.10.0
  - comments: 1.17.0
  - contacts: 5.5.0
  - contactsinteraction: 1.8.0
  - dashboard: 7.7.0
  - dav: 1.27.0
  - federatedfilesharing: 1.17.0
  - federation: 1.17.0
  - files: 1.22.0
  - files_external: 1.19.0
  - files_pdfviewer: 2.8.0
  - files_reminders: 1.0.0
  - files_retention: 1.16.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - firstrunwizard: 2.16.0
  - forms: 3.4.3
  - groupfolders: 15.3.1
  - imageconverter: 1.3.5
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - memories: 6.1.5
  - music: 1.9.1
  - nextcloud_announcements: 1.16.0
  - notifications: 2.15.0
  - notify_push: 0.6.6
  - oauth2: 1.15.1
  - onlyoffice: 8.2.4
  - password_policy: 1.17.0
  - polls: 5.4.2
  - previewgenerator: 5.4.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - recommendations: 1.6.0
  - related_resources: 1.2.0
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - support: 1.10.0
  - survey_client: 1.15.0
  - systemtags: 1.17.0
  - tasks: 0.15.0
  - text: 3.8.0
  - theming: 2.2.0
  - twofactor_backupcodes: 1.16.0
  - twofactor_nextcloud_notification: 3.8.0
  - twofactor_totp: 9.0.0
  - updatenotification: 1.17.0
  - user_status: 1.7.0
  - viewer: 2.1.0
  - weather_status: 1.7.0
  - workflowengine: 2.9.0
Disabled:
  - bruteforcesettings: 2.7.0 (installed 2.4.0)
  - cms_pico: 1.0.21 (installed 1.0.21)
  - encryption: 2.15.0
  - photos: 2.3.0 (installed 2.0.1)
  - suspicious_login: 5.0.0
  - user_ldap: 1.17.0

Nextcloud Signing status

Fails du to the following patch (and issue): https://github.com/nextcloud/groupfolders/issues/2559#issuecomment-1807372455

Technical information
=====================
The following list covers which files have failed the integrity check. Please read
the previous linked documentation to learn more about the errors and how to fix
them.

Results
=======
- groupfolders
    - INVALID_HASH
        - lib/Versions/GroupVersionsExpireManager.php

Raw output
==========
Array
(
    [groupfolders] => Array
        (
            [INVALID_HASH] => Array
                (
                    [lib/Versions/GroupVersionsExpireManager.php] => Array
                        (
                            [expected] => 3875bed12bb76cc339a59c778ffcdcf80d886220b37bd9b03a2c3fbd158b727072444cebc2d4a026582e72717e3ccb7a0b52b0f715cffa7f78c3104ca569141e
                            [current] => d73d8ae6cb15ab7f9d3861ffd0d28a77f8d7fe329f20190f501429d23ce6dcc7b178a393944f8c9d05df86ed736a7f8c2c1b0287706ae0e3e39527c844f9cc72
                        )

                )

        )

)

Nextcloud Logs

- -  17/Dec/2023:10:43:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:10:43:18 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:10:53:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:10:53:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:11:03:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:11:03:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:11:11:35 +0000 "GET /remote.php" 200
- -  17/Dec/2023:11:13:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:11:13:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:11:23:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:11:23:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:11:33:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:11:33:18 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:11:38:25 +0000 "PROPFIND /remote.php" 401
- - User1 17/Dec/2023:11:38:25 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:11:43:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:11:43:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:11:53:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:11:53:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:12:01:26 +0000 "OPTIONS /remote.php" 401
- - User2 17/Dec/2023:12:01:26 +0000 "OPTIONS /remote.php" 200
- -  17/Dec/2023:12:01:29 +0000 "OPTIONS /remote.php" 401
- - User2 17/Dec/2023:12:01:29 +0000 "OPTIONS /remote.php" 200
- -  17/Dec/2023:12:01:32 +0000 "PROPFIND /remote.php" 401
- - User2 17/Dec/2023:12:01:32 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:12:01:34 +0000 "PROPFIND /remote.php" 401
- - User2 17/Dec/2023:12:01:34 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:12:01:45 +0000 "PROPFIND /remote.php" 401
- - User2 17/Dec/2023:12:01:45 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:12:01:47 +0000 "REPORT /remote.php" 401
- - User2 17/Dec/2023:12:01:48 +0000 "REPORT /remote.php" 207
- -  17/Dec/2023:12:01:50 +0000 "REPORT /remote.php" 401
- - User2 17/Dec/2023:12:01:50 +0000 "REPORT /remote.php" 207
- -  17/Dec/2023:12:03:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:12:03:18 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:12:13:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:12:13:18 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:12:23:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:12:23:18 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:12:26:21 +0000 "GET /remote.php" 200
- -  17/Dec/2023:12:33:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:12:33:19 +0000 "HEAD /index.php" 200
- - User3 17/Dec/2023:12:37:16 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:12:39:25 +0000 "OPTIONS /remote.php" 401
- - User1 17/Dec/2023:12:39:25 +0000 "OPTIONS /remote.php" 200
- -  17/Dec/2023:12:39:27 +0000 "REPORT /remote.php" 401
- - User1 17/Dec/2023:12:39:27 +0000 "REPORT /remote.php" 207
- -  17/Dec/2023:12:43:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:12:43:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:12:53:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:12:53:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:13:03:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:13:03:18 +0000 "HEAD /index.php" 200
- - User4 17/Dec/2023:13:03:29 +0000 "PROPFIND /remote.php" 207
- - User4 17/Dec/2023:13:06:08 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:13:13:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:13:13:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:13:23:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:13:23:18 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:13:33:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:13:33:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:13:43:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:13:43:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:13:53:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:13:53:19 +0000 "HEAD /index.php" 200
- - User4 17/Dec/2023:13:55:39 +0000 "GET /index.php" 204
- - User4 17/Dec/2023:13:55:40 +0000 "GET /index.php" 204
- - User4 17/Dec/2023:13:55:40 +0000 "GET /status.php" 200
- - User4 17/Dec/2023:13:55:40 +0000 "PROPFIND /remote.php" 207
- - User4 17/Dec/2023:13:55:40 +0000 "GET /ocs/v1.php" 200
- - User4 17/Dec/2023:13:55:40 +0000 "GET /ocs/v2.php" 200
- - User4 17/Dec/2023:13:55:40 +0000 "GET /ocs/v2.php" 200
- - User4 17/Dec/2023:13:55:40 +0000 "GET /ocs/v1.php" 200
- - User4 17/Dec/2023:13:55:41 +0000 "GET /index.php" 200
- - User4 17/Dec/2023:13:55:41 +0000 "GET /remote.php" 200
- - User4 17/Dec/2023:13:55:41 +0000 "GET /ocs/v2.php" 200
- - User4 17/Dec/2023:13:55:41 +0000 "PROPFIND /remote.php" 207
- - User4 17/Dec/2023:13:55:41 +0000 "GET /ocs/v2.php" 200
- - User4 17/Dec/2023:13:55:41 +0000 "PROPFIND /remote.php" 207
- - User4 17/Dec/2023:13:55:42 +0000 "PROPFIND /remote.php" 207
- - User4 17/Dec/2023:13:55:45 +0000 "PROPFIND /remote.php" 207
- - User4 17/Dec/2023:13:55:45 +0000 "PROPFIND /remote.php" 207
- - User4 17/Dec/2023:13:55:46 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:14:03:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:14:03:18 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:14:06:24 +0000 "PROPFIND /remote.php" 401
- - User2 17/Dec/2023:14:06:24 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:14:06:27 +0000 "REPORT /remote.php" 401
- - User2 17/Dec/2023:14:06:27 +0000 "REPORT /remote.php" 207
- -  17/Dec/2023:14:06:29 +0000 "REPORT /remote.php" 401
- - User2 17/Dec/2023:14:06:29 +0000 "REPORT /remote.php" 207
- -  17/Dec/2023:14:06:31 +0000 "PROPFIND /remote.php" 401
- - User2 17/Dec/2023:14:06:31 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:14:06:33 +0000 "PROPFIND /remote.php" 401
- - User2 17/Dec/2023:14:06:33 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:14:06:36 +0000 "PROPFIND /remote.php" 401
- - User2 17/Dec/2023:14:06:36 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:14:06:38 +0000 "REPORT /remote.php" 401
- - User2 17/Dec/2023:14:06:38 +0000 "REPORT /remote.php" 207
- -  17/Dec/2023:14:06:40 +0000 "REPORT /remote.php" 401
- - User2 17/Dec/2023:14:06:40 +0000 "REPORT /remote.php" 207
- -  17/Dec/2023:14:13:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:14:13:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:14:18:23 +0000 "PROPFIND /remote.php" 401
- - User2 17/Dec/2023:14:18:23 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:14:18:25 +0000 "REPORT /remote.php" 401
- - User2 17/Dec/2023:14:18:26 +0000 "REPORT /remote.php" 207
- -  17/Dec/2023:14:18:28 +0000 "REPORT /remote.php" 401
- - User2 17/Dec/2023:14:18:28 +0000 "REPORT /remote.php" 207
- -  17/Dec/2023:14:18:30 +0000 "PROPFIND /remote.php" 401
- - User2 17/Dec/2023:14:18:30 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:14:18:32 +0000 "PROPFIND /remote.php" 401
- - User2 17/Dec/2023:14:18:32 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:14:18:36 +0000 "PROPFIND /remote.php" 401
- - User2 17/Dec/2023:14:18:36 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:14:18:38 +0000 "REPORT /remote.php" 401
- - User2 17/Dec/2023:14:18:38 +0000 "REPORT /remote.php" 207
- -  17/Dec/2023:14:18:40 +0000 "REPORT /remote.php" 401
- - User2 17/Dec/2023:14:18:40 +0000 "REPORT /remote.php" 207
- -  17/Dec/2023:14:23:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:14:23:18 +0000 "HEAD /index.php" 200
- - User5 17/Dec/2023:14:32:45 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:14:33:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:14:33:18 +0000 "HEAD /index.php" 200
- - User3 17/Dec/2023:14:38:30 +0000 "GET /status.php" 200
- - User3 17/Dec/2023:14:38:30 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:14:38:30 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:38:32 +0000 "REPORT /remote.php" 207
- - User3 17/Dec/2023:14:38:32 +0000 "GET /ocs/v1.php" 200
- - User3 17/Dec/2023:14:38:32 +0000 "GET /index.php" 304
- - User3 17/Dec/2023:14:38:32 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:14:38:32 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:14:38:32 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:14:38:32 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:38:31 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:38:34 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:38:34 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:38:37 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:38:38 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:38:43 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:38:44 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:39:01 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:39:01 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:39:03 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:39:04 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:39:51 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:39:52 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:39:56 +0000 "REPORT /remote.php" 207
- - User3 17/Dec/2023:14:39:57 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:40:01 +0000 "GET /status.php" 200
- - User3 17/Dec/2023:14:40:01 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:14:40:01 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:14:40:01 +0000 "REPORT /remote.php" 207
- - User3 17/Dec/2023:14:40:01 +0000 "GET /index.php" 304
- - User3 17/Dec/2023:14:40:01 +0000 "GET /ocs/v1.php" 200
- - User3 17/Dec/2023:14:40:02 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:14:40:02 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:14:41:05 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:06 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:19 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:24 +0000 "GET /status.php" 200
- - User3 17/Dec/2023:14:41:24 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:14:41:24 +0000 "REPORT /remote.php" 207
- - User3 17/Dec/2023:14:41:24 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:14:41:24 +0000 "GET /ocs/v1.php" 200
- - User3 17/Dec/2023:14:41:24 +0000 "GET /index.php" 304
- - User3 17/Dec/2023:14:41:25 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:14:41:25 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:14:41:25 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:25 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:25 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:26 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:26 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:27 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:29 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:29 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:31 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:32 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:41 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:42 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:42 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:43 +0000 "GET /index.php" 200
- - User3 17/Dec/2023:14:41:45 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:45 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:41:58 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:42:01 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:42:04 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:42:05 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:42:06 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:42:07 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:42:11 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:42:12 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:42:13 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:42:15 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:42:19 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:42:22 +0000 "GET /status.php" 200
- - User3 17/Dec/2023:14:42:22 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:14:42:22 +0000 "REPORT /remote.php" 207
- - User3 17/Dec/2023:14:42:22 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:14:42:22 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:42:22 +0000 "GET /index.php" 304
- - User3 17/Dec/2023:14:42:22 +0000 "GET /ocs/v1.php" 200
- - User3 17/Dec/2023:14:42:23 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:14:42:23 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:14:42:23 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:42:23 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:42:26 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:42:27 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:42:28 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:14:43:18 +0000 "HEAD /index.php" 302
- - User3 17/Dec/2023:14:43:19 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:43:19 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:14:43:19 +0000 "HEAD /index.php" 200
- - User3 17/Dec/2023:14:43:23 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:43:24 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:14:43:27 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:14:53:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:14:53:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:14:58:28 +0000 "GET /remote.php" 200
- -  17/Dec/2023:14:58:28 +0000 "GET /remote.php" 200
- -  17/Dec/2023:15:03:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:15:03:19 +0000 "HEAD /index.php" 200
- - User3 17/Dec/2023:15:07:25 +0000 "GET /status.php" 200
- - User3 17/Dec/2023:15:07:26 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:15:07:25 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:15:07:27 +0000 "REPORT /remote.php" 207
- - User3 17/Dec/2023:15:07:27 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:15:07:27 +0000 "GET /index.php" 304
- - User3 17/Dec/2023:15:07:27 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:15:07:27 +0000 "GET /ocs/v1.php" 200
- - User3 17/Dec/2023:15:07:27 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:15:07:27 +0000 "GET /ocs/v2.php" 200
- - User3 17/Dec/2023:15:07:27 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:15:07:26 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:15:07:26 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:15:07:29 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:15:07:30 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:15:07:31 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:15:07:33 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:15:07:35 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:15:07:35 +0000 "PROPFIND /remote.php" 207
- - User3 17/Dec/2023:15:07:47 +0000 "DELETE /remote.php" 404
- - User3 17/Dec/2023:15:07:55 +0000 "DELETE /remote.php" 423
- - User3 17/Dec/2023:15:08:00 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:15:13:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:15:13:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:15:23:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:15:23:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:15:23:59 +0000 "PROPFIND /remote.php" 401
- -  17/Dec/2023:15:28:00 +0000 "REPORT /remote.php" 401
- -  17/Dec/2023:15:32:01 +0000 "REPORT /remote.php" 401
- -  17/Dec/2023:15:33:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:15:33:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:15:36:02 +0000 "PROPFIND /remote.php" 401
- -  17/Dec/2023:15:42:58 +0000 "OPTIONS /remote.php" 401
- - User1 17/Dec/2023:15:42:58 +0000 "OPTIONS /remote.php" 200
- -  17/Dec/2023:15:43:01 +0000 "REPORT /remote.php" 401
- - User1 17/Dec/2023:15:43:02 +0000 "REPORT /remote.php" 207
- -  17/Dec/2023:15:43:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:15:43:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:15:46:40 +0000 "OPTIONS /remote.php" 401
- -  17/Dec/2023:15:49:53 +0000 "GET /index.php" 204
- -  17/Dec/2023:15:49:53 +0000 "GET /index.php" 204
- -  17/Dec/2023:15:49:53 +0000 "GET /index.php" 204
- -  17/Dec/2023:15:49:53 +0000 "GET /index.php" 204
- -  17/Dec/2023:15:49:53 +0000 "GET /index.php" 204
- -  17/Dec/2023:15:49:53 +0000 "GET /index.php" 204
- -  17/Dec/2023:15:49:53 +0000 "GET /index.php" 204
- -  17/Dec/2023:15:53:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:15:53:18 +0000 "HEAD /index.php" 200
[17-Dec-2023 16:03:01] WARNING: [pool www] server reached pm.max_children setting (120), consider raising it
- -  17/Dec/2023:16:03:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:16:03:19 +0000 "HEAD /index.php" 200
- -  17/Dec/2023:16:05:59 +0000 "REPORT /remote.php" 401
- - User1 17/Dec/2023:16:05:59 +0000 "REPORT /remote.php" 207
- -  17/Dec/2023:16:06:07 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:16:06:07 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:16:06:08 +0000 "PROPFIND /remote.php" 207
- -  17/Dec/2023:16:13:18 +0000 "HEAD /index.php" 302
- -  17/Dec/2023:16:13:19 +0000 "HEAD /index.php" 200
[18-Dec-2023 02:58:02] NOTICE: Finishing ...
[18-Dec-2023 02:58:02] NOTICE: exiting, bye-bye!

Additional info

Webserver is Caddy 2.1.1 with hundreds of error messages like this indicating an upstream error:

ERR ts=1702868282.5951242 logger=http.log.error msg=EOF request={"remote_ip":"89.0.53.193","remote_port":"47455","client_ip":"XXX","proto":"HTTP/1.1","method":"GET","host":"nc.example.com","uri":"/index.php/204","headers":{"Accept":["*/*"],"Cookie":[],"Connection":["Keep-Alive"],"Accept-Language":["de-DE,en,*"],"Authorization":[],"Accept-Encoding":["gzip, deflate"],"User-Agent":["Mozilla/5.0 (Windows) mirall/3.11.0stable-Win64 (build 20231211) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)"],"X-Request-Id":["b583ae60-4c5b-4d1b-8224-2813b0b42d62"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"nc.example.com"}} duration=31924.128253927 status=502 err_id=azhufdqiz err_trace=reverseproxy.statusError (reverseproxy.go:1248)
Virusmater commented 4 months ago

Having the same issue, nexcloud doesn't lock up, but starting to use too much CPU. Trying to truncate the oc_filecache now and resync everything.