Closed marcello-dev closed 4 months ago
As @szaimen suggested, removing the file daily_backup_running
in the mastercontainer volume inside the data folder allows to cancel the backup. Then reloading AIO dashboard should show the controls again.
After that I restarted the containers and Nextcloud came back online.
The actual backup only took 1 minute to complete, but for some reason something got stuck and Nextcloud stayed offline for hours.
This issue can be closed if you want.
Thanks for the report! My idea is to add a warning to the AIO interface what to do if it should be stuck.
My backup got stuck last night as well. First time this has happened in a few months.
I also encountered this. New installation, first backup went without a problem, secont time it got stuck. I manually killed borg container, deleted whole borg repo and retried from the start with initial backup and after it finished successfully I tried changing 1 file in nextcloud and tried creating new backup manually and it's stuck unfortunately.
Borg container:
Performing backup...
Starting the backup...
Creating archive at "/mnt/borgbackup/borg::20240604_085132-nextcloud-aio"
I see that the borg is doing "something" but it's not doing much, during initial backup it was using much more IO around 60Mb read/writes. My backup is 1.2tb so with 400kb it will take approximately forever to scan it at this rate.
EDIT: Okay so the backup finished but it's extremely slow. I don't think 1h15m backup every day to backup 100mb of changed files is warranted.
Performing backup...
Starting the backup...
Creating archive at "/mnt/borgbackup/borg::20240604_085132-nextcloud-aio"
------------------------------------------------------------------------------
Repository: /mnt/borgbackup/borg
Archive name: 20240604_085132-nextcloud-aio
Archive fingerprint: 6427d553db9d62893e4064693cae2577187ab908421742fd1f2d6dcf648ceb3b
Time (start): Tue, 2024-06-04 08:51:33
Time (end): Tue, 2024-06-04 10:07:01
Duration: 1 hours 15 minutes 28.08 seconds
Number of files: 699133
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 1.20 TB 1.17 TB 103.17 MB
All archives: 2.39 TB 2.33 TB 1.13 TB
Unique chunks Total chunks
Chunk index: 1087923 2278692
------------------------------------------------------------------------------
BORG_PRUNE_OPTS are --keep-within=7d --keep-weekly=4
Pruning the archives...
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
Deleted data: 0 B 0 B 0 B
All archives: 2.39 TB 2.33 TB 1.13 TB
Unique chunks Total chunks
Chunk index: 1087923 2278692
------------------------------------------------------------------------------
Compacting the archives...
Backup finished successfully on 04.06.2024 - 10:07:10 (01 hours 15 minutes 38 seconds).
Is there a way to disable backup of data directory in this case? It's extremely slow compared to ZFS snapshots.
EDIT2:
Performing backup...
Starting the backup...
Creating archive at "/mnt/borgbackup/borg::20240604_210138-nextcloud-aio"
------------------------------------------------------------------------------
Repository: /mnt/borgbackup/borg
Archive name: 20240604_210138-nextcloud-aio
Archive fingerprint: 1772546d0652f93a3367eeb37d7340321ba19a457cba38ebf9b2b054a44e512c
Time (start): Tue, 2024-06-04 21:01:39
Time (end): Tue, 2024-06-04 22:17:55
Duration: 1 hours 16 minutes 15.25 seconds
Number of files: 699203
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 1.20 TB 1.17 TB 98.04 MB
All archives: 3.59 TB 3.50 TB 1.13 TB
Unique chunks Total chunks
Chunk index: 1088473 3416416
------------------------------------------------------------------------------
BORG_PRUNE_OPTS are --keep-within=7d --keep-weekly=4
Pruning the archives...
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
Deleted data: 0 B 0 B 0 B
All archives: 3.59 TB 3.50 TB 1.13 TB
Unique chunks Total chunks
Chunk index: 1088473 3416416
------------------------------------------------------------------------------
Compacting the archives...
Backup finished successfully on 04.06.2024 - 22:18:03 (01 hours 16 minutes 25 seconds).
Another 100mb backup took an 1h15m. Is there something wrong here? Shouldn't it be faster?
The system wasnt under any significant load nor it isn't that slow:
Gigabyte Z690 UD
Core i3-12100
Kingston Fury Beast, DDR5, 32 GB, 5200MHz
4x20tb HDD
I am not sure if this is a separate issue or belongs here:
My backup isn't stuck, but my AIO interface shows that it's running even though it has finished. I only noticed this after a reboot when I saw that the nextcloud containers weren't running. Restarting the mastercontainer or the backup container didn't help (the backup ran through, but the containers didn't start). The AIO interface still shows the backup is running. I manually started the nextcloud containers and it's working fine apart from the AIO interface:
Mastercontainer log:
Trying to fix docker.sock permissions internally...
Creating docker group internally with id 988
[0;92mInitial startup of Nextcloud All-in-One complete!
You should be able to open the Nextcloud AIO Interface now on port 8080 of this server!
E.g. https://internal.ip.of.this.server:8080
⚠️ Important: do always use an ip-address if you access this port and not a domain as HSTS might block access to it later!
If your server has port 80 and 8443 open and you point a domain to your server, you can get a valid certificate automatically by opening the Nextcloud AIO Interface via:
https://your-domain-that-points-to-this-server.tld:8443[0m
{"level":"info","ts":1717676142.3378007,"msg":"using provided configuration","config_file":"/Caddyfile","config_adapter":""}
[06-Jun-2024 12:15:42] NOTICE: fpm is running, pid 161
[06-Jun-2024 12:15:42] NOTICE: ready to handle connections
[Thu Jun 06 12:15:42.345958 2024] [mpm_event:notice] [pid 155:tid 140267496704776] AH00489: Apache/2.4.59 (Unix) OpenSSL/3.1.5 configured -- resuming normal operations
[Thu Jun 06 12:15:42.346007 2024] [core:notice] [pid 155:tid 140267496704776] AH00094: Command line: 'httpd -D FOREGROUND'
NOTICE: PHP message: 404 Not Found
Type: Slim\Exception\HttpNotFoundException
4
Message: Not found.
File: /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/RoutingMiddleware.php
Line: 76
Trace: #0 /var/www/docker-aio/php/vendor/slim/slim/Slim/Routing/RouteRunner.php(56): Slim\Middleware\RoutingMiddleware->performRouting(Object(GuzzleHttp\Psr7\ServerRequest))
#1 /var/www/docker-aio/php/vendor/slim/csrf/src/Guard.php(482): Slim\Routing\RouteRunner->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#2 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(168): Slim\Csrf\Guard->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Slim\Routing\RouteRunner))
#3 /var/www/docker-aio/php/vendor/slim/twig-view/src/TwigMiddleware.php(115): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#4 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Views\TwigMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#5 /var/www/docker-aio/php/src/Middleware/AuthMiddleware.php(38): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#6 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(269): AIO\Middleware\AuthMiddleware->__invoke(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#7 /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/ErrorMiddleware.php(76): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#8 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Middleware\ErrorMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#9 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(65): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#10 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(199): Slim\MiddlewareDispatcher->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#11 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(183): Slim\App->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#12 /var/www/docker-aio/php/public/index.php(185): Slim\App->run()
#13 {main}
Tips: To display error details in HTTP response set "displayErrorDetails" to true in the ErrorHandler constructor.
NOTICE: PHP message: 404 Not Found
Type: Slim\Exception\HttpNotFoundException
4
Message: Not found.
File: /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/RoutingMiddleware.php
Line: 76
Trace: #0 /var/www/docker-aio/php/vendor/slim/slim/Slim/Routing/RouteRunner.php(56): Slim\Middleware\RoutingMiddleware->performRouting(Object(GuzzleHttp\Psr7\ServerRequest))
#1 /var/www/docker-aio/php/vendor/slim/csrf/src/Guard.php(482): Slim\Routing\RouteRunner->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#2 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(168): Slim\Csrf\Guard->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Slim\Routing\RouteRunner))
#3 /var/www/docker-aio/php/vendor/slim/twig-view/src/TwigMiddleware.php(115): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#4 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Views\TwigMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#5 /var/www/docker-aio/php/src/Middleware/AuthMiddleware.php(38): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#6 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(269): AIO\Middleware\AuthMiddleware->__invoke(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#7 /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/ErrorMiddleware.php(76): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#8 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Middleware\ErrorMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#9 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(65): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#10 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(199): Slim\MiddlewareDispatcher->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#11 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(183): Slim\App->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#12 /var/www/docker-aio/php/public/index.php(185): Slim\App->run()
#13 {main}
Tips: To display error details in HTTP response set "displayErrorDetails" to true in the ErrorHandler constructor.
I am using openmediavault 7 / Debian Bookworm. There was a Docker/containerd update yesterday. Could this be related in any way?
Edit: Removing the file daily_backup_running
did the trick for me as well. The next backup didn't get stuck and I got an update notification and succesfully updated Nextcloud from 28.0.5.1 to 28.0.6.1. I still have no idea what caused this problem though...
Thanks for the report! My idea is to add a warning to the AIO interface what to do if it should be stuck.
This is now released with v9.0.0 Beta. Testing and feedback is welcome! See https://github.com/nextcloud/all-in-one#how-to-switch-the-channel
Hello, the backup process is stuck again. My AIO was automatically upgraded to 8.3.0, and the issue looks a bit different:
Logs:
Daily backup script has started
grep: write error: Broken pipe
Connection to nextcloud-aio-apache (172.18.0.10) 11000 port [tcp/*] succeeded!
Starting mastercontainer update...
(The script might get exited due to that. In order to update all the other containers correctly, you need to run this script with the same settings a second time.)
Waiting for watchtower to stop
Creating daily backup...
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
.....
Waiting for backup container to stop
Waiting for backup container to stop
Deleting duplicate sessions
Waiting for backup container to stop
NOTICE: PHP message: 404 Not Found
Type: Slim\Exception\HttpNotFoundException
4
Message: Not found.
File: /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/RoutingMiddleware.php
Line: 76
Trace: #0 /var/www/docker-aio/php/vendor/slim/slim/Slim/Routing/RouteRunner.php(56): Slim\Middleware\RoutingMiddleware->performRouting(Object(GuzzleHttp\Psr7\ServerRequest))
#1 /var/www/docker-aio/php/vendor/slim/csrf/src/Guard.php(482): Slim\Routing\RouteRunner->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#2 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(168): Slim\Csrf\Guard->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Slim\Routing\RouteRunner))
#3 /var/www/docker-aio/php/vendor/slim/twig-view/src/TwigMiddleware.php(115): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#4 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Views\TwigMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#5 /var/www/docker-aio/php/src/Middleware/AuthMiddleware.php(38): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#6 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(269): AIO\Middleware\AuthMiddleware->__invoke(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#7 /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/ErrorMiddleware.php(76): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#8 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Middleware\ErrorMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#9 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(65): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#10 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(199): Slim\MiddlewareDispatcher->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#11 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(183): Slim\App->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#12 /var/www/docker-aio/php/public/index.php(185): Slim\App->run()
#13 {main}
Tips: To display error details in HTTP response set "displayErrorDetails" to true in the ErrorHandler constructor.
Waiting for backup container to stop
Waiting for backup container to stop
Waiting for backup container to stop
...
I still don't have the feature to manually unblock the backup from the dashboard, because it was implemented for AIO v9. Maybe it is something to investigate? Also how to know if Borg is still doing something? I did upload some big files yesterday but it should not take that long.
New feature idea: it would be good to add a timestamp next to the log message for better troubleshooting.
Steps to reproduce
Expected behavior
The backup should start at 4am UTC and finish after a few minutes
Actual behavior
The backup is still ongoing at 2pm UTC (~10hours later)
Host OS
Ubuntu 22.04
Nextcloud AIO version
v8.2.1
Current channel
Can't see it, there is only this message:
Other valuable info
Not sure if relevant but the AIO data dir is on an external ssd, and the backup is stored in a second external drive. I checked the drives and they are fine. I also don't see rsync running, I don't know why the dashboard says backup in progress.
Master container logs:
Watch tower logs: