Not able to backup running LXC containers on Proxmox 8.2.2

Foxi352 commented 6 months ago

I upgraded to Proxmox 8.2.2 and latest version of DRBD and Linstor. DRBD is running on a ZFS pool. After some initial problems (drbd-dkms not yet compatible with kernel 6.8 e.t.c.) all seems to run stable.

But i can't do backups of running LXC containers. VM's seem OK. And stopped containers also do backup without any problems. But running LXC containers yield some problems about snapshots .... Not sure though if it is linstor-proxmox related or more of a drbd issue ?

Linstor:

ii  linstor-client                       1.22.1-1                            all          Linstor client command line tool
ii  linstor-common                       1.27.1-1                            all          DRBD distributed resource management utility
ii  linstor-controller                   1.27.1-1                            all          DRBD distributed resource management utility
ii  linstor-proxmox                      8.0.2-1                             all          DRBD distributed resource management utility
ii  linstor-satellite                    1.27.1-1                            all          DRBD distributed resource management utility
ii  python-linstor                       1.22.0-1                            all          Linstor python api library

DRBD:

ii  drbd-dkms                            9.2.9-1                             all          RAID 1 over TCP/IP for Linux module source
rc  drbd-reactor                         0.9.0-1                             amd64        Monitors DRBD resources via plugins.
ii  drbd-utils                           9.28.0-1                            amd64        RAID 1 over TCP/IP for Linux (user utilities)
ii  drbdtop                              0.2.3-1                             amd64        like top, but for drbd

Kernel: Linux pve2 6.8.4-3-pve LINBIT/linstor-proxmox#1 SMP PREEMPT_DYNAMIC PMX 6.8.4-3 (2024-05-02T11:55Z) x86_64 GNU/Linux

Error log

INFO: starting new backup job: vzdump 210 --mode snapshot --storage Nas1 --notes-template '{{guestname}}' --notification-mode auto --remove 0 --node pve2
INFO: Starting Backup of VM 210 (lxc)
INFO: Backup started at 2024-05-08 10:16:36
INFO: status = running
INFO: CT Name: atm9
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 210 failed - API Return-Code: 500. Message: Could not restore snapshot snap_pm-dc477ac9_vzdump of pm-dc477ac9 to new snap_pm-dc477ac9_vzdump, because:
ERROR: [{"ret_code":20185089,"message":"Resource 'snap_pm-dc477ac9_vzdump' restored from resource 'pm-dc477ac9', snapshot 'snap_pm-dc477ac9_vzdump'.","details":"Resource UUIDs: 4e325662-2b1e-4460-93f7-67467c4aafa9","obj_refs":{"RscDfn":"snap_pm-dc477ac9_vzdump"},"created_at":"2024-05-08T10:16:40.58354495+02:00"},{"ret_code":-4611686018409823258,"message":"(Node: 'pve2') Generated resource file for resource 'snap_pm-dc477ac9_vzdump' is invalid.","cause":"Verification of resource file failed","details":"The error reported by the runtime environment or operating system is:\nThe external command 'drbdadm' exited with error code 10\n","error_report_ids":["663A2241-F21CA-000052"],"obj_refs":{"RscDfn":"snap_pm-dc477ac9_vzdump"},"created_at":"2024-05-08T10:16:46.659076668+02:00"}]
ERROR:  at /usr/share/perl5/PVE/Storage/Custom/LINSTORPlugin.pm line 533.
ERROR:  PVE::Storage::Custom::LINSTORPlugin::activate_volume("PVE::Storage::Custom::LINSTORPlugin", "ssd_group", HASH(0x5fa72a785cf0), "pm-dc477ac9_210", "vzdump", HASH(0x5fa72aa34488)) called at /usr/share/perl5/PVE/Storage.pm line 1237
ERROR:  PVE::Storage::activate_volumes(HASH(0x5fa7225c34b0), ARRAY(0x5fa72aad3008), "vzdump") called at /usr/share/perl5/PVE/VZDump/LXC.pm line 241
ERROR:  PVE::VZDump::LXC::snapshot(PVE::VZDump::LXC=HASH(0x5fa72aa72168), HASH(0x5fa7225a8518), 210) called at /usr/share/perl5/PVE/VZDump.pm line 1162
ERROR:  eval {...} called at /usr/share/perl5/PVE/VZDump.pm line 940
ERROR:  PVE::VZDump::exec_backup_task(PVE::VZDump=HASH(0x5fa72a718a88), HASH(0x5fa7225a8518)) called at /usr/share/perl5/PVE/VZDump.pm line 1409
ERROR:  eval {...} called at /usr/share/perl5/PVE/VZDump.pm line 1404
ERROR:  PVE::VZDump::exec_backup(PVE::VZDump=HASH(0x5fa72a718a88), PVE::RPCEnvironment=HASH(0x5fa723cbb9d8), "swagener\@authentik") called at /usr/share/perl5/PVE/API2/VZDump.pm line 142
ERROR:  PVE::API2::VZDump::__ANON__("UPID:pve2:00199687:006CC85D:663B34E4:vzdump:210:swagener\@auth"...) called at /usr/share/perl5/PVE/RESTEnvironment.pm line 620
ERROR:  eval {...} called at /usr/share/perl5/PVE/RESTEnvironment.pm line 611
ERROR:  PVE::RESTEnvironment::fork_worker(PVE::RPCEnvironment=HASH(0x5fa723cbb9d8), "vzdump", 210, "swagener\@authentik", CODE(0x5fa72aafb9b8)) called at /usr/share/perl5/PVE/API2/VZDump.pm line 166
ERROR:  PVE::API2::VZDump::__ANON__(HASH(0x5fa72aae7ec0)) called at /usr/share/perl5/PVE/RESTHandler.pm line 499
ERROR:  PVE::RESTHandler::handle("PVE::API2::VZDump", HASH(0x5fa72905c308), HASH(0x5fa72aae7ec0)) called at /usr/share/perl5/PVE/HTTPServer.pm line 180
ERROR:  eval {...} called at /usr/share/perl5/PVE/HTTPServer.pm line 141
ERROR:  PVE::HTTPServer::rest_handler(PVE::HTTPServer=HASH(0x5fa72a64f580), "::ffff:192.168.253.2", "POST", "/nodes/pve2/vzdump", HASH(0x5fa72aa19898), HASH(0x5fa72aadea20), "extjs") called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 938
ERROR:  eval {...} called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 912
ERROR:  PVE::APIServer::AnyEvent::handle_api2_request(PVE::HTTPServer=HASH(0x5fa72a64f580), HASH(0x5fa72aad3050), HASH(0x5fa72aa19898), "POST", "/api2/extjs/nodes/pve2/vzdump") called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1162
ERROR:  eval {...} called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1154
ERROR:  PVE::APIServer::AnyEvent::handle_request(PVE::HTTPServer=HASH(0x5fa72a64f580), HASH(0x5fa72aad3050), HASH(0x5fa72aa19898), "POST", "/api2/extjs/nodes/pve2/vzdump") called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1622
ERROR:  PVE::APIServer::AnyEvent::__ANON__(AnyEvent::Handle=HASH(0x5fa72aab7f30), "notes-template=%7B%7Bguestname%7D%7D&vmid=210&notification-mo"...) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Handle.pm line 1505
ERROR:  AnyEvent::Handle::__ANON__(AnyEvent::Handle=HASH(0x5fa72aab7f30)) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Handle.pm line 1315
ERROR:  AnyEvent::Handle::_drain_rbuf(AnyEvent::Handle=HASH(0x5fa72aab7f30)) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Handle.pm line 2015
ERROR:  AnyEvent::Handle::__ANON__() called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Loop.pm line 248
ERROR:  AnyEvent::Loop::one_event() called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Impl/Perl.pm line 46
ERROR:  AnyEvent::CondVar::Base::_wait(AnyEvent::CondVar=HASH(0x5fa723c90768)) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent.pm line 2034
ERROR:  AnyEvent::CondVar::Base::recv(AnyEvent::CondVar=HASH(0x5fa723c90768)) called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1936
ERROR:  PVE::APIServer::AnyEvent::run(PVE::HTTPServer=HASH(0x5fa72a64f580)) called at /usr/share/perl5/PVE/Service/pvedaemon.pm line 52
ERROR:  PVE::Service::pvedaemon::run(PVE::Service::pvedaemon=HASH(0x5fa72a6492c0)) called at /usr/share/perl5/PVE/Daemon.pm line 171
ERROR:  eval {...} called at /usr/share/perl5/PVE/Daemon.pm line 171
ERROR:  PVE::Daemon::__ANON__(PVE::Service::pvedaemon=HASH(0x5fa72a6492c0)) called at /usr/share/perl5/PVE/Daemon.pm line 390
ERROR:  eval {...} called at /usr/share/perl5/PVE/Daemon.pm line 379
ERROR:  PVE::Daemon::__ANON__(PVE::Service::pvedaemon=HASH(0x5fa72a6492c0), undef) called at /usr/share/perl5/PVE/Daemon.pm line 551
ERROR:  eval {...} called at /usr/share/perl5/PVE/Daemon.pm line 549
ERROR:  PVE::Daemon::start(PVE::Service::pvedaemon=HASH(0x5fa72a6492c0), undef) called at /usr/share/perl5/PVE/Daemon.pm line 659
ERROR:  PVE::Daemon::__ANON__(HASH(0x5fa7225a25c0)) called at /usr/share/perl5/PVE/RESTHandler.pm line 499
ERROR:  PVE::RESTHandler::handle("PVE::Service::pvedaemon", HASH(0x5fa72a649608), HASH(0x5fa7225a25c0), 1) called at /usr/share/perl5/PVE/RESTHandler.pm line 985
ERROR:  eval {...} called at /usr/share/perl5/PVE/RESTHandler.pm line 968
ERROR:  PVE::RESTHandler::cli_handler("PVE::Service::pvedaemon", "pvedaemon start", "start", ARRAY(0x5fa7225c9610), ARRAY(0x5fa7225c29e8), undef, undef, undef) called at /usr/share/perl5/PVE/CLIHandler.pm line 594
ERROR:  PVE::CLIHandler::__ANON__(ARRAY(0x5fa7225a2788), CODE(0x5fa7229b4ba8), undef) called at /usr/share/perl5/PVE/CLIHandler.pm line 673
ERROR:  PVE::CLIHandler::run_cli_handler("PVE::Service::pvedaemon", "prepare", CODE(0x5fa7229b4ba8)) called at /usr/bin/pvedaemon line 27
INFO: Failed at 2024-05-08 10:16:47
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

ghernadi commented 6 months ago

There was an error from LINSTOR:

[{...} ,
{"ret_code":-4611686018409823258,"message":"(Node: 'pve2') Generated resource file for resource 'snap_pm-dc477ac9_vzdump' is invalid.","cause":"Verification of resource file failed","details":"The error reported by the runtime environment or operating system is:\nThe external command 'drbdadm' exited with error code 10\n","error_report_ids":["663A2241-F21CA-000052"],"obj_refs":{"RscDfn":"snap_pm-dc477ac9_vzdump"},"created_at":"2024-05-08T10:16:46.659076668+02:00"}]

Can you show us linstor err show 663A2241-F21CA-000052 and possibly also the files /var/lib/linstor.d/snap_pm-dc477ac9_vzdump.res as well as /var/lib/linstor.d/snap_pm-dc477ac9_vzdump.res_tmp (if exists) from the node pve2? Also if /var/lib/linstor.d/.backup contains a file with this resource name, that would also be interesting. Alternatively feel free to send me an SOS report (linstor sos dl) via email.

Foxi352 commented 6 months ago

I just sent you the sos via email. Is that enough, or do you still need one of the files you asked here ?

ghernadi commented 6 months ago

No, that should not be necessary. I found the ErrorReport, but strangely neither snap_pm-dc477ac9_vzdump nor the other resource the generated .res file was conflicting with...

Can you please restart the satellite on pve2 and see if this problem persists?

Foxi352 commented 6 months ago

EDIT: Restarting satellite does not bring any result

Both exist:

/var/lib/linstor.d

-rw-r--r--  1 root root  916 May  7 14:45 pm-dc477ac9.res
-rw-r--r--  1 root root  826 May  8 10:16 snap_pm-dc477ac9_vzdump.res_tmp

cat pm-dc477ac9.res

# This file was generated by linstor(1.27.1), do not edit manually.
# Local node: pve2
# Host name : pve2

resource "pm-dc477ac9"
{

    options
    {
        quorum off;
    }

    net
    {
        cram-hmac-alg     sha1;
        shared-secret     "3zLn8FPpaZVYebk7zPpy";
        after-sb-0pri discard-zero-changes;
        after-sb-1pri discard-secondary;
        after-sb-2pri disconnect;
        allow-two-primaries yes;
        max-buffers 40960;
        rcvbuf-size 10485760;
        sndbuf-size 10485760;
        verify-alg "crct10dif";
    }

    on "pve2"
    {
        volume 0
        {
            disk        /dev/zvol/ssd/pm-dc477ac9_00000;
            disk
            {
                discard-zeroes-if-aligned yes;
                rs-discard-granularity 16384;
            }
            meta-disk   internal;
            device      minor 1022;
        }
        node-id    0;
    }
}

cat snap_pm-dc477ac9_vzdump.res_tmp

# This file was generated by linstor(1.27.1), do not edit manually.
# Local node: pve2
# Host name : pve2

resource "snap_pm-dc477ac9_vzdump"
{

    options
    {
        quorum off;
    }

    net
    {
        after-sb-0pri discard-zero-changes;
        after-sb-1pri discard-secondary;
        after-sb-2pri disconnect;
        allow-two-primaries yes;
        max-buffers 40960;
        rcvbuf-size 10485760;
        sndbuf-size 10485760;
    }

    on "pve2"
    {
        volume 0
        {
            disk        /dev/zvol/ssd/snap_pm-dc477ac9_vzdump_00000;
            disk
            {
                discard-zeroes-if-aligned yes;
                rs-discard-granularity 16384;
            }
            meta-disk   internal;
            device      minor 1010;
        }
        node-id    0;
    }
}

Foxi352 commented 6 months ago

Some additional information:

Restarted satellite, controller and later even PVE server. Problem still exists.
I can snapshot stopped and running containers via PVE web UI.
But i can only backup stopped containers. When trying to backup running containers the above mentioned error raises.

ghernadi commented 6 months ago

Sorry for the delay, I was already re-reading through this issue when your new reply popped up...

Since this is apparently reproducible for you, can you please enable TRACE logging (using linstor c setloglevel TRACE --global), reproduce this issue, turn TRACE off again (same command as before but with INFO this time) and send me another SOS report?

Foxi352 commented 6 months ago

It is indeed reproductible for all running LXC containers since the update. SOS report with TRACE log enabled sent via mail.

INFO: starting new backup job: vzdump 218 --node pve2 --remove 0 --notification-mode auto --notes-template '{{guestname}}' --storage Nas1 --mode snapshot
INFO: Starting Backup of VM 218 (lxc)
INFO: Backup started at 2024-05-15 10:24:30
INFO: status = running
INFO: CT Name: nicolas-01
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 218 failed - API Return-Code: 500. Message: Could not restore snapshot snap_pm-a38b884b_vzdump of pm-a38b884b to new snap_pm-a38b884b_vzdump, because:
ERROR: [{"ret_code":20185089,"message":"Resource 'snap_pm-a38b884b_vzdump' restored from resource 'pm-a38b884b', snapshot 'snap_pm-a38b884b_vzdump'.","details":"Resource UUIDs: 1cab605c-9885-4bc3-bff3-500c88f20296","obj_refs":{"RscDfn":"snap_pm-a38b884b_vzdump"},"created_at":"2024-05-15T10:24:33.744604046+02:00"},{"ret_code":-4611686018409823258,"message":"(Node: 'pve2') Failed to adjust DRBD resource snap_pm-a38b884b_vzdump","error_report_ids":["663B681D-F21CA-000008"],"obj_refs":{"RscDfn":"snap_pm-a38b884b_vzdump"},"created_at":"2024-05-15T10:24:39.510601033+02:00"}]
ERROR:  at /usr/share/perl5/PVE/Storage/Custom/LINSTORPlugin.pm line 533.
ERROR:  PVE::Storage::Custom::LINSTORPlugin::activate_volume("PVE::Storage::Custom::LINSTORPlugin", "ssd_group", HASH(0x5fa72a7857f0), "pm-a38b884b_218", "vzdump", HASH(0x5fa72a7f8da8)) called at /usr/share/perl5/PVE/Storage.pm line 1237
ERROR:  PVE::Storage::activate_volumes(HASH(0x5fa72a701bf8), ARRAY(0x5fa7244c7320), "vzdump") called at /usr/share/perl5/PVE/VZDump/LXC.pm line 241
ERROR:  PVE::VZDump::LXC::snapshot(PVE::VZDump::LXC=HASH(0x5fa72a785b08), HASH(0x5fa72a718c90), 218) called at /usr/share/perl5/PVE/VZDump.pm line 1162
ERROR:  eval {...} called at /usr/share/perl5/PVE/VZDump.pm line 940
ERROR:  PVE::VZDump::exec_backup_task(PVE::VZDump=HASH(0x5fa7225a8398), HASH(0x5fa72a718c90)) called at /usr/share/perl5/PVE/VZDump.pm line 1409
ERROR:  eval {...} called at /usr/share/perl5/PVE/VZDump.pm line 1404
ERROR:  PVE::VZDump::exec_backup(PVE::VZDump=HASH(0x5fa7225a8398), PVE::RPCEnvironment=HASH(0x5fa723cbb960), "swagener\@authentik") called at /usr/share/perl5/PVE/API2/VZDump.pm line 142
ERROR:  PVE::API2::VZDump::__ANON__("UPID:pve2:003D87FC:04085B5A:6644713E:vzdump:218:swagener\@auth"...) called at /usr/share/perl5/PVE/RESTEnvironment.pm line 620
ERROR:  eval {...} called at /usr/share/perl5/PVE/RESTEnvironment.pm line 611
ERROR:  PVE::RESTEnvironment::fork_worker(PVE::RPCEnvironment=HASH(0x5fa723cbb960), "vzdump", 218, "swagener\@authentik", CODE(0x5fa72a825ef0)) called at /usr/share/perl5/PVE/API2/VZDump.pm line 166
ERROR:  PVE::API2::VZDump::__ANON__(HASH(0x5fa72a7f90d8)) called at /usr/share/perl5/PVE/RESTHandler.pm line 499
ERROR:  PVE::RESTHandler::handle("PVE::API2::VZDump", HASH(0x5fa72905c308), HASH(0x5fa72a7f90d8)) called at /usr/share/perl5/PVE/HTTPServer.pm line 180
ERROR:  eval {...} called at /usr/share/perl5/PVE/HTTPServer.pm line 141
ERROR:  PVE::HTTPServer::rest_handler(PVE::HTTPServer=HASH(0x5fa72a64f580), "::ffff:192.168.253.2", "POST", "/nodes/pve2/vzdump", HASH(0x5fa72a7f8538), HASH(0x5fa72a7b12f0), "extjs") called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 938
ERROR:  eval {...} called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 912
ERROR:  PVE::APIServer::AnyEvent::handle_api2_request(PVE::HTTPServer=HASH(0x5fa72a64f580), HASH(0x5fa72a7f87a8), HASH(0x5fa72a7f8538), "POST", "/api2/extjs/nodes/pve2/vzdump") called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1162
ERROR:  eval {...} called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1154
ERROR:  PVE::APIServer::AnyEvent::handle_request(PVE::HTTPServer=HASH(0x5fa72a64f580), HASH(0x5fa72a7f87a8), HASH(0x5fa72a7f8538), "POST", "/api2/extjs/nodes/pve2/vzdump") called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1622
ERROR:  PVE::APIServer::AnyEvent::__ANON__(AnyEvent::Handle=HASH(0x5fa72a7b8f20), "notes-template=%7B%7Bguestname%7D%7D&storage=Nas1&vmid=218&no"...) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Handle.pm line 1505
ERROR:  AnyEvent::Handle::__ANON__(AnyEvent::Handle=HASH(0x5fa72a7b8f20)) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Handle.pm line 1315
ERROR:  AnyEvent::Handle::_drain_rbuf(AnyEvent::Handle=HASH(0x5fa72a7b8f20)) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Handle.pm line 2015
ERROR:  AnyEvent::Handle::__ANON__() called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Loop.pm line 248
ERROR:  AnyEvent::Loop::one_event() called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Impl/Perl.pm line 46
ERROR:  AnyEvent::CondVar::Base::_wait(AnyEvent::CondVar=HASH(0x5fa723c90768)) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent.pm line 2034
ERROR:  AnyEvent::CondVar::Base::recv(AnyEvent::CondVar=HASH(0x5fa723c90768)) called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1936
ERROR:  PVE::APIServer::AnyEvent::run(PVE::HTTPServer=HASH(0x5fa72a64f580)) called at /usr/share/perl5/PVE/Service/pvedaemon.pm line 52
ERROR:  PVE::Service::pvedaemon::run(PVE::Service::pvedaemon=HASH(0x5fa72a6492c0)) called at /usr/share/perl5/PVE/Daemon.pm line 171
ERROR:  eval {...} called at /usr/share/perl5/PVE/Daemon.pm line 171
ERROR:  PVE::Daemon::__ANON__(PVE::Service::pvedaemon=HASH(0x5fa72a6492c0)) called at /usr/share/perl5/PVE/Daemon.pm line 390
ERROR:  eval {...} called at /usr/share/perl5/PVE/Daemon.pm line 379
ERROR:  PVE::Daemon::__ANON__(PVE::Service::pvedaemon=HASH(0x5fa72a6492c0), undef) called at /usr/share/perl5/PVE/Daemon.pm line 551
ERROR:  eval {...} called at /usr/share/perl5/PVE/Daemon.pm line 549
ERROR:  PVE::Daemon::start(PVE::Service::pvedaemon=HASH(0x5fa72a6492c0), undef) called at /usr/share/perl5/PVE/Daemon.pm line 659
ERROR:  PVE::Daemon::__ANON__(HASH(0x5fa7225a25c0)) called at /usr/share/perl5/PVE/RESTHandler.pm line 499
ERROR:  PVE::RESTHandler::handle("PVE::Service::pvedaemon", HASH(0x5fa72a649608), HASH(0x5fa7225a25c0), 1) called at /usr/share/perl5/PVE/RESTHandler.pm line 985
ERROR:  eval {...} called at /usr/share/perl5/PVE/RESTHandler.pm line 968
ERROR:  PVE::RESTHandler::cli_handler("PVE::Service::pvedaemon", "pvedaemon start", "start", ARRAY(0x5fa7225c9610), ARRAY(0x5fa7225c29e8), undef, undef, undef) called at /usr/share/perl5/PVE/CLIHandler.pm line 594
ERROR:  PVE::CLIHandler::__ANON__(ARRAY(0x5fa7225a2788), CODE(0x5fa7229b4ba8), undef) called at /usr/share/perl5/PVE/CLIHandler.pm line 673
ERROR:  PVE::CLIHandler::run_cli_handler("PVE::Service::pvedaemon", "prepare", CODE(0x5fa7229b4ba8)) called at /usr/bin/pvedaemon line 27
INFO: Failed at 2024-05-15 10:24:40
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

ghernadi commented 6 months ago

Thanks for the SOS report. The logs show that LINSTOR tries to create a new resource (I believe this is just a temporary resource used by the proxmox plugin, which I am not entirely familiar with its implementation), but that new resource creation fails due to (in the SOS report in _Controller/log-syslog):

2024-05-15T10:24:39.250687+02:00 pve2 drbdsetup: new-minor snap_pm-a38b884b_vzdump 1010 0: sysfs node '/sys/devices/virtual/block/drbd1010' (already? still?) exists

That minor number is however not allocated by any resource in the LINSTOR database. So I have also checked pve2/drbd-events2 what DRBD resources exist and compared them to the LINSTOR database and found the following:

resources (with 1 volume each) that are unknown to LINSTOR:
- vm-201-disk-1, using minor 1030
- vm-206-disk-2, using minor 1017
- vm-211-disk-2, using minor 1010
- vm-217-disk-1, using minor 1034
- vm-241-disk-1, using minor 1037
resources (with no volumes at all) also unknown to LINSTOR:
- vm-111-disk-2
- vm-112-disk-1
- vm-203-disk-1
- vm-208-disk-1
- vm-212-disk-1
- vm-213-disk-1
- vm-216-disk-1
- vm-218-disk-1

The second list is "only" unnecessary, but does not really hurt. The first list is indeed a problem, since those resources being up and running do occupy a minor number without LINSTOR knowing about.

So my suggestion is to double check if you really no longer have those VMs in Proxmox registered and if so either use drbdsetup down $rsc to remove the orphaned DRBD resources, or reboot the pve2 node (if possibly, i.e. no VMs are actively running on that node, etc...)

I have only checked for orphaned DRBD resources on pve2, since that seems to cause the issues here, so you might want to check / compare also the other nodes and take action accordingly.

Hope this helps and fixes the issue.

Foxi352 commented 6 months ago

I had indeed some clutter on my server. Orphaned ZFS volumes not related to any drbd resource. drbd resources which were inconsistent although only existing on one host. drbd resources not (no longer) related to any pve VM.

I did a two hours cleanup comparing vm's and ct's with linstor resources down to drbd resources and down to zfs volumes. Now everything is neat and clean. I also downed the unused drbd resources and rebooted the host. As you suggested. All the minor's from your list now disappeared from /sys/devices/virtual/block

drbdadm status shows no inconsistencies and only correct and in use resources
zfs list shows corrent and really used volumes. zfs list -t snapshot only shows 1 snapshot for 1 CT which is correct and needed.
linstor rd l and linstor r l show only correct and needed definitiosn and resources

But this did not resolve the issue. Now the backup of running containers fails telling me that the vzdump resource definition already exists. But i wonder where ?

I did another trace log when trying to backup CT 201, in case it's needed.

ERROR: Backup of VM 201 failed - API Return-Code: 500. Message: Could not create cluster wide snapshot snap_vm-201-disk-2_vzdump of vm-201-disk-2, because:
ERROR: [{"ret_code":-4611686018409823742,"message":"A snapshot definition with the name 'snap_vm-201-disk-2_vzdump' already exists in resource definition 'vm-201-disk-2'.","cause":"The SnapshotDefinition already exists","details":"Resource: vm-201-disk-2, Snapshot: snap_vm-201-disk-2_vzdump","obj_refs":{"RscDfn":"vm-201-disk-2","Snapshot":"snap_vm-201-disk-2_vzdump"},"created_at":"2024-05-16T08:29:55.017663146+02:00"}]
ERROR:  at /usr/share/perl5/PVE/Storage/Custom/LINSTORPlugin.pm line 602.
ERROR:  PVE::Storage::Custom::LINSTORPlugin::volume_snapshot("PVE::Storage::Custom::LINSTORPlugin", HASH(0x5a34ff2d1c08), "ssd_group", "vm-201-disk-2", "vzdump") called at /usr/share/perl5/PVE/Storage.pm line 340
ERROR:  PVE::Storage::volume_snapshot(HASH(0x5a34ff31d7f8), "ssd_group:vm-201-disk-2", "vzdump") called at /usr/share/perl5/PVE/LXC/Config.pm line 211
ERROR:  PVE::LXC::Config::__snapshot_create_vol_snapshot("PVE::LXC::Config", 201, "rootfs", HASH(0x5a34ff2d2ff0), "vzdump") called at /usr/share/perl5/PVE/AbstractConfig.pm line 831
ERROR:  PVE::AbstractConfig::__ANON__("rootfs", HASH(0x5a34ff2d2ff0)) called at /usr/share/perl5/PVE/AbstractConfig.pm line 475
ERROR:  PVE::AbstractConfig::foreach_volume_full("PVE::LXC::Config", HASH(0x5a34f85d4518), undef, CODE(0x5a34f7a65c58)) called at /usr/share/perl5/PVE/AbstractConfig.pm line 484
ERROR:  PVE::AbstractConfig::foreach_volume("PVE::LXC::Config", HASH(0x5a34f85d4518), CODE(0x5a34f7a65c58)) called at /usr/share/perl5/PVE/AbstractConfig.pm line 833
ERROR:  eval {...} called at /usr/share/perl5/PVE/AbstractConfig.pm line 819
ERROR:  PVE::AbstractConfig::snapshot_create("PVE::LXC::Config", 201, "vzdump", 0, "vzdump backup snapshot") called at /usr/share/perl5/PVE/VZDump/LXC.pm line 225
ERROR:  PVE::VZDump::LXC::__ANON__() called at /usr/share/perl5/PVE/AbstractConfig.pm line 299
ERROR:  PVE::AbstractConfig::__ANON__() called at /usr/share/perl5/PVE/Tools.pm line 259
ERROR:  eval {...} called at /usr/share/perl5/PVE/Tools.pm line 259
ERROR:  PVE::Tools::lock_file_full("/run/lock/lxc/pve-config-201.lock", 10, 0, CODE(0x5a34ff15b738)) called at /usr/share/perl5/PVE/AbstractConfig.pm line 302
ERROR:  PVE::AbstractConfig::__ANON__("PVE::LXC::Config", 201, 10, 0, CODE(0x5a34ff0d92f0)) called at /usr/share/perl5/PVE/AbstractConfig.pm line 322
ERROR:  PVE::AbstractConfig::lock_config_full("PVE::LXC::Config", 201, 10, CODE(0x5a34ff0d92f0)) called at /usr/share/perl5/PVE/AbstractConfig.pm line 330
ERROR:  PVE::AbstractConfig::lock_config("PVE::LXC::Config", 201, CODE(0x5a34ff0d92f0)) called at /usr/share/perl5/PVE/VZDump/LXC.pm line 227
ERROR:  PVE::VZDump::LXC::snapshot(PVE::VZDump::LXC=HASH(0x5a34ff2a21e8), HASH(0x5a34ff031fa8), 201) called at /usr/share/perl5/PVE/VZDump.pm line 1162
ERROR:  eval {...} called at /usr/share/perl5/PVE/VZDump.pm line 940
ERROR:  PVE::VZDump::exec_backup_task(PVE::VZDump=HASH(0x5a34ff031eb8), HASH(0x5a34ff031fa8)) called at /usr/share/perl5/PVE/VZDump.pm line 1409
ERROR:  eval {...} called at /usr/share/perl5/PVE/VZDump.pm line 1404
ERROR:  PVE::VZDump::exec_backup(PVE::VZDump=HASH(0x5a34ff031eb8), PVE::RPCEnvironment=HASH(0x5a34f85da530), "root\@pam") called at /usr/share/perl5/PVE/API2/VZDump.pm line 142
ERROR:  PVE::API2::VZDump::__ANON__("UPID:pve2:0000AD0A:00017556:6645A7E2:vzdump:201:root\@pam:") called at /usr/share/perl5/PVE/RESTEnvironment.pm line 620
ERROR:  eval {...} called at /usr/share/perl5/PVE/RESTEnvironment.pm line 611
ERROR:  PVE::RESTEnvironment::fork_worker(PVE::RPCEnvironment=HASH(0x5a34f85da530), "vzdump", 201, "root\@pam", CODE(0x5a34ff2358b0)) called at /usr/share/perl5/PVE/API2/VZDump.pm line 166
ERROR:  PVE::API2::VZDump::__ANON__(HASH(0x5a34ff264d78)) called at /usr/share/perl5/PVE/RESTHandler.pm line 499
ERROR:  PVE::RESTHandler::handle("PVE::API2::VZDump", HASH(0x5a34fd9a2f78), HASH(0x5a34ff264d78)) called at /usr/share/perl5/PVE/HTTPServer.pm line 180
ERROR:  eval {...} called at /usr/share/perl5/PVE/HTTPServer.pm line 141
ERROR:  PVE::HTTPServer::rest_handler(PVE::HTTPServer=HASH(0x5a34f6ec7e90), "::ffff:192.168.9.3", "POST", "/nodes/pve2/vzdump", HASH(0x5a34ff2a2878), HASH(0x5a34ff264eb0), "extjs") called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 938
ERROR:  eval {...} called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 912
ERROR:  PVE::APIServer::AnyEvent::handle_api2_request(PVE::HTTPServer=HASH(0x5a34f6ec7e90), HASH(0x5a34ff2f8418), HASH(0x5a34ff2a2878), "POST", "/api2/extjs/nodes/pve2/vzdump") called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1162
ERROR:  eval {...} called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1154
ERROR:  PVE::APIServer::AnyEvent::handle_request(PVE::HTTPServer=HASH(0x5a34f6ec7e90), HASH(0x5a34ff2f8418), HASH(0x5a34ff2a2878), "POST", "/api2/extjs/nodes/pve2/vzdump") called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1622
ERROR:  PVE::APIServer::AnyEvent::__ANON__(AnyEvent::Handle=HASH(0x5a34ff2f8328), "remove=0&notification-mode=auto&notes-template=%7B%7Bguestnam"...) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Handle.pm line 1505
ERROR:  AnyEvent::Handle::__ANON__(AnyEvent::Handle=HASH(0x5a34ff2f8328)) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Handle.pm line 1315
ERROR:  AnyEvent::Handle::_drain_rbuf(AnyEvent::Handle=HASH(0x5a34ff2f8328)) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Handle.pm line 2015
ERROR:  AnyEvent::Handle::__ANON__() called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Loop.pm line 248
ERROR:  AnyEvent::Loop::one_event() called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Impl/Perl.pm line 46
ERROR:  AnyEvent::CondVar::Base::_wait(AnyEvent::CondVar=HASH(0x5a34f85af6c8)) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent.pm line 2034
ERROR:  AnyEvent::CondVar::Base::recv(AnyEvent::CondVar=HASH(0x5a34f85af6c8)) called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1936
ERROR:  PVE::APIServer::AnyEvent::run(PVE::HTTPServer=HASH(0x5a34f6ec7e90)) called at /usr/share/perl5/PVE/Service/pvedaemon.pm line 52
ERROR:  PVE::Service::pvedaemon::run(PVE::Service::pvedaemon=HASH(0x5a34fef67998)) called at /usr/share/perl5/PVE/Daemon.pm line 171
ERROR:  eval {...} called at /usr/share/perl5/PVE/Daemon.pm line 171
ERROR:  PVE::Daemon::__ANON__(PVE::Service::pvedaemon=HASH(0x5a34fef67998)) called at /usr/share/perl5/PVE/Daemon.pm line 390
ERROR:  eval {...} called at /usr/share/perl5/PVE/Daemon.pm line 379
ERROR:  PVE::Daemon::__ANON__(PVE::Service::pvedaemon=HASH(0x5a34fef67998), undef) called at /usr/share/perl5/PVE/Daemon.pm line 551
ERROR:  eval {...} called at /usr/share/perl5/PVE/Daemon.pm line 549
ERROR:  PVE::Daemon::start(PVE::Service::pvedaemon=HASH(0x5a34fef67998), undef) called at /usr/share/perl5/PVE/Daemon.pm line 659
ERROR:  PVE::Daemon::__ANON__(HASH(0x5a34f6ec14b0)) called at /usr/share/perl5/PVE/RESTHandler.pm line 499
ERROR:  PVE::RESTHandler::handle("PVE::Service::pvedaemon", HASH(0x5a34fef67ce0), HASH(0x5a34f6ec14b0), 1) called at /usr/share/perl5/PVE/RESTHandler.pm line 985
ERROR:  eval {...} called at /usr/share/perl5/PVE/RESTHandler.pm line 968
ERROR:  PVE::RESTHandler::cli_handler("PVE::Service::pvedaemon", "pvedaemon start", "start", ARRAY(0x5a34f6ee8350), ARRAY(0x5a34fef68388), undef, undef, undef) called at /usr/share/perl5/PVE/CLIHandler.pm line 594
ERROR:  PVE::CLIHandler::__ANON__(ARRAY(0x5a34f6ec1678), CODE(0x5a34f72d33e8), undef) called at /usr/share/perl5/PVE/CLIHandler.pm line 673
ERROR:  PVE::CLIHandler::run_cli_handler("PVE::Service::pvedaemon", "prepare", CODE(0x5a34f72d33e8)) called at /usr/bin/pvedaemon line 27

ghernadi commented 6 months ago

From this error message:

{"ret_code":-4611686018409823742,"message":"A snapshot definition with the name 'snap_vm-201-disk-2_vzdump' already exists in resource definition 'vm-201-disk-2'.","cause":"The SnapshotDefinition already exists","details":"Resource: vm-201-disk-2, Snapshot: snap_vm-201-disk-2_vzdump","obj_refs":{"RscDfn":"vm-201-disk-2","Snapshot":"snap_vm-201-disk-2_vzdump"},"created_at":"2024-05-16T08:29:55.017663146+02:00"}]

I'd guess that the previous error had some left-overs that are hard to detect for automated cleanup. Therefore I'd suggest that you manually remove that snapshot definition (using linstor s d vm-201-disk-2 snap_vm-201-disk-2_vzdump) and retry the process.

Foxi352 commented 6 months ago

Jesus .... so many different places to look for all these definitions, resources, volumes e.t.c. All related to one vm's disk or snapshot :-)

You were right. Had 6 incompleted snapshots in Linstor probably due to the problems and the tests. After removing all except the one real existing snapshot, the backup went through.

Thank you very very much for your support and your patience. Sorry to have created an issue when finally it was more of a support case.

jesulo commented 1 month ago

Jesus .... so many different places to look for all these definitions, resources, volumes e.t.c. All related to one vm's disk or snapshot :-)

You were right. Had 6 incompleted snapshots in Linstor probably due to the problems and the tests. After removing all except the one real existing snapshot, the backup went through.

Thank you very very much for your support and your patience. Sorry to have created an issue when finally it was more of a support case.

I'm getting a disk space error but I have 190 GB free... I don't know exactly where it's trying to save or if I need to tweak settings in ZFS... INFO: starting new backup job: vzdump 101 --mode snapshot --notification-mode auto --node proxmox1 --storage PBS --notes-template '{{guestname}}' --remove 0 INFO: Starting Backup of VM 101 (lxc) INFO: Backup started at 2024-10-18 18:30:08 INFO: status = running INFO: CT Name: docker INFO: including mount point rootfs ('/') in backup INFO: backup mode: snapshot INFO: ionice priority: 7 INFO: create storage snapshot 'vzdump' INFO: cleanup temporary 'vzdump' snapshot ERROR: Backup of VM 101 failed - API Return-Code: 500. Message: Could not restore snapshot snap_pm-6ae4d75b_vzdump of pm-6ae4d75b to new snap_pm-6ae4d75b_vzdump, because: ERROR: [{"ret_code":-4611686018407202610,"message":"Not enough free space available for volume 0 of resource 'snap_pm-6ae4d75b_vzdump'.","details":"Resource: snap_pm-6ae4d75b_vzdump","error_report_ids":["6712BBF1-00000-000005"],"obj_refs":{"RscDfn":"snap_pm-6ae4d75b_vzdump"},"created_at":"2024-10-18T18:30:09.252863712-03:00"}] ERROR: at /usr/share/perl5/PVE/Storage/Custom/LINSTORPlugin.pm line 550. ERROR: PVE::Storage::Custom::LINSTORPlugin::activate_volume("PVE::Storage::Custom::LINSTORPlugin", "linstor_storage", HASH(0x5b5f5f2fe130), "pm-6ae4d75b_101", "vzdump", HASH(0x5b5f5f2e2148)) called at /usr/share/perl5/PVE/Storage.pm line 1237 ERROR: PVE::Storage::activate_volumes(HASH(0x5b5f5886d338), ARRAY(0x5b5f5f395128), "vzdump") called at /usr/share/perl5/PVE/VZDump/LXC.pm line 241 ERROR: PVE::VZDump::LXC::snapshot(PVE::VZDump::LXC=HASH(0x5b5f5f2ee9b8), HASH(0x5b5f5f26d0d8), 101) called at /usr/share/perl5/PVE/VZDump.pm line 1162 ERROR: eval {...} called at /usr/share/perl5/PVE/VZDump.pm line 940 ERROR: PVE::VZDump::exec_backup_task(PVE::VZDump=HASH(0x5b5f5717b990), HASH(0x5b5f5f26d0d8)) called at /usr/share/perl5/PVE/VZDump.pm line 1409 ERROR: eval {...} called at /usr/share/perl5/PVE/VZDump.pm line 1404 ERROR: PVE::VZDump::exec_backup(PVE::VZDump=HASH(0x5b5f5717b990), PVE::RPCEnvironment=HASH(0x5b5f58873a00), "root@pam") called at /usr/share/perl5/PVE/API2/VZDump.pm line 142 ERROR: PVE::API2::VZDump::ANON("UPID:proxmox1:0008F2E9:009A4015:6712D360:vzdump:101:root@pam:") called at /usr/share/perl5/PVE/RESTEnvironment.pm line 620 ERROR: eval {...} called at /usr/share/perl5/PVE/RESTEnvironment.pm line 611 ERROR: PVE::RESTEnvironment::fork_worker(PVE::RPCEnvironment=HASH(0x5b5f58873a00), "vzdump", 101, "root@pam", CODE(0x5b5f5f2ee688)) called at /usr/share/perl5/PVE/API2/VZDump.pm line 166 ERROR: PVE::API2::VZDump::ANON(HASH(0x5b5f5f300158)) called at /usr/share/perl5/PVE/RESTHandler.pm line 499 ERROR: PVE::RESTHandler::handle("PVE::API2::VZDump", HASH(0x5b5f5dc7e638), HASH(0x5b5f5f300158)) called at /usr/share/perl5/PVE/HTTPServer.pm line 180 ERROR: eval {...} called at /usr/share/perl5/PVE/HTTPServer.pm line 141 ERROR: PVE::HTTPServer::rest_handler(PVE::HTTPServer=HASH(0x5b5f57160eb8), "::ffff:192.168.100.85", "POST", "/nodes/proxmox1/vzdump", HASH(0x5b5f5f2ef138), HASH(0x5b5f5f2fe2f8), "extjs") called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 938 ERROR: eval {...} called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 912 ERROR: PVE::APIServer::AnyEvent::handle_api2_request(PVE::HTTPServer=HASH(0x5b5f57160eb8), HASH(0x5b5f57bf9510), HASH(0x5b5f5f2ef138), "POST", "/api2/extjs/nodes/proxmox1/vzdump") called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1162 ERROR: eval {...} called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1154 ERROR: PVE::APIServer::AnyEvent::handle_request(PVE::HTTPServer=HASH(0x5b5f57160eb8), HASH(0x5b5f57bf9510), HASH(0x5b5f5f2ef138), "POST", "/api2/extjs/nodes/proxmox1/vzdump") called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1622 ERROR: PVE::APIServer::AnyEvent::ANON(AnyEvent::Handle=HASH(0x5b5f5f26d8d0), "notes-template=%7B%7Bguestname%7D%7D&remove=0&mode=snapshot&v"...) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Handle.pm line 1505 ERROR: AnyEvent::Handle::ANON(AnyEvent::Handle=HASH(0x5b5f5f26d8d0)) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Handle.pm line 1315 ERROR: AnyEvent::Handle::_drain_rbuf(AnyEvent::Handle=HASH(0x5b5f5f26d8d0)) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Handle.pm line 2015 ERROR: AnyEvent::Handle::ANON() called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Loop.pm line 248 ERROR: AnyEvent::Loop::one_event() called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Impl/Perl.pm line 46 ERROR: AnyEvent::CondVar::Base::_wait(AnyEvent::CondVar=HASH(0x5b5f58848798)) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent.pm line 2034 ERROR: AnyEvent::CondVar::Base::recv(AnyEvent::CondVar=HASH(0x5b5f58848798)) called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1936 ERROR: PVE::APIServer::AnyEvent::run(PVE::HTTPServer=HASH(0x5b5f57160eb8)) called at /usr/share/perl5/PVE/Service/pvedaemon.pm line 52 ERROR: PVE::Service::pvedaemon::run(PVE::Service::pvedaemon=HASH(0x5b5f5ed6b9d0)) called at /usr/share/perl5/PVE/Daemon.pm line 171 ERROR: eval {...} called at /usr/share/perl5/PVE/Daemon.pm line 171 ERROR: PVE::Daemon::ANON(PVE::Service::pvedaemon=HASH(0x5b5f5ed6b9d0)) called at /usr/share/perl5/PVE/Daemon.pm line 390 ERROR: eval {...} called at /usr/share/perl5/PVE/Daemon.pm line 379 ERROR: PVE::Daemon::ANON(PVE::Service::pvedaemon=HASH(0x5b5f5ed6b9d0), undef) called at /usr/share/perl5/PVE/Daemon.pm line 551 ERROR: eval {...} called at /usr/share/perl5/PVE/Daemon.pm line 549 ERROR: PVE::Daemon::start(PVE::Service::pvedaemon=HASH(0x5b5f5ed6b9d0), undef) called at /usr/share/perl5/PVE/Daemon.pm line 659 ERROR: PVE::Daemon::ANON(HASH(0x5b5f5715a5c0)) called at /usr/share/perl5/PVE/RESTHandler.pm line 499 ERROR: PVE::RESTHandler::handle("PVE::Service::pvedaemon", HASH(0x5b5f5ed6bd18), HASH(0x5b5f5715a5c0), 1) called at /usr/share/perl5/PVE/RESTHandler.pm line 985 ERROR: eval {...} called at /usr/share/perl5/PVE/RESTHandler.pm line 968 ERROR: PVE::RESTHandler::cli_handler("PVE::Service::pvedaemon", "pvedaemon start", "start", ARRAY(0x5b5f57181610), ARRAY(0x5b5f5717ada8), undef, undef, undef) called at /usr/share/perl5/PVE/CLIHandler.pm line 594 ERROR: PVE::CLIHandler::ANON(ARRAY(0x5b5f5715a788), CODE(0x5b5f5756c2d8), undef) called at /usr/share/perl5/PVE/CLIHandler.pm line 673 ERROR: PVE::CLIHandler::run_cli_handler("PVE::Service::pvedaemon", "prepare", CODE(0x5b5f5756c2d8)) called at /usr/bin/pvedaemon line 27 INFO: Failed at 2024-10-18 18:30:09 INFO: Backup job finished with errors INFO: notified via target mail-to-root TASK ERROR: job errors

......

LINBIT / linstor-server

Not able to backup running LXC containers on Proxmox 8.2.2 #409