RHEL 9 compatibility - Githubissues

cosecit commented 1 year ago

[root@123 dist]# dnf install virtnbdbackup-1.9-1.noarch.rpm Updating Subscription Management repositories. Last metadata expiration check: 2:55:47 ago on Mon 07 Nov 2022 05:07:37 PM CET. Error: Problem: conflicting requests

nothing provides python3-dataclasses needed by virtnbdbackup-1.9-1.noarch
nothing provides python3-lz4 needed by virtnbdbackup-1.9-1.noarch (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) [root@123 dist]#

i am not able to fix ist. System is RHEL 9

abbbi commented 1 year ago

hi,

the provided rpm package is built on alma 8, so it is only compatible to rhel 8. You have to pick one of the other installation methods such as virtualenv or consider using docker.

cosecit commented 1 year ago

is there an RHEL 9 rpm in planing?

abbbi commented 1 year ago

The provided python version in rhel 9 comes with the dataclasses library already included:

python3 -c 'from dataclasses import dataclass'

so this dependency of the rpm package can be dropped on rhel9, but i dont find any packages for the required lz4 modules neither in the standard or epel repositories.. So i currently see no sane way to provide an rpm package that would fulfill all requirements.

abbbi commented 1 year ago

After some quick investigation it seems to me as if the python lz4 bindings did not make it into the fedora release rhel 9 is based on due to some issues, iam however not sure which issue it was, likely one of those: https://bugzilla.redhat.com/buglist.cgi?component=python-lz4&product=Fedora

In any way, .. use one of the other installation methods for now, until another rhel 9 release may re-introduce the required packages.

Using this should pull all required dependencies on RHEL9, only lz4 module has to be installed via pip

 yum install epel-release
 yum makecache
 yum install python3-pip python3-libnbd python3-paramiko python3-tqdm python3-libvirt  python3-lxml
 pip install lz4

cosecit commented 1 year ago

thanks for the support. same on RHEL 9 [root@vdr dist]# yum install python3-pip python3-libnbd python3-paramiko python3-tqdm python3-libvirt python3-lxml Updating Subscription Management repositories. Last metadata expiration check: 0:08:47 ago on Wed 09 Nov 2022 05:27:54 PM CET. Package python3-pip-21.2.3-6.el9.noarch is already installed. Package python3-libnbd-1.10.5-1.el9.x86_64 is already installed. Package python3-paramiko-2.11.0-1.el9.noarch is already installed. Package python3-tqdm-4.64.0-1.el9.noarch is already installed. Package python3-libvirt-8.0.0-1.el9.x86_64 is already installed. Package python3-lxml-4.6.5-2.el9.x86_64 is already installed. Dependencies resolved. Nothing to do. Complete! [root@vdr dist]# pip install lz4 Requirement already satisfied: lz4 in /usr/local/lib64/python3.9/site-packages (4.0.2) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [root@vdr dist]# dnf install virtnbdbackup-1.9-1.noarch.rpm Updating Subscription Management repositories. Last metadata expiration check: 0:09:22 ago on Wed 09 Nov 2022 05:27:54 PM CET. Error: Problem: conflicting requests

nothing provides python3-dataclasses needed by virtnbdbackup-1.9-1.noarch
nothing provides python3-lz4 needed by virtnbdbackup-1.9-1.noarch (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) [root@vdr dist]#

on docker i get en error Domain is missing. but the domain i right [root@vdr 2022]# docker run --rm -v /var/run:/run -v /var/lock:/run/lock -v /var/tmp:/var/tmp -v /daten_sich/vm/2022:/mnt/backups virtnbdbackup-docker virtnbdbackup -d win11-p -l full -o /mnt/backups/ Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. [2022-11-09 16:41:29] INFO common - printVersion [MainThread]: Version: 0.55 Arguments: /usr/local/bin/virtnbdbackup -d win11-p -l full -o /mnt/backups/ [2022-11-09 16:41:29] INFO virtnbdbackup - main [MainThread]: Backup level: [full] [2022-11-09 16:41:29] ERROR virtnbdbackup - main [MainThread]: Domain is missing required incremental-backup capability. [root@vdr 2022]#

[root@vdr 2022]# virsh list |grep win11-p 6 win11-p running

abbbi commented 1 year ago

error is:

ERROR virtnbdbackup - main [MainThread]: **Domain is missing required incremental-backup capability.**

Please read the documentation:

https://github.com/abbbi/virtnbdbackup#libvirt-versions--760-debian-bullseye-ubuntu-20x

cosecit commented 1 year ago

Sorry abbbi, I read it again and found no hint

abbbi commented 1 year ago

Sorry abbbi, I read it again and found no hint

The backup fails because the virtual machine has not enabled the backup features required. It may seem that on RHEL9 these features are not enabled by default.

You need to edit the virtual machine configuration via "virsh" and enable the incremental backup feature to be able to backup the virtutal machine via NBD as described here:

https://github.com/abbbi/virtnbdbackup#libvirt-versions--760-debian-bullseye-ubuntu-20x

To enable the incremental backup feature on these libvirt versions change the virtual machine config using virsh edit like so: (the first line must be changed, too!):
 <domain type='kvm' id='1' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
 [..]
 <qemu:capabilities>
   <qemu:add capability='incremental-backup'/>
 </qemu:capabilities>
 [..]
 </domain>
Note:

You must restart the virtual machine after enabling the feature!

cosecit commented 1 year ago

OK, thanks. i am using libvirt 8.0.0. i checked it and thought it was for versions <= 7.6.0.0 if i edit (virsh edit win11-p) and changed the first line. I can save it but when I check it, it was changed back

if i add the qemu lines (see the example) i geht an error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng

abbbi commented 1 year ago

i dont know. I just tried on RHEL9 and everything works like expected.

0) power off virtual machine 1) virsh edit VM 2) delete first line 3) copy paste first line from the manual to the editor 4) copy paste capabilites statement to the editor 5) save vm config 6) start vm 7) backup works like expected:

qemu-img create /tmp/test.qcow -f qcow2 10G
Formatting '/tmp/test.qcow', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=10737418240 lazy_refcounts=off refcount_bits=16
virt-install --memory 2000 --name testvm --disk path=/tmp/test.qcow --print-xml -v > testvm.xml
[root@rhel9-cli vagrant]# virsh define testvm.xml 
Domain 'testvm' defined from testvm.xml
[root@rhel9-cli vagrant]# virsh edit testvm
  copy paste as outlined in doucmentation
[root@rhel9-cli vagrant]# virsh dumpxml testvm | head -1
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
[root@rhel9-cli vagrant]# virsh dumpxml testvm | grep incremental
    <qemu:add capability='incremental-backup'/>
[root@rhel9-cli vagrant]# virsh start testvm
Domain 'testvm' started
[root@rhel9-cli virtnbdbackup]# ./virtnbdbackup  -d testvm -l full -o $(mktemp -d) 2>&1 | tail -n 1
[2022-11-10 19:48:54] INFO virtnbdbackup - main [MainThread]: Finished successfully

the only way i could see this happening is that your virtual machine is not type KVM but type QEMU. (domain type='qemu' in first line of the vm config). Then changing the vm config wont work. Your Host must support KVM backend and virtual machine must be using KVM too.

Anyways, these are all issues in your environment, so i think it would be better to continue anything related to THIS particular issue as a discussion, instead of diverging from the original issue topic.

cosecit commented 1 year ago

thanks abbbi, great job. It works well. The issue was on me.

I have made few tests. Only input, no big issues

in the docker image the auto option is not availible invalid choice: 'auto' (choose from 'copy', 'full', 'inc', 'diff')

in the docker commands docker-virtnbdbackup (docker.io/library/docker-virtnbdbackup:latest) not found it is virtnbdbackup-docker

in any backup jobs (like Windows vm´s) i get a warning WARNING libvirthelper - fsFreeze [MainThread]: argument unsupported: QEMU guest agent is not configured

only the first disk will be backuped. the second one is skipped. This is not so nice. Maybe it is possible that you can update the docker image

today i tested a litte bit. Windows 11 backup & restore -> fine

Rhel9 backup -> fine

freebsd (OPNsense) backup -> fine

Host system is RHEL 9 & podman (system is up2date)

best regards Lenny

abbbi commented 1 year ago

hi,

as for your input

1) the docker image which is available on docker.io was not created by myself, it does not include the latest virtnbdbackup version, thats the reason for the missing auto backup function. You may want to consider building your own container image. Unfortunately as i dont have access to that docker account i cant update the image.

2) the warning has its cause, see:

https://github.com/abbbi/virtnbdbackup#application-consistent-backups

3) if the disk is excluded form bacup automatically, it has a reasoning which is to be found in the backup log, most probably the disk that is excluded is a raw disk volume, see:

https://github.com/abbbi/virtnbdbackup#supported-disk-formats--raw-disks

raw disks or directly attached disks via passthrough cannot be saved via the dirty bitmap feature backup, so you have to check the virtual machines configuration and backup logs. Closing this "issue", now -> feel free to start a new discussion for other topics.

JSkolnik commented 1 year ago

I will use this topic, already closed. Is there any way to install RHEL9? There is a conflict when installing the package:

Error: 
 Problem: conflicting requests
  - nothing provides python3-dataclasses needed by virtnbdbackup-1.9.37-1.noarch

abbbi commented 1 year ago

the packages provided are built for rhel 8 compatibility. Ive not yet have time to prepare builds for rhel 9. you have to rebuild your own packages from source by the steps outlined in the vagrant file that builds these packages for rhel 8:

https://github.com/abbbi/virtnbdbackup/blob/master/vagrant/rhel/Vagrantfile#L17

or use other installation methods, see also:

https://github.com/abbbi/virtnbdbackup/tree/master#rpm-package

(python versions used on rhel 9 might differ)

You also have to activate the epel repositories to have all required package dependencies.

Also, i dont know wether if the required missing lz4 packages have yet made it into an rhel release .. see older comments

JSkolnik commented 1 year ago

I just built the RPM on RHEL9.2 where the Python version is 3.9. Build succeeded. But installing the built RPM package itself does not. The error is a missing python3-dataclasses dependency, because there is no longer such a package in RHEL9. Because dataclasses is already part of Python since version 3.7. Is it possible to remove the dependency? I can try it. Then rebuild the RPM package. The problem with LZ4 didn't show it to me.

abbbi commented 1 year ago

yes, you can try to remove the dependency from the requires in:

https://github.com/abbbi/virtnbdbackup/blob/master/setup.cfg#L9

and rebuild the packages. I need to check out how to make the requirements conditional. Will probably release an rpm package with the next version.

JSkolnik commented 1 year ago

Yes, I found the dependencies and removed python3-dataclasses. Compiled the package and installed it. I am attaching the necessary dependencies for installation on RHEL9. I'm going to test the backup now. Do you want to send a log? screenshot 2023-08-21 13-18-22

JSkolnik commented 1 year ago

I've done a test backup, an incremental backup, including a restore test and everything looks OK.

[2023-08-21 13:47:46] INFO lib common - printVersion [MainThread]: Version: 1.9.37 Arguments: /usr/bin/virtnbdbackup -d rocky8 -l auto -o /mnt/ssd_sas/backup_repo/test
[2023-08-21 13:47:46] INFO root virtnbdbackup - main [MainThread]: Backup level: [auto]
[2023-08-21 13:47:46] INFO root virtnbdbackup - main [MainThread]: Backup mode auto, target folder is empty: executing full backup.
[2023-08-21 13:47:46] INFO root virtnbdbackup - main [MainThread]: Libvirt library version: [9000000]
[2023-08-21 13:47:46] INFO root disktype - Optical [MainThread]: Skipping attached [cdrom] device: [sda].
[2023-08-21 13:47:46] INFO root virtnbdbackup - main [MainThread]: Backup will save [1] attached disks.
[2023-08-21 13:47:46] INFO root virtnbdbackup - main [MainThread]: Concurrent backup processes: [1]
[2023-08-21 13:47:46] INFO root checkpoint - redefine [MainThread]: Loading checkpoint list from: [/mnt/ssd_sas/backup_repo/test/checkpoints]
[2023-08-21 13:47:46] INFO root checkpoint - create [MainThread]: Checkpoint handling.
[2023-08-21 13:47:46] INFO root checkpoint - create [MainThread]: Using checkpoint name: [virtnbdbackup.0].
[2023-08-21 13:47:46] INFO root virtnbdbackup - main [MainThread]: Local NBD Endpoint socket: [/var/tmp/virtnbdbackup.38126]
[2023-08-21 13:47:46] INFO root virtnbdbackup - main [MainThread]: Temporary scratch file target directory: [/var/tmp]
[2023-08-21 13:47:46] INFO root virtnbdbackup - startBackupJob [MainThread]: Starting backup job.
[2023-08-21 13:47:47] INFO fs fs - freeze [MainThread]: Freezed [2] filesystems.
[2023-08-21 13:47:47] INFO fs fs - thaw [MainThread]: Thawed [2] filesystems.
[2023-08-21 13:47:47] INFO root virtnbdbackup - main [MainThread]: Started backup job with checkpoint, saving information.
[2023-08-21 13:47:47] INFO root checkpoint - backup [MainThread]: Saving checkpoint config to: [/mnt/ssd_sas/backup_repo/test/checkpoints/virtnbdbackup.0.xml]
[2023-08-21 13:47:47] INFO nbd client - printVersion [vda]: libnbd version: 1.14.2
[2023-08-21 13:47:47] INFO nbd client - connect [vda]: Waiting until NBD server at [nbd+unix:///vda?socket=/var/tmp/virtnbdbackup.38126] is up.
[2023-08-21 13:47:48] INFO nbd client - _getBlockInfo [vda]: Using Maximum Block size supported by NBD server: [33554432]
[2023-08-21 13:47:48] INFO nbd client - connect [vda]: Connection to NBD backend succeeded.
[2023-08-21 13:47:48] INFO root virtnbdbackup - backupDisk [vda]: Got 41 extents to backup.
[2023-08-21 13:47:48] INFO root virtnbdbackup - backupDisk [vda]: 10737418240 bytes disk size
[2023-08-21 13:47:48] INFO root virtnbdbackup - backupDisk [vda]: 2501574656 bytes of data extents to backup
[2023-08-21 13:47:48] INFO root virtnbdbackup - getWriter [vda]: Write data to target file: [/mnt/ssd_sas/backup_repo/test/vda.full.data.partial].
[2023-08-21 13:47:48] INFO root virtnbdbackup - backupDisk [vda]: Creating thin provisioned stream backup image
[2023-08-21 13:47:53] INFO root virtnbdbackup - main [MainThread]: Backup jobs finished, stopping backup task.                                                                                                                                                                  
[2023-08-21 13:47:53] INFO root metadata - backupConfig [MainThread]: Saving VM config to: [/mnt/ssd_sas/backup_repo/test/vmconfig.virtnbdbackup.0.xml]
[2023-08-21 13:47:53] INFO root metadata - backupDiskInfo [MainThread]: Saved qcow image config to: [/mnt/ssd_sas/backup_repo/test/vda.virtnbdbackup.0.qcow.json]
[2023-08-21 13:47:53] INFO root virtnbdbackup - main [MainThread]: Finished successfully

[2023-08-21 13:49:25] INFO lib common - printVersion [MainThread]: Version: 1.9.37 Arguments: /usr/bin/virtnbdbackup -d rocky8 -l auto -o /mnt/ssd_sas/backup_repo/test
[2023-08-21 13:49:25] INFO root virtnbdbackup - main [MainThread]: Backup level: [auto]
[2023-08-21 13:49:25] INFO root virtnbdbackup - main [MainThread]: Backup mode auto: executing incremental backup.
[2023-08-21 13:49:25] INFO root virtnbdbackup - main [MainThread]: Libvirt library version: [9000000]
[2023-08-21 13:49:25] INFO root disktype - Optical [MainThread]: Skipping attached [cdrom] device: [sda].
[2023-08-21 13:49:25] INFO root virtnbdbackup - main [MainThread]: Backup will save [1] attached disks.
[2023-08-21 13:49:25] INFO root virtnbdbackup - main [MainThread]: Concurrent backup processes: [1]
[2023-08-21 13:49:25] INFO root checkpoint - redefine [MainThread]: Loading checkpoint list from: [/mnt/ssd_sas/backup_repo/test/checkpoints]
[2023-08-21 13:49:25] INFO root checkpoint - create [MainThread]: Checkpoint handling.
[2023-08-21 13:49:25] INFO root checkpoint - create [MainThread]: Next checkpoint id: [1].
[2023-08-21 13:49:25] INFO root checkpoint - create [MainThread]: Parent checkpoint name [virtnbdbackup.0].
[2023-08-21 13:49:25] INFO root checkpoint - create [MainThread]: Using checkpoint name: [virtnbdbackup.1].
[2023-08-21 13:49:25] INFO root virtnbdbackup - main [MainThread]: Local NBD Endpoint socket: [/var/tmp/virtnbdbackup.38164]
[2023-08-21 13:49:25] INFO root virtnbdbackup - main [MainThread]: Temporary scratch file target directory: [/var/tmp]
[2023-08-21 13:49:25] INFO root virtnbdbackup - startBackupJob [MainThread]: Starting backup job.
[2023-08-21 13:49:25] INFO fs fs - freeze [MainThread]: Freezed [2] filesystems.
[2023-08-21 13:49:25] INFO fs fs - thaw [MainThread]: Thawed [2] filesystems.
[2023-08-21 13:49:25] INFO root virtnbdbackup - main [MainThread]: Started backup job with checkpoint, saving information.
[2023-08-21 13:49:25] INFO root checkpoint - backup [MainThread]: Saving checkpoint config to: [/mnt/ssd_sas/backup_repo/test/checkpoints/virtnbdbackup.1.xml]
[2023-08-21 13:49:25] INFO root context - get [vda]: Using NBD meta context [qemu:dirty-bitmap:backup-vda]
[2023-08-21 13:49:25] INFO nbd client - printVersion [vda]: libnbd version: 1.14.2
[2023-08-21 13:49:25] INFO nbd client - connect [vda]: Waiting until NBD server at [nbd+unix:///vda?socket=/var/tmp/virtnbdbackup.38164] is up.
[2023-08-21 13:49:26] INFO nbd client - _getBlockInfo [vda]: Using Maximum Block size supported by NBD server: [33554432]
[2023-08-21 13:49:26] INFO nbd client - connect [vda]: Connection to NBD backend succeeded.
[2023-08-21 13:49:26] INFO root virtnbdbackup - backupDisk [vda]: Got 17 extents to backup.
[2023-08-21 13:49:26] INFO root virtnbdbackup - backupDisk [vda]: 10737418240 bytes disk size
[2023-08-21 13:49:26] INFO root virtnbdbackup - backupDisk [vda]: 524288 bytes of data extents to backup
[2023-08-21 13:49:26] INFO root virtnbdbackup - getWriter [vda]: Write data to target file: [/mnt/ssd_sas/backup_repo/test/vda.inc.virtnbdbackup.1.data.partial].
[2023-08-21 13:49:26] INFO root virtnbdbackup - backupDisk [vda]: Creating thin provisioned stream backup image
[2023-08-21 13:49:26] INFO root virtnbdbackup - main [MainThread]: Backup jobs finished, stopping backup task.                                                                                                                                                                  
[2023-08-21 13:49:26] INFO root metadata - backupConfig [MainThread]: Saving VM config to: [/mnt/ssd_sas/backup_repo/test/vmconfig.virtnbdbackup.1.xml]
[2023-08-21 13:49:27] INFO root metadata - backupDiskInfo [MainThread]: Saved qcow image config to: [/mnt/ssd_sas/backup_repo/test/vda.virtnbdbackup.1.qcow.json]
[2023-08-21 13:49:27] INFO root virtnbdbackup - main [MainThread]: Finished successfully

abbbi commented 1 year ago

thanks, ive opened issue for this topic and will include an rhel9 compatible package in the next release:

https://github.com/abbbi/virtnbdbackup/issues/130

abbbi commented 1 year ago

latest release now also includes packages built on alma 9, hopefully compatible to rhel9.

abbbi / virtnbdbackup

RHEL 9 compatibility #78

nothing provides python3-lz4 needed by virtnbdbackup-1.9-1.noarch (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) [root@vdr dist]#