VDO - Import Volume

[penguinpages]

Is it possible and then what are commands to take a VDO single disk volume and move it to another server.

I would assume that the versions of VDO must match (I see other posting on that ) but assuming that is done.. Once the device is visible, how do you import it?

Ex: OS was damaged or upgraded and now that it is recovered, you want to import the VDO + XFS bricks that gluster would use :)

[rhawalsh]

We're actually working on putting together a 'vdo import' command [0] to make this easier, but in the meantime, we have a thread [1] in the vdo-devel mailing list archive that speaks to re-creating the VDO config file.

VDO versions should match to ensure that it works as expected.

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1737619 [1] https://www.redhat.com/archives/vdo-devel/2018-December/msg00004.html

[rhawalsh]

Hi @penguinpages,

Our latest update for the vdo project [1] contains the 'vdo import' command. Please take a look and let us know if you have any questions!

[1] https://github.com/dm-vdo/vdo/releases/tag/6.2.2.24

Thanks, Andy

[penguinpages]

thx. I may give that a try next week when back in town.

Thanks for all you do!

On Fri, Nov 1, 2019 at 12:45 PM Andy Walsh notifications@github.com wrote:

Hi @penguinpages https://github.com/penguinpages,

Our latest update for the vdo project [1] contains the 'vdo import' command. Please take a look and let us know if you have any questions!

[1] https://github.com/dm-vdo/vdo/releases/tag/6.2.2.24

Thanks, Andy

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dm-vdo/vdo/issues/24?email_source=notifications&email_token=AEC36VBPWPD6IGWF73A4JKLQRRMK5A5CNFSM4I5R5NVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC3PASQ#issuecomment-548859978, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEC36VECRKTT4IQY2ZIC5ALQRRMK5ANCNFSM4I5R5NVA .

-- jeremey.wise@gmail.com

[penguinpages]

did a fresh build of RHEV 4 Create VDO volume via UI tool Attempted to deploy the KVM managmet VM... which did not fail or complete.. just closed shell and no log I could find... so .. I reboot... and got a grub prompt.

Figured it would be good time to test reload import of VDO :)

yum install setroubleshoot-server wget gcc -y wget https://github.com/dm-vdo/vdo/archive/6.2.2.24.zip unzip 6.2.2.24.zip cd vdo-6.2.2.24/ make;make install ## -Wcast-qual -Wfloat-equal -Wformat=2 -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wswitch-default -pedantic -I../base -I../../uds -Wno-write-strings -DINTERNAL -DCURRENT_VERSION="\"6.2.2.24\"" "-DPLUGIN_NAME=\"libdevmapper-event-lvm2vdo.so\"" -c -MMD -MF .deps/vdodmeventd.d.new -MP -MT vdodmeventd.o -o vdodmeventd.o vdodmeventd.c vdodmeventd.c:32:26: fatal error: libdevmapper.h: No such file or directory #include ^ compilation terminated. make[3]: *** [vdodmeventd.o] Error 1 make[3]: Leaving directory `/root/vdo-6.2.2.24/utils/vdo/user' [vdo_compile_error.txt](https://github.com/dm-vdo/vdo/files/3824747/vdo_compile_error.txt)

[bgurney-rh]

The libdevmapper.h header file is available in the package "device-mapper-devel".

[rhawalsh]

Hi @penguinpages. I apologize for you having to deal with that error. @bgurney-rh is correct in his response. You will need to also install device-mapper-devel. I will make sure we update the requirements page to reflect the new package that we depend on.

[penguinpages]

Seems that package is not part of typical RHEV subscription service. ######## [root@eir vdo-6.2.2.24]# yum install setroubleshoot-server wget gcc device-mapper-devel -y Loaded plugins: imgbased-persist, product-id, search-disabled-repos, subscription-manager, vdsmupgrade, versionlock Excluding 4 updates due to versionlock (use "yum versionlock status" to show them) Package setroubleshoot-server-3.2.30-7.el7.x86_64 already installed and latest version Package wget-1.14-18.el7_6.1.x86_64 already installed and latest version Package gcc-4.8.5-39.el7.x86_64 already installed and latest version No package device-mapper-devel available. Nothing to do [root@eir vdo-6.2.2.24]#

So others can avoid my mistakes.... it is not in standard RHEV subscriptions: Googled around and found this: https://bugzilla.redhat.com/show_bug.cgi?id=1650686

RHEV does not have git... so that is another rathole to run down... so just downloaded and copied over zip

wget https://github.com/dm-vdo/kvdo/archive/6.2.0.239.zip unzip kvdo-6.2.0.239.zip cd /root/kvdo-6.2.0.239 make -C /usr/src/kernels/uname -r M=pwd

But I don't think that solved the device-mapper-devel missing issue..

Maybe I am also missing something but the above command compiles the code for the specified kernel tree, but does not install the modules. https://github.com/dm-vdo/kvdo/tree/6.2.0.239#building

Maybe I am missing something obivious here. 1) "device-mapper-devel" is not with RHEV's subscription library and I hesitate to use rpmfind RPM... when other more supported RedHat options can be followed. 2) with the new VDO module compiled for the current kernel. Does it not need to be installed into given binary / path locations?

[rhawalsh]

Seems that package is not part of typical RHEV subscription service. ######## [root@eir vdo-6.2.2.24]# yum install setroubleshoot-server wget gcc device-mapper-devel -y Loaded plugins: imgbased-persist, product-id, search-disabled-repos, subscription-manager, vdsmupgrade, versionlock Excluding 4 updates due to versionlock (use "yum versionlock status" to show them) Package setroubleshoot-server-3.2.30-7.el7.x86_64 already installed and latest version Package wget-1.14-18.el7_6.1.x86_64 already installed and latest version Package gcc-4.8.5-39.el7.x86_64 already installed and latest version No package device-mapper-devel available. Nothing to do [root@eir vdo-6.2.2.24]#

What version of RHEL is this running against? Looking at the gcc version you've got, it looks like somewhere around RHEL-7.7. The code for VDO-6.2.x.x is intended to be used with .EL8 systems, so you may actually run into further issues trying that. If you look on the README.md page for both the kvdo and vdo projects, you'll see the 'Releases' section. If you want to get it working on a different release, then that's fine; I just don't think it'll always be guaranteed to work "out of the box" that way.

That being said, the 'vdo import' operation wouldn't be available on the EL7 compatible versions of VDO (6.1.x.x).

So others can avoid my mistakes.... it is not in standard RHEV subscriptions: Googled around and found this: https://bugzilla.redhat.com/show_bug.cgi?id=1650686

Yeah, this was a different issue that was in relation to the dmeventd implementation.

RHEV does not have git... so that is another rathole to run down... so just downloaded and copied over zip

wget https://github.com/dm-vdo/kvdo/archive/6.2.0.239.zip unzip kvdo-6.2.0.239.zip cd /root/kvdo-6.2.0.239 make -C /usr/src/kernels/uname -r M=pwd

I noticed that you were referencing the VDO version 6.2.2.24 above, and now you're pointing at kvdo-6.2.0.239. You may notice some strange behaviors with that. I've not tested that configuration.

But I don't think that solved the device-mapper-devel missing issue..

If the build passed (i.e. make didn't fail), then you were able to compile it fine (just not install it... that's next)

Maybe I am also missing something but the above command compiles the code for the specified kernel tree, but does not install the modules. https://github.com/dm-vdo/kvdo/tree/6.2.0.239#building

Maybe I am missing something obivious here.

I think we may be unclear here, so I'll take that as an action item to fix. You will need to run make -C /usr/src/kernels/$(uname -r) M=$(pwd) modules_install in order to actually install the modules into the right kernel module directory. You'll then want to do a depmod -a to update the system so that modprobe kvdo will work.

1. "device-mapper-devel" is not with RHEV's subscription library and I hesitate to use rpmfind RPM... when other more supported RedHat options can be followed.

I'm not sure what the right thing to do is for that. Even before we were acquired by Red Hat, I always generally used a RHEL sub, not a RHEV sub. Sorry I can't help further with this one.

2. with the new VDO module compiled for the current kernel. Does it not need to be installed into given binary / path locations?

The kernel modules, once you run the modules_install step I mentioned above, should be available via modprobe. For the binaries and scripts, doing the make install step should put things in a place that is on the default path (typically /usr/bin, assuming you're not overriding things like bindir).

[penguinpages]

What version of RHEL is this running against? RHVH-4.3-20191023.0-RHVH-x86_64-dvd1.iso [root@eir ~]# uname -a Linux eir.penguinpages.local 3.10.0-1062.4.1.el7.x86_64 #1 SMP Wed Sep 25 09:42:57 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux That being said, the 'vdo import' operation wouldn't be available on the EL7 compatible versions of VDO (6.1.x.x).

--> Understood. I was using CentOS7 as it avoids bothering RH for NFR, but with these various support issues movinig builds to RHEV so no suport version escapes :)

If the build passed (i.e. make didn't fail), then you were able to compile it fine (just not install it... that's next)..... You will need to run make -C /usr/src/kernels/$(uname -r) M=$(pwd) modules_install... --> thx.. that was step missing I guess.. used to "make; make install" [root@eir kvdo-6.2.0.239]# make -C /usr/src/kernels/$(uname -r) M=$(pwd) modules_install make: Entering directory /usr/src/kernels/3.10.0-1062.4.1.el7.x86_64' INSTALL /root/kvdo-6.2.0.239/uds/uds.ko Can't read private key INSTALL /root/kvdo-6.2.0.239/vdo/kvdo.ko Can't read private key DEPMOD 3.10.0-1062.4.1.el7.x86_64 make: Leaving directory/usr/src/kernels/3.10.0-1062.4.1.el7.x86_64' [root@eir kvdo-6.2.0.239]# make install make: *** No rule to make target `install'. Stop. [root@eir kvdo-6.2.0.239]#

Again this is a lab... and I break things.... a lot.. in learning....

I think my take away is: 1) RHEV not tested and likely will never work and certainly is not supported for VDO import 2) RHEL may work and some future patch level of RHEL7 will incorporate this module / feature enhancement for VDO import

It would just be nice to know at what version of CentOS , RHEL , RHEV, that this kind of feature will be included as a part of distro standard stream.

Thanks for all you do. These kinds of technologies (Dedup and Compression) is 'table stakes' in the enterprise, but for us supporting education labs for our own carrier preservation, it saves us a LOT of $$ in these lab environments. Your work is appreciated.

[rhawalsh]

What version of RHEL is this running against? RHVH-4.3-20191023.0-RHVH-x86_64-dvd1.iso [root@eir ~]# uname -a Linux eir.penguinpages.local 3.10.0-1062.4.1.el7.x86_64 #1 SMP Wed Sep 25 09:42:57 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux That being said, the 'vdo import' operation wouldn't be available on the EL7 compatible versions of VDO (6.1.x.x).

--> Understood. I was using CentOS7 as it avoids bothering RH for NFR, but with these various support issues movinig builds to RHEV so no suport version escapes :)

Understandable.

If the build passed (i.e. make didn't fail), then you were able to compile it fine (just not install it... that's next)..... You will need to run make -C /usr/src/kernels/$(uname -r) M=$(pwd) modules_install... --> thx.. that was step missing I guess.. used to "make; make install" [root@eir kvdo-6.2.0.239]# make -C /usr/src/kernels/$(uname -r) M=$(pwd) modules_install make: Entering directory /usr/src/kernels/3.10.0-1062.4.1.el7.x86_64' INSTALL /root/kvdo-6.2.0.239/uds/uds.ko Can't read private key INSTALL /root/kvdo-6.2.0.239/vdo/kvdo.ko Can't read private key DEPMOD 3.10.0-1062.4.1.el7.x86_64 make: Leaving directory/usr/src/kernels/3.10.0-1062.4.1.el7.x86_64'

You should be able to modprobe kvdo after this point.

[root@eir kvdo-6.2.0.239]# make install make: *** No rule to make target `install'. Stop. [root@eir kvdo-6.2.0.239]#

Sorry, I should have been a bit more clear. The make install is for the VDO specific project, not the KVDO project. Going through with the modules_install and the depmod -a is sufficient for the KVDO project.

Again this is a lab... and I break things.... a lot.. in learning....

I think my take away is:

1. RHEV not tested and likely will never work and certainly is not supported for VDO import

I don't know that I would say "will never work". We, the VDO team, do not actively test against RHEV, so it's an issue that we're not aware of.

I also can't say that RHEV is not tested with VDO. I am not on the team who implements RHEV, and can only speak for the direct VDO team when discussing what is and is not a tested platform.

2. RHEL may work and some future patch level of RHEL7 will incorporate this module / feature enhancement for VDO import

We can consider backporting this to RHEL7. I don't know what that timeline would look like, however.

It would just be nice to know at what version of CentOS , RHEL , RHEV, that this kind of feature will be included as a part of distro standard stream.

Based on our earlier conversation related to 6.1.x.x and 6.2.x.x being intended for EL7 and EL8 respectively can help provide some insight as to where a feature will most likely land.

Longer term, and wider scope, I might point you at CentOS Stream to get a preview on what is coming in later releases (at least on EL8, since that's where Stream is).

Thanks for all you do. These kinds of technologies (Dedup and Compression) is 'table stakes' in the enterprise, but for us supporting education labs for our own carrier preservation, it saves us a LOT of $$ in these lab environments. Your work is appreciated.

We appreciate the feedback! I am certainly always happy to report back what I've heard from the community to the rest of the team, in case they haven't seen it.

[rhawalsh]

I am closing this issue as all questions have been answered. If you have further questions or issues related to this, you can either re-open this or open a new issue.

Thanks.