doesn't support lvmcache LVs

[aspiers]

I added lvmcache cachepools to the LVs for my / and /home filesystems, and now YaST's bootloader module fails with:

    ┌──────────────────────────────────────────────────────┐
    │ Detected LVM logical volumes of unsupported types:   │
    │                                                      │
    │ /dev/system/home                                     │
    │ /dev/system/root                                     │
    │                                                      │
    │ These logical volumes are ignored. Operations on the │
    │ correponding volume groups may fail.                 │
    │                                                      │
    │ Unexpected situation found in the system.            │
    │                                                      │
    │ Continue despite the error?                          │
    │                                                      │
    │                                                      │
    │                  [Continue] [Abort]                  │
    └──────────────────────────────────────────────────────┘

y2log reveals that this is coming from libstorage-ng:

2019-08-27 14:41:50 <1> ionian-wired(795) [Ruby] callbacks/libstorage_callback.rb:75 libstorage-ng reported an error, asking the user whether to continue
2019-08-27 14:41:50 <1> ionian-wired(795) [Ruby] callbacks/libstorage_callback.rb:76 Error details. Message: Detected LVM logical volumes of unsupported types:

/dev/system/home
/dev/system/root

These logical volumes are ignored. Operations on the correponding volume groups may fail.

and scrolling back further reveals the problem. libstorage-ng runs lvs as follows:

2019-08-27 14:41:50 <1> ionian-wired(795) [libstorage] SystemCmd.cc:188 SystemCmd Executing:"/sbin/lvs --reportformat json --units b --nosuffix --all --options lv_name,lv_uuid,vg_name,vg_uuid,lv_role,lv_attr,lv_size,stripes,stripe_size,chunk_size,pool_lv,pool_lv_uuid,data_lv,data_lv_uuid,metadata_lv,metadata_lv_uuid"

and the relevant lines of JSON returned are:

2019-08-27 14:41:50 <1> ionian-wired(795) [libstorage] SystemCmd.cc:663 Adding Line 11 "                  {"lv_name":"home", "lv_uuid":"lQsEa7-rkSY-L0AV-4hFV-YMEi-ZvCD-Raj2Nc", "vg_name":"system", "vg_uuid":"m4f2UJ-widt-pyk9-QkKY-KyJQ-pbwq-V8zzwL", "lv_role":"public", "lv_attr":"Cwi-aoC---", "lv_size":"1099511627776", "stripes":"1", "stripe_size":"0", "chunk_size":"327680", "pool_lv":"[cache-home]", "pool_lv_uuid":"Ik8Glo-8dEb-8mr8-KBwo-DG2X-00wC-4O7XJY", "data_lv":"", "data_lv_uuid":"", "metadata_lv":"", "metadata_lv_uuid":""},"

and

2019-08-27 14:41:50 <1> ionian-wired(795) [libstorage] SystemCmd.cc:663 Adding Line 16 "                  {"lv_name":"root", "lv_uuid":"MsUIK0-r51d-MJck-aQwk-ZiAS-A2BI-XdT2vl", "vg_name":"system", "vg_uuid":"m4f2UJ-widt-pyk9-QkKY-KyJQ-pbwq-V8zzwL", "lv_role":"public", "lv_attr":"Cwi-aoC---", "lv_size":"536870912000", "stripes":"1", "stripe_size":"0", "chunk_size":"131072", "pool_lv":"[cache-root]", "pool_lv_uuid":"sHAdfk-fqKi-iNTQ-e3OR-w1mz-WBqi-zm4Ure", "data_lv":"", "data_lv_uuid":"", "metadata_lv":"", "metadata_lv_uuid":""},"

I found this code which registers unknown types of LV:

https://github.com/openSUSE/libstorage-ng/blob/542877a5a6bb92c55b3a36159dee74b8aed34a52/storage/Devices/LvmLvImpl.cc#L261-L272

and also found that it will only be set to a recognised type if the first character of lv_attr is recognised:

https://github.com/openSUSE/libstorage-ng/blob/542877a5a6bb92c55b3a36159dee74b8aed34a52/storage/SystemInfo/CmdLvm.cc#L202-L208

but from above we can see that when an LV has cache enabled, the first character of lv_attr is actually C, which is not currently recognised.

So I am guessing that the only way to successfully run yast2 bootloader is to first split the cache off, e.g. in my case:

lvconvert --splitcache system/root
lvconvert --splitcache system/home

and then reattach it after yast2 has completed. Obviously it would be much better if libstorage-ng properly supported lvmcache.

[aspiers]

P.S. y2logs are saved and available on request.

[aschnell]

You should be able to continue after the popup. If not please make a bug report for YaST.

Otherwise LVM cache is indeed not supported.

[ancorgs]

@aspiers Does YaST Bootloader still work if you click on "continue"? If that's the case, I think this is a reasonable behavior. As long as libstorage-ng does not support cachepools (something that is planned in the mid-term, but not shortly), YaST alerts about the circumstance but allows to continue. That's pretty ok, I would say.

On the other hand, maybe we can make it less scary. As far as I know, YaST Bootloader only uses libstorage-ng for reading the system, not for modifying it. @jreidinger Is that correct?

For YaST Bootloader and other modules using yast-storage-ng in read-only mode, we could offer a slightly different behavior. For example, some messages about unsupported technologies could be presented in a less scary way or even skipped in some cases.

[aspiers]

@aschnell @ancorgs Thanks a lot for the quick replies. Unfortunately it does not work if I click Continue. Here are more details from my testing:

1. Start with lvmcache disabled (i.e. normal LVM / filesystem, backed by mdraid)

2. Observe yast2 bootloader works fine

3. Activate the cache LV which I previously setup:

   lvconvert --type cache --cachepool system/cache-root system/root

4. Run yast2 bootloader again

5. Observe that an Error dialog pops up, filled with a) lots of (apparently harmless) warnings about an fd leak, and b) an error right at the end (scroll to the far right) showing that it cannot find the disk by LVM UUID:

Execution of command "[["/usr/bin/grub2-editenv", "list"]]" failed. Exit code: 1 Error output: File descriptor 7 (pipe:[4428632]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 8 (pipe:[4428632]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 9 (pipe:[4428633]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 10 (pipe:[4428633]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 7 (pipe:[4428632]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 8 (pipe:[4428632]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 9 (pipe:[4428633]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 10 (pipe:[4428633]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 7 (pipe:[4428632]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 8 (pipe:[4428632]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 9 (pipe:[4428633]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 10 (pipe:[4428633]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 7 (pipe:[4428632]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 8 (pipe:[4428632]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 9 (pipe:[4428633]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 10 (pipe:[4428633]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 7 (pipe:[4428632]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 8 (pipe:[4428632]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 9 (pipe:[4428633]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 10 (pipe:[4428633]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 7 (pipe:[4428632]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 8 (pipe:[4428632]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 9 (pipe:[4428633]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 10 (pipe:[4428633]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 7 (pipe:[4428632]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 8 (pipe:[4428632]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 9 (pipe:[4428633]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 10 (pipe:[4428633]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 7 (pipe:[4428632]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 8 (pipe:[4428632]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 9 (pipe:[4428633]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv File descriptor 10 (pipe:[4428633]) leaked on vgs invocation. Parent PID 18319: /usr/bin/grub2-editenv /usr/bin/grub2-editenv: error: disk `lvmid/m4f2UJ-widt-pyk9-QkKY-KyJQ-pbwq-V8zzwL/MsUIK0-r51d-MJck-aQwk-ZiAS-A2BI-XdT2vl' not found.

6. Click OK (the only available button) on this dialog
7. Now I see the same dialog I originally reported at the top of this issue.
8. Click Continue
9. Observe another dialog:

Probing file system with UUID cad8b62c-88e5-4c19-ad24-06e75c0a8987 failed

Unexpected situation found in the system.

Click below to see more details (English only).

Continue despite the error?

           Continue   Abort  Details

10. Clicking Details shows:

    device not found, name:/dev/mapper/system-root

11. Click OK then Continue to continue despite the error

12. Observe another error dialog:

Internal error. Please report a bug report with logs.
Run save_y2logs to get complete logs.

Caller: /usr/share/YaST2/modules/BootStorage.rb:246:in `detect_disks'

Details: Missing '/' mount point

13. Click OK and yast exits.

Clearly there is some side effect of attaching the cache LV which somehow confuses the disk detection code. I see in the error above that it's looking for lvmid/m4f2UJ-widt-pyk9-QkKY-KyJQ-pbwq-V8zzwL/MsUIK0-r51d-MJck-aQwk-ZiAS-A2BI-XdT2vl, which seems to correspond OK to the uuids shown here:

# dmsetup info system-root
Name:              system-root
State:             ACTIVE
Read Ahead:        1024
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      254, 0
Number of targets: 1
UUID: LVM-m4f2UJwidtpyk9QkKYKyJQpbwqV8zzwLMsUIK0r51dMJckaQwkZiASA2BIXdT2vl

# lvdisplay system/root                             
  --- Logical volume ---
  LV Path                /dev/system/root
  LV Name                root
  VG Name                system
  LV UUID                MsUIK0-r51d-MJck-aQwk-ZiAS-A2BI-XdT2vl
  LV Write Access        read/write
  LV Creation host, time install, 2018-12-31 13:15:41 +0000
  LV Cache pool name     cache-root
  LV Cache origin name   root_corig
  LV Status              available
  # open                 1
  LV Size                500.00 GiB
  Cache used blocks      0.04%
  Cache metadata blocks  6.47%
  Cache dirty blocks     0.00%
  Cache read hits/misses 2 / 15
  Cache wrt hits/misses  653 / 572
  Cache demotions        0
  Cache promotions       336
  Current LE             128000
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     1024
  Block device           254:0

[aspiers]

As before, y2logs available on request - just let me know where I should upload them.

[aschnell]

The errors seem not related to libstorage-ng. I suggest to make a bug report at bugzilla.opensuse.org (including the YaST logs).

[jreidinger]

looks like grub does not support it. That is what grub2-editenv failure saying.

[aspiers]

@aschnell commented on August 29, 2019 1:30 PM:

The errors seem not related to libstorage-ng. I suggest to make a bug report at bugzilla.opensuse.org (including the YaST logs).

Thanks. Submitted as boo#1148809 (yast2 bootloader fails when lvmcache is in use).

[aspiers]

@jreidinger commented on August 29, 2019 1:47 PM:

looks like grub does not support it. That is what grub2-editenv failure saying.

grub2-editenv list works fine when I run it from the shell, but fails when invoked from yast2 bootloader. However this is probably not related to libstorage-ng so I've filed the details here:

https://bugzilla.opensuse.org/show_bug.cgi?id=1148809#c2

[aspiers]

@aschnell commented on August 29, 2019 1:30 PM:

The errors seem not related to libstorage-ng.

Which errors are you referring to? Please re-read my original analysis above in the description of the issue, which definitely shows that the problem is lack of support within libstorage-ng for LVs which have C as the first character of lv_attr.

[aschnell]

For example the one from grub2-editenv or the one in BootStorage.rb.

That libstorage-ng does not support LVM cache is not a bug but instead a missing feature (that might or might not be implemented some day).

[aspiers]

@aschnell commented on September 24, 2019 8:35 AM:

For example the one from grub2-editenv or the one in BootStorage.rb.

Sure but those errors are not the main point of this issue.

That libstorage-ng does not support LVM cache is not a bug but instead a missing feature (that might or might not be implemented some day).

Sure, I never said it was a bug ;-) Here is the last sentence of my original report:

Obviously it would be much better if libstorage-ng properly supported lvmcache.

If libstorage-ng had the proper support then it would successfully detect the root filesystem, and then most likely YaST's bootloader module would work happily (or at least be closer to working). I'm not sure it would even need to do anything clever to handle cached LVs different; I suspect it could be fixed easily with a tiny patch:

• add a new LvType::CACHE type
• tweak libstorage-ng/storage/SystemInfo/CmdLvm.cc so that it recognises lv_attr[0] == 'C' as the new LvType::CACHE type
• add LvType::CACHE to this case: https://github.com/openSUSE/libstorage-ng/blob/542877a5a6bb92c55b3a36159dee74b8aed34a52/storage/Devices/LvmLvImpl.cc#L245-L247

[aschnell]

Sorry, I do not understand your complain to start with. This issue is a feature requests and handled that way. Where is the problem you seem to have now?

[aspiers]

Firstly I'm not complaining about anything ;-) Just agreeing with you that it's a feature request not a bug.

The problem is the lack of support for cached LVs, as the issue title says. The side effect of this missing support is that yast2 bootloader breaks when cached LVs are in use.

In my last comment I suggested that maybe it could be fixed easily with those three small tweaks.

[aschnell]

There might be more things to do, e.g. about the space calculation of the volume group. The YaST team will look at it if the product owner schedules it.

[aspiers]

By the way this bug causes zypper updates to make systems unbootable when they're using cached LVs.

[ancorgs]

By the way this bug causes zypper updates to make systems unbootable when they're using cached LVs.

Can you please elaborate? How is the lack of support for cached LVM in libstorage-ng making zypper updates fail?

[aspiers]

Sorry, I meant just for when zypper updates the kernel. At that point it tries to update the bootloader as normal for the new kernel, but that fails so grub is left in an unusable state.

[ancorgs]

Ah, ok. So you are actually referring to https://bugzilla.opensuse.org/show_bug.cgi?id=1148809, which is about Grub2 not supporting lvmcache scenarios properly. Neither YaST or libstorage-ng are involved there.

I mean, the zypper update problem is not exactly a direct consequence of this Github issue (which is about libstorage-ng not supporting lvmcache).

Thanks for clarification.

PS.- For the libstorage-ng support for lvmcache we are targeting SLE-15-SP3. Let's see if other priorities allow that.

[aschnell]

With https://github.com/openSUSE/libstorage-ng/pull/721 libstorage-ng support probing of LVM cache.

[aspiers]

Thanks a lot @aschnell and @ancorgs !

[ancorgs]

I checked this does not happen anymore. I did setup a Tumbleweed system with LVM cache. YaST Bootloader do not show any pop-up related to problems reading the system (using libstorage-ng1-4.3.26 and yast2-storage-ng-4.3.8 and yast2-bootloader-4.3.5).

So I'm closing this. Please reopen if you can reproduce it with recent versions of libstorage-ng.

[aspiers]

Thanks, will do!