to-bcache fails with "device-mapper: reload ioctl on synthetic-12b0a96e-74d5-11e3-b6c9-ac220b50c8fe failed: Invalid argument

[ianburrell]

This is on Fedora 20. Converting ext4 LV to bcache. The filesystem has already been shrunk with previous run.

# blocks to-bcache /dev/vg0/scratch
The filesystem (ext4) leaves enough room, no need to shrink it
device-mapper: reload ioctl on synthetic-12b0a96e-74d5-11e3-b6c9-ac220b50c8fe failed: Invalid argument
Command failed
Traceback (most recent call last):
  File "/root/.local/bin/blocks", line 9, in <module>
    load_entry_point('blocks==0.1.4', 'console_scripts', 'blocks')()
  File "/root/src/blocks/blocks/__main__.py", line 2002, in script_main
    sys.exit(main())
  File "/root/src/blocks/blocks/__main__.py", line 1668, in main
    return args.action(args)
  File "/root/src/blocks/blocks/__main__.py", line 1751, in cmd_to_bcache
    device=device, debug=args.debug, progress=progress, join=args.join)
  File "/root/src/blocks/blocks/__main__.py", line 1500, in lv_to_bcache
    synth_bdev = make_bcache_sb(pe_size, data_size, join)
  File "/root/src/blocks/blocks/__main__.py", line 1472, in make_bcache_sb
    with synth_device(bsb_size, data_size) as synth_bdev:
  File "/usr/lib64/python3.3/contextlib.py", line 48, in __enter__
    return next(self.gen)
  File "/root/src/blocks/blocks/__main__.py", line 1264, in synth_device
    exit_stack=st)
  File "/root/src/blocks/blocks/__main__.py", line 141, in mk_dm
    assert proc.returncode == 0
AssertionError

[ianburrell]

My suspicion is that it is the problem in #2 that /dev/mapper/rozeros-* is not being created by OS. That issue mentioned a commit that creates rozeros device automatically but I don't see the commit in master. That device doesn't exist on Fedora and I can't find any rules for udev to create it. Is this a Ubuntu specific device?

[g2p]

You're right, I hadn't integrated the commit mentioned at https://github.com/g2p/blocks/issues/2#issuecomment-22809049, not exactly sure why. Can you cherry-pick 4138c45471db45804e7c77d62237ea14d06bcebd and tell me if that works?

[ianburrell]

Git can't find that commit.

$ git cherry-pick 4138c45 fatal: bad revision '4138c45'

Was it pushed?

[g2p]

Sorry, this one at the top of the troubleshooting/dm-table branch: cdeb810c95a7478e9fb2f00ca915b8e44dcf809d

[ianburrell]

That worked. I rebased troubleshooting/dm-table branch against master to get that commit and the debugging output.

$ sudo blocks to-bcache /dev/vg0/scratch --join=670fd5d0-ff7d-4143-8193-f05407026746 The filesystem (ext4) leaves enough room, no need to shrink it TABLE '0 2147475456 error\n' TABLE '0 8192 linear /dev/loop0 0\n8192 2147475456 linear /dev/mapper/rozeros-bfa238f8-764d-11e3-97b6-ac220b50c8fe 0\n' Copying the bcache superblock... ok Loading LVM metadata... ok Rotating the last extent to be the first... ok

[g2p]

Good. It isn't perfect (there's a slight slowdown, a higher chance of udev triggering scans which hold the devices open, cause confusing error messages), but I've put it in master as the best alternative.

[qpfiffer]

@g2p @ianburrell Helped me with my problem. Thanks.

[g2p]

udev varies a bit across distros, what were you using @qpfiffer?

[qpfiffer]

Gentoo 3.12.2 with the CK patchset running the BFQ IO Scheduler. Systemd/udev 208.

I'm actually surprised it's working at all seeing how far away I am from my data. I've got an encrypted LVM on a RAID 5, and the SSDs are RAID 1. Somehow it all runs perfectly and I didn't lose anything. This was mostly for fun, but I'll swear by blocks now.

[g2p]

Sounds like the stack I use (lvm, bcache and encryption, just no raid). I've tweaked things a bit (pass --noudevsync first, then try both) because --verifyudev isn't available on the CI system. I'd appreciate a sanity check.

[wengole]

I am trying to use blocks to convert a btrfs partition to a bcache backing device

sudo blocks to-bcache /dev/sdb3 --join c2cf390f-3546-4c04-982b-5cd4ae138770
The filesystem (btrfs) leaves enough room, no need to shrink it
device-mapper: reload ioctl on synthetic-2c0aacb8-e594-11e3-acf9-6817295cac38 failed: Invalid argument
Command failed
device-mapper: reload ioctl on synthetic-2c0aacb8-e594-11e3-acf9-6817295cac38 failed: Invalid argument
Command failed
Traceback (most recent call last):
  File "/usr/bin/blocks", line 9, in <module>
    load_entry_point('blocks==0.1.4', 'console_scripts', 'blocks')()
  File "/home/manjaro/src/blocks/blocks/__main__.py", line 2012, in script_main
    sys.exit(main())
  File "/home/manjaro/src/blocks/blocks/__main__.py", line 1678, in main
    return args.action(args)
  File "/home/manjaro/src/blocks/blocks/__main__.py", line 1761, in cmd_to_bcache
    device=device, debug=args.debug, progress=progress, join=args.join)
  File "/home/manjaro/src/blocks/blocks/__main__.py", line 1580, in part_to_bcache
    synth_bdev = make_bcache_sb(bsb_size, data_size, join)
  File "/home/manjaro/src/blocks/blocks/__main__.py", line 1482, in make_bcache_sb
    with synth_device(bsb_size, data_size) as synth_bdev:
  File "/usr/lib/python3.4/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/home/manjaro/src/blocks/blocks/__main__.py", line 1274, in synth_device
    exit_stack=st)
  File "/home/manjaro/src/blocks/blocks/__main__.py", line 150, in mk_dm
    assert proc.returncode == 0, 'Report to https://github.com/g2p/blocks/issues/8 if you see this'
AssertionError: Report to https://github.com/g2p/blocks/issues/8 if you see this

[g2p]

@wengole, that's a manjaro install? No version numbers, just as current as Arch?

Maybe dmesg has some detail on how the ioctl fails?

[wengole]

@g2p This was run from a Manjaro LiveCD 0.8.9 64-bit which I believe is running 3.10 kernel. This was done so that I could convert in-place my root partition, but my actual install is running 3.14 (not sure if that actually makes a difference)

Now I've managed to workaround it. I added a pdb in the mk_dm function to try and debug the problem, and just stepping through, the error didn't occur. I assume therefore it's a timing issue in the calls to dmadm.

[tmolitor-stud-tu]

I have this problem, too: I'm at master commit a50575bfb9e837473c8e672b5d109eb2e69633da and using Debian testing (jessie).

blocks to-bcache /dev/system/vm_disks

The filesystem (btrfs) leaves enough room, no need to shrink it device-mapper: reload ioctl on synthetic-bdf1a314-ed57-11e3-905f-0013e826cd63 failed: Das Argument ist ungültig Command failed device-mapper: reload ioctl on synthetic-bdf1a314-ed57-11e3-905f-0013e826cd63 failed: Das Argument ist ungültig Command failed Traceback (most recent call last): File "/usr/local/bin/blocks", line 9, in load_entry_point('blocks==0.1.4', 'console_scripts', 'blocks')() File "/usr/src/blocks/src/blocks/blocks/main.py", line 2012, in script_main sys.exit(main()) File "/usr/src/blocks/src/blocks/blocks/main.py", line 1678, in main return args.action(args) File "/usr/src/blocks/src/blocks/blocks/main.py", line 1761, in cmd_to_bcache device=device, debug=args.debug, progress=progress, join=args.join) File "/usr/src/blocks/src/blocks/blocks/main.py", line 1510, in lv_to_bcache synth_bdev = make_bcache_sb(pe_size, data_size, join) File "/usr/src/blocks/src/blocks/blocks/main.py", line 1482, in make_bcache_sb with synth_device(bsb_size, data_size) as synth_bdev: File "/usr/lib/python3.3/contextlib.py", line 48, in enter return next(self.gen) File "/usr/src/blocks/src/blocks/blocks/main.py", line 1274, in synth_device exit_stack=st) File "/usr/src/blocks/src/blocks/blocks/main.py", line 150, in mk_dm assert proc.returncode == 0, 'Report to https://github.com/g2p/blocks/issues/8 if you see this' AssertionError: Report to https://github.com/g2p/blocks/issues/8 if you see this

[bjmgeek]

I have this issue on master as of 07137704a394ad539ca0ca116fd9f07f6f5e6ecc

# blocks to-lvm /dev/sde3
Will shrink the filesystem (ext4) by 7340032 bytes
Checking the filesystem before resizing it
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
scratch: 743174/30285824 files (0.2% non-contiguous), 101978975/121116416 blocks
Copying 4194304 bytes from pos 0 to pos 496085499904... ok
Preparing LVM metadata... device-mapper: reload ioctl on synthetic-764b1b8a-53d7-11e4-af58-aa0004000a04 failed: Invalid argument
Command failed
device-mapper: reload ioctl on synthetic-764b1b8a-53d7-11e4-af58-aa0004000a04 failed: Invalid argument
Command failed
Traceback (most recent call last):
  File "/usr/local/bin/blocks", line 9, in <module>
    load_entry_point('blocks==0.1.4', 'console_scripts', 'blocks')()
  File "/usr/local/lib/python3.4/dist-packages/blocks-0.1.4-py3.4.egg/blocks/__main__.py", line 2012, in script_main
  File "/usr/local/lib/python3.4/dist-packages/blocks-0.1.4-py3.4.egg/blocks/__main__.py", line 1678, in main
  File "/usr/local/lib/python3.4/dist-packages/blocks-0.1.4-py3.4.egg/blocks/__main__.py", line 1886, in cmd_to_lvm
  File "/usr/lib/python3.4/contextlib.py", line 283, in enter_context
    result = _cm_type.__enter__(cm)
  File "/usr/lib/python3.4/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.4/dist-packages/blocks-0.1.4-py3.4.egg/blocks/__main__.py", line 1274, in synth_device
  File "/usr/local/lib/python3.4/dist-packages/blocks-0.1.4-py3.4.egg/blocks/__main__.py", line 150, in mk_dm
AssertionError: Report to https://github.com/g2p/blocks/issues/8 if you see this
# blocks --debug to-lvm /dev/sde3
pe 4194304 pe_newpos 496085499904 devsize 496092839936
The filesystem (ext4) leaves enough room, no need to shrink it
Copying 4194304 bytes from pos 0 to pos 496085499904... ok
Preparing LVM metadata... device-mapper: reload ioctl on synthetic-53e1ce80-53d8-11e4-af58-aa0004000a04 failed: Invalid argument
Command failed
device-mapper: reload ioctl on synthetic-53e1ce80-53d8-11e4-af58-aa0004000a04 failed: Invalid argument
Command failed
Traceback (most recent call last):
  File "/usr/local/bin/blocks", line 9, in <module>
    load_entry_point('blocks==0.1.4', 'console_scripts', 'blocks')()
  File "/usr/local/lib/python3.4/dist-packages/blocks-0.1.4-py3.4.egg/blocks/__main__.py", line 2012, in script_main
  File "/usr/local/lib/python3.4/dist-packages/blocks-0.1.4-py3.4.egg/blocks/__main__.py", line 1678, in main
  File "/usr/local/lib/python3.4/dist-packages/blocks-0.1.4-py3.4.egg/blocks/__main__.py", line 1886, in cmd_to_lvm
  File "/usr/lib/python3.4/contextlib.py", line 283, in enter_context
    result = _cm_type.__enter__(cm)
  File "/usr/lib/python3.4/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.4/dist-packages/blocks-0.1.4-py3.4.egg/blocks/__main__.py", line 1274, in synth_device
  File "/usr/local/lib/python3.4/dist-packages/blocks-0.1.4-py3.4.egg/blocks/__main__.py", line 150, in mk_dm
AssertionError: Report to https://github.com/g2p/blocks/issues/8 if you see this

[wencheng]

Same in Debian Jassie. Cloned troubleshooting/dm-table and rebased to master with conflict removed. As you can tell, consus--nas-share is a LUKSed ext4 joining an SSD cache.

root@consus:/home/wencheng# /root/.local/bin/blocks --debug to-bcache /dev/mapper/consus--nas-share --join 9924ec3a-82b1-463c-8e53-02afbb7ff5df Enter passphrase for /dev/mapper/consus--nas-share: The filesystem (ext4) leaves enough room, no need to shrink it TABLE '0 7813513216 error\n' TABLE '0 8192 linear /dev/loop0 0\n8192 7813513216 linear /dev/mapper/rozeros-79c53938-5dc5-11e4-8cc3-001cc07f661c 0\n' device-mapper: reload ioctl on synthetic-79c547a2-5dc5-11e4-8cc3-001cc07f661c failed: Invalid argument Command failed device-mapper: reload ioctl on synthetic-79c547a2-5dc5-11e4-8cc3-001cc07f661c failed: Invalid argument Command failed Traceback (most recent call last): File "/root/.local/bin/blocks", line 9, in load_entry_point('blocks==0.1.4', 'console_scripts', 'blocks')() File "/home/wencheng/src/blocks/blocks/main.py", line 2013, in script_main sys.exit(main()) File "/home/wencheng/src/blocks/blocks/main.py", line 1679, in main return args.action(args) File "/home/wencheng/src/blocks/blocks/main.py", line 1762, in cmd_to_bcache device=device, debug=args.debug, progress=progress, join=args.join) File "/home/wencheng/src/blocks/blocks/main.py", line 1511, in lv_to_bcache synth_bdev = make_bcache_sb(pe_size, data_size, join) File "/home/wencheng/src/blocks/blocks/main.py", line 1483, in make_bcache_sb with synth_device(bsb_size, data_size) as synth_bdev: File "/usr/lib/python3.4/contextlib.py", line 59, in enter return next(self.gen) File "/home/wencheng/src/blocks/blocks/main.py", line 1275, in synth_device exit_stack=st) File "/home/wencheng/src/blocks/blocks/main.py", line 151, in mk_dm assert proc.returncode == 0, 'Report to https://github.com/g2p/blocks/issues/8 if you see this' AssertionError: Report to https://github.com/g2p/blocks/issues/8 if you see this

[bjmgeek]

I'm using Debian Sid, linux kernel 3.14-2 on x86_64.

On Mon, Oct 27, 2014 at 6:54 AM, wencheng notifications@github.com wrote:

Same in Debian Jassie. Cloned troubleshooting/dm-table and rebased to master with conflict removed. As you can tell, consus--nas-share is a LUKSed ext4 joining an SSD cache.

root@consus:/home/wencheng# /root/.local/bin/blocks --debug to-bcache /dev/mapper/consus--nas-share --join 9924ec3a-82b1-463c-8e53-02afbb7ff5df Enter passphrase for /dev/mapper/consus--nas-share: The filesystem (ext4) leaves enough room, no need to shrink it TABLE '0 7813513216 error\n' TABLE '0 8192 linear /dev/loop0 0\n8192 7813513216 linear /dev/mapper/rozeros-79c53938-5dc5-11e4-8cc3-001cc07f661c 0\n' device-mapper: reload ioctl on synthetic-79c547a2-5dc5-11e4-8cc3-001cc07f661c failed: Invalid argument Command failed device-mapper: reload ioctl on synthetic-79c547a2-5dc5-11e4-8cc3-001cc07f661c failed: Invalid argument Command failed Traceback (most recent call last): File "/root/.local/bin/blocks", line 9, in load_entry_point('blocks==0.1.4', 'console_scripts', 'blocks')() File "/home/wencheng/src/blocks/blocks/main.py", line 2013, in script_main sys.exit(main()) File "/home/wencheng/src/blocks/blocks/main.py", line 1679, in main return args.action(args) File "/home/wencheng/src/blocks/blocks/main.py", line 1762, in cmd_to_bcache device=device, debug=args.debug, progress=progress, join=args.join) File "/home/wencheng/src/blocks/blocks/main.py", line 1511, in lv_to_bcache synth_bdev = make_bcache_sb(pe_size, data_size, join) File "/home/wencheng/src/blocks/blocks/main.py", line 1483, in make_bcache_sb with synth_device(bsb_size, data_size) as synth_bdev: File "/usr/lib/python3.4/contextlib.py", line 59, in enter return next(self.gen) File "/home/wencheng/src/blocks/blocks/main.py", line 1275, in synth_device exit_stack=st) File "/home/wencheng/src/blocks/blocks/main.py", line 151, in mk_dm assert proc.returncode == 0, 'Report to #8 https://github.com/g2p/blocks/issues/8 if you see this' AssertionError: Report to #8 https://github.com/g2p/blocks/issues/8 if you see this

— Reply to this email directly or view it on GitHub https://github.com/g2p/blocks/issues/8#issuecomment-60575947.

[wencheng]

(kind of) solved. As everytime i run blocks it returns 3 types of error in random:

1. synth create failed (seems rozeros has been closed)
2. rozeros close failed (seems synth is using it)
3. synth close failed (seems synth is busy)

so I put sleep at the end of mk_dm(...)

 cmd[3:3] = ['--verifyudev']
 time.sleep(5)
 exit_stack.callback(lambda: quiet_call(cmd))

up and running now. cheers~

(let me know if you need more details of my env)

[mkhon]

Fails for me on Debian jessie.

root@lab-hn2:~/blocks# dpkg -l dmsetup Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=====================-===============-===============-================================================ ii dmsetup 2:1.02.90-2.2+d amd64 Linux Kernel Device Mapper userspace library root@lab-hn2:~/blocks# uname -a Linux lab-hn2 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3 (2016-07-02) x86_64 GNU/Linux root@lab-hn2:~/blocks#

In /var/log/syslog:

Aug 18 15:01:15 lab-hn2 kernel: [ 2166.249875] device-mapper: table: 253:1: linear: dm-linear: Device lookup failed Aug 18 15:01:15 lab-hn2 kernel: [ 2166.249947] device-mapper: ioctl: error adding target to table Aug 18 15:01:15 lab-hn2 kernel: [ 2166.255183] device-mapper: table: 253:1: linear: dm-linear: Device lookup failed Aug 18 15:01:15 lab-hn2 kernel: [ 2166.255250] device-mapper: ioctl: error adding target to table Aug 18 15:01:15 lab-hn2 kernel: [ 2166.266572] Buffer I/O error on device dm-0, logical block 244054768 Aug 18 15:01:15 lab-hn2 kernel: [ 2166.266625] Buffer I/O error on device dm-0, logical block 244054768 Aug 18 15:01:15 lab-hn2 kernel: [ 2166.268382] Buffer I/O error on device dm-0, logical block 244054768 Aug 18 15:01:15 lab-hn2 kernel: [ 2166.268439] Buffer I/O error on device dm-0, logical block 244054768 Aug 18 15:01:15 lab-hn2 kernel: [ 2166.270469] Buffer I/O error on device dm-0, logical block 1 Aug 18 15:01:15 lab-hn2 kernel: [ 2166.270513] Buffer I/O error on device dm-0, logical block 1

[mkhon]

The log is:

The filesystem (ext4) leaves enough room, no need to shrink it Copying 4194304 bytes from pos 0 to pos 999645249536... ok Preparing LVM metadata... 0 1952438272 error device-mapper: reload ioctl on synthetic-492c9a8a-6522-11e6-a249-90b11c14983e failed: Invalid argument Command failed device-mapper: reload ioctl on synthetic-492c9a8a-6522-11e6-a249-90b11c14983e failed: Invalid argument Command failed Traceback (most recent call last): File "/usr/local/bin/blocks", line 9, in load_entry_point('blocks==0.1.4', 'console_scripts', 'blocks')() File "/usr/local/lib/python3.4/dist-packages/blocks-0.1.4-py3.4.egg/blocks/main.py", line 2015, in script_main File "/usr/local/lib/python3.4/dist-packages/blocks-0.1.4-py3.4.egg/blocks/main.py", line 1681, in main File "/usr/local/lib/python3.4/dist-packages/blocks-0.1.4-py3.4.egg/blocks/main.py", line 1889, in cmd_to_lvm File "/usr/lib/python3.4/contextlib.py", line 283, in enter_context result = _cm_type.enter(cm) File "/usr/lib/python3.4/contextlib.py", line 59, in enter return next(self.gen) File "/usr/local/lib/python3.4/dist-packages/blocks-0.1.4-py3.4.egg/blocks/main.py", line 1277, in synth_device File "/usr/local/lib/python3.4/dist-packages/blocks-0.1.4-py3.4.egg/blocks/main.py", line 153, in mk_dm AssertionError: Report to https://github.com/g2p/blocks/issues/8 if you see this

[mkhon]

If I remove --noudevsync the command succeeds.

[etbuira]

I faced the same issue, adding time.sleep(5) before exit_stack.callback in md_dm resolved the issue.