Cannot mount encrypted volume on thumb drive

[gratay]

Type	Version/Name
Distribution Name	Gentoo & Knoppix (Debian)
Distribution Version	N/A (Gentoo) & 8.1 (Knoppix)
Linux Kernel	4.14.15-aufs (Gentoo) & 4.12.7-64 (Knoppix)
Architecture	amd64
ZFS Version	git repository HEAD 2018/2/14
SPL Version	git repository HEAD 2018/2/14

Describe the problem you're observing

Using a thumb drive with zpool on it: one volume (video) created without encryption, one volume (data) created with encryption. Have been exporting and importing it between different machines with different kernel versions as noted above, but always same version of git repository on them (kept in sync). Now it has started failing to mount the encrypted volume (sometimes) and I can't explain why. Doesn't happen 100% of the time, but once it fails to mount it will continue failing to mount, no matter how many times kernel modules are unloaded and reloaded. Reboot doesn't always help either.

eg: root@atlantis:~ $ zpool import -l thumb Enter passphrase for 'thumb/data': 1 / 1 keys successfully loaded filesystem 'thumb/data' can not be mounted: Read-only file system cannot mount 'thumb/data': Invalid argument

If we look at the volumes where they should be mounted, the files on the unencrypted volume are fine, but nothing shows up for the encrypted volume (as expected from error above):

root@atlantis:~ $ ls -la /thumb/video total 3627904 drwxr-xr-x 2 gtaylor root 11 Feb 14 21:16 . drwxr-xr-x 4 root root 4 Dec 18 11:15 .. -rw-rw-r-- 1 gtaylor gtaylor 371 Feb 12 10:11 m4v.md5

root@atlantis:~ $ ls -la /thumb/data total 1 drwxr-xr-x 2 root root 2 Dec 18 11:15 . drwxr-xr-x 4 root root 4 Dec 18 11:15 ..

So I did some basic property checks to see if I was missing a feature somehow, and I can't see a problem here:

root@atlantis:~ $ zpool get all thumb NAME PROPERTY VALUE SOURCE thumb size 58.5G - thumb capacity 12% - thumb altroot - default thumb health ONLINE - thumb guid 18171553514569541670 - thumb version - default thumb bootfs - default thumb delegation on default thumb autoreplace off default thumb cachefile - default thumb failmode wait default thumb listsnapshots off default thumb autoexpand off default thumb dedupditto 0 default thumb dedupratio 1.00x - thumb free 50.9G - thumb allocated 7.60G - thumb readonly off - thumb ashift 0 default thumb comment - default thumb expandsize - - thumb freeing 0 - thumb fragmentation 0% - thumb leaked 0 - thumb multihost off default thumb feature@async_destroy enabled local thumb feature@empty_bpobj active local thumb feature@lz4_compress active local thumb feature@multi_vdev_crash_dump enabled local thumb feature@spacemap_histogram active local thumb feature@enabled_txg active local thumb feature@hole_birth active local thumb feature@extensible_dataset active local thumb feature@embedded_data active local thumb feature@bookmarks enabled local thumb feature@filesystem_limits enabled local thumb feature@large_blocks enabled local thumb feature@large_dnode enabled local thumb feature@sha512 enabled local thumb feature@skein enabled local thumb feature@edonr enabled local thumb feature@userobj_accounting active local thumb feature@encryption active local

root@atlantis:~ $ zfs get all thumb/data NAME PROPERTY VALUE SOURCE thumb/data type filesystem - thumb/data creation Mon Dec 18 11:15 2017 - thumb/data used 4.14G - thumb/data available 648G - thumb/data referenced 4.14G - thumb/data compressratio 1.04x - thumb/data mounted no - thumb/data quota none default thumb/data reservation none default thumb/data recordsize 128K default thumb/data mountpoint /thumb/data default thumb/data sharenfs off default thumb/data checksum on default thumb/data compression on local thumb/data atime on default thumb/data devices on default thumb/data exec on default thumb/data setuid on default thumb/data readonly off default thumb/data zoned off default thumb/data snapdir hidden default thumb/data aclinherit restricted default thumb/data createtxg 255 - thumb/data canmount on default thumb/data xattr on default thumb/data copies 1 default thumb/data version 5 - thumb/data utf8only off - thumb/data normalization none - thumb/data casesensitivity sensitive - thumb/data vscan off default thumb/data nbmand off default thumb/data sharesmb off default thumb/data refquota none default thumb/data refreservation none default thumb/data guid 8854241042863058303 - thumb/data primarycache all default thumb/data secondarycache all default thumb/data usedbysnapshots 0B - thumb/data usedbydataset 4.14G - thumb/data usedbychildren 0B - thumb/data usedbyrefreservation 0B - thumb/data logbias latency default thumb/data dedup on local thumb/data mlslabel none default thumb/data sync standard default thumb/data dnodesize legacy default thumb/data refcompressratio 1.04x - thumb/data written 4.14G - thumb/data logicalused 4.31G - thumb/data logicalreferenced 4.31G - thumb/data volmode default default thumb/data filesystem_limit none default thumb/data snapshot_limit none default thumb/data filesystem_count none default thumb/data snapshot_count none default thumb/data snapdev hidden default thumb/data acltype off default thumb/data context none default thumb/data fscontext none default thumb/data defcontext none default thumb/data rootcontext none default thumb/data relatime off default thumb/data redundant_metadata all default thumb/data overlay off default thumb/data encryption aes-256-ccm - thumb/data keylocation prompt local thumb/data keyformat passphrase - thumb/data pbkdf2iters 342K - thumb/data encryptionroot thumb/data - thumb/data keystatus available -

So I reloaded the module with some debug, and I've attached that here. I've looked at it myself, though, and I can't see any debug that looks helpful in working out why it is failing. The volume is definitely not read-only, just encrypted. If I take it to another machine after it fails this, it can sometimes mount there, despite the fact both are running same zfs version.

root@atlantis:~ $ insmod /lib/modules/4.14.5-aufs/extra/zfs/zfs.ko zfs_dbgmsg_enable=1 zfs_flags=545 zfs_dbgmsg_maxsize=67108864

(reran all the commands above)

root@atlantis:~ $ zpool export thumb

It doesn't make any difference if I try to manually load the key and mount the volume. That fails the same as the import -l.

proc_spl_kstat_zfs_dbgmsg.gz

[tcaputi]

@gratay We have fixed a number of bugs with encryption since you created the dataset, some of which technically required rebuilding the encrypted datasets due to on-disk format problems that could arise. Can you try doing that with the latest code from master? Would it be ok if we closed this ticket for now and you can reopen it and @ me if it happens again?

[gratay]

Ok, that sounds all right. I was using the key for a couple of weeks, so if there's still a problem it will probably take that long again for it to show. I'll nuke the key and restart.

[rageagainstthebugs]

@gratay can this be close?

[behlendorf]

Closing, this was very likely resolved by one of the follow up encryption fixed. Let's open a new issue if you run in to problems with the latest version from master.