search

johndoe31415 / luksipc

Official master repository of the LUKS in-place-conversion utility (luksipc)
GNU General Public License v3.0
91 stars 12 forks source link

Implausible values encountered in regards to disk sizes #15

Closed haumacher closed 5 years ago

haumacher commented 5 years ago

If I'm encountering the following, is the data gone? It seams that the header is much larger as documented (16384 kiB) is that the problem?

# ./luksipc -d /dev/sda1
WARNING! luksipc will perform the following actions:
   => Normal LUKSification of plain device /dev/sda1
   -> luksFormat will be performed on /dev/sda1

Please confirm you have completed the checklist:
    [1] You have resized the contained filesystem(s) appropriately
    [2] You have unmounted any contained filesystem(s)
    [3] You will ensure secure storage of the keyfile that will be generated at /root/initial_keyfile.bin
    [4] Power conditions are satisfied (i.e. your laptop is not running off battery)
    [5] You have a backup of all important data on /dev/sda1

    /dev/sda1: 1907728 MiB = 1863.0 GiB
    Chunk size: 10485760 bytes = 10.0 MiB
    Keyfile: /root/initial_keyfile.bin
    LUKS format parameters: None given

Are all these conditions satisfied, then answer uppercase yes: YES
[I]: Created raw device alias: /dev/sda1 -> /dev/mapper/alias_luksipc_raw_ead9863f
[I]: Size of reading device /dev/sda1 is 2000397795328 bytes (1907728 MiB + 0 bytes)
[I]: Backing up physical disk /dev/sda1 header to backup file header_backup.img
[I]: Performing luksFormat of /dev/sda1
[I]: Performing luksOpen of /dev/sda1 (opening as mapper name luksipc_95ed46b0)
[I]: Size of luksOpened writing device is 2000381018112 bytes (1907712 MiB + 0 bytes)
[I]: Write disk smaller than read disk by 16777216 bytes (16384 kiB + 0 bytes, occupied by LUKS header)
[W]: LUKS header larger than chunk copy size. LUKS format probably has overwritten data that cannot be recovered.
[E]: Implausible values encountered in regards to disk sizes (R = 3238002688l, W = 465l), aborting.
[E]: Exit with code 9 [EC_DEVICE_SIZES_IMPLAUSIBLE]: Device sizes are implausible
haumacher commented 5 years ago

Ok, after a reboot, the device is no longer bussy and can again be cleanly mounted...

haumacher commented 5 years ago

Btw. I used master version at a5580fca02ec19bcd4bf3e5568b54b7541c5c784 on a Raspberry PI 4. The maintainer release version 0.04 immediately crashed with a SIGSEGV.

johndoe31415 commented 5 years ago

Hi Bernhard,

this would be a HUGE problem, please be super careful with the next steps (even though it seems the volume is okay now) -- definitely keep the backup file around.

An important question is: How large is your backup file? Is it 128 MB? Because if so, then no data is lost. Which version of luksipc did you use?

But, yes, it is indeed a very big problem if LUKS suddenly creates 16 MB of header when previously it always were just 2 MB. However, we anticipate a maximum of 128 MB of header (hence the backup file).

What happens when this occurs is that luksips copies back the first original chunk and aborts, but this might not be enough. With a chunksize of 10 MB and a header size of 16 MB, I'd guess that everything between 10 MB and 16 MB is now garbage (overwritten by luksFormat) unless you recover that from your backup file. So I would be very careful with the volume as is.

My recommendation would be to cleanly unmount the volume. Then draw a backup of the disk header (first 128 MB) as they are now. Then restore the header from the backup version. Then proceed as normal. Since you've mounted the volume already, some things might have changed, but I would find it unlikely that something drastically has changed in the FS structure itself.

haumacher commented 5 years ago

Hi Johannes,

my backup file 128M:

-rw------- 1 root root 134217728 Oct 18 12:14 header_backup.img

as mentioned above, I built the master branch at a5580fca02ec19bcd4bf3e5568b54b7541c5c784 myself, since the maintainer version (0.04) directly crashed with SIGSEGV.

My volume I want to encrypt is a 2T Seagate external USB drive. I shrunk my file system by 10M as described in the docs.

I'm currently running an fschk on the volume - not yet finished but no errors yet.

haumacher commented 5 years ago

No error during check.

# e2fsck -f /dev/sda1
e2fsck 1.44.5 (15-Dec-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Backup: 3561685/122101760 files (0.5% non-contiguous), 59543172/488375868 blocks

Would it help to shrink my file system by another 10M and try again, or must something changed within luksipc?

johndoe31415 commented 5 years ago

Hi Bernhard, in order to make sure everything runs smoothly, I would want to change some things around in luksipc. Currently it seems it could also work if you shrunk the device by a lot more than 10 MB (I usually would choose 100-200 MB to be on the safe side), then specify a larger blocksize, e.g., 50331648 (48 MiB). However, I'll try to make those values the default for the next version which I plan to release tonight -- but right now I'm still at work and can't do it. I can promise you, however, to release v0.05 tonight with these tiny changes and then it should work seamlessly. I'm still puzzled as to why your luksFormat creates such a large header, I've converted volumes WAY larger than yours and the header always was 2 MiB. In any case I am quite thankful that your data is still there. Even though the docs say PLEASE make a backup, sometimes... well you know... sometimes you just trust everything will go according to plan :-)

johndoe31415 commented 5 years ago

One more thing, I'm assuming you're running Raspbian? Because then I can test with my RPi4 as well.

haumacher commented 5 years ago

Hi Johannes, yes, I'm running Raspbian 10:

# cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

The luksipc version of the distribution however does not work as said above:

ii  luksipc        0.04-3       armhf        LUKS in-place conversion tool

Therefore it would be a good idea to release an update. And no problem, I can wait until tomorrow before testing agin.

johndoe31415 commented 5 years ago

Hey Bernhard, thanks for the clarification. If you dare :-) to re-test this, I think 39454450 fixes your issue. If you'd confirm that it's fixed, I'd make a 0.05 release out of it. No idea why the version segfaults, but I also had no idea luksipc was part of standard Debian to be honest.

johndoe31415 commented 5 years ago

The 16 MiB header seems to be the default now. I can confirm the same when I create headers locally here. The good news is that I've anticipated this and the backup is 128 MiB. Therefore, no data should be lost, ever, unless the person doing the conversion falls into panic and starts doing odd things. Bad news is that this probably is not working anymore for anyone with a new version of dm-crypt.

haumacher commented 5 years ago

Thanks Johannes, your change fixes the problem! When releasing the new version, you should also update the dokumentation noting that the volume has to be shrunk by at least 16MB, otherwise people will run into trouble after 16h of waiting... :-)

My PI is working now - it will take the whole day:

# ./luksipc -d /dev/sda1
WARNING! luksipc will perform the following actions:
   => Normal LUKSification of plain device /dev/sda1
   -> luksFormat will be performed on /dev/sda1

Please confirm you have completed the checklist:
    [1] You have resized the contained filesystem(s) appropriately
    [2] You have unmounted any contained filesystem(s)
    [3] You will ensure secure storage of the keyfile that will be generated at /root/initial_keyfile.bin
    [4] Power conditions are satisfied (i.e. your laptop is not running off battery)
    [5] You have a backup of all important data on /dev/sda1

    /dev/sda1: 1907728 MiB = 1863.0 GiB
    Chunk size: 50331648 bytes = 48.0 MiB
    Keyfile: /root/initial_keyfile.bin
    LUKS format parameters: None given
    luksipc version: 0.04-31-g602c85abb4

Are all these conditions satisfied, then answer uppercase yes: YES
[I]: Created raw device alias: /dev/sda1 -> /dev/mapper/alias_luksipc_raw_bcba50d1
[I]: Size of reading device /dev/sda1 is 2000397795328 bytes (1907728 MiB + 0 bytes)
[I]: Backing up physical disk /dev/sda1 header to backup file header_backup.img
[I]: Performing luksFormat of /dev/sda1
[I]: Performing luksOpen of /dev/sda1 (opening as mapper name luksipc_f03bbc0e)
[I]: Size of luksOpened writing device is 2000381018112 bytes (1907712 MiB + 0 bytes)
[I]: Write disk smaller than read disk by 16777216 bytes (16384 kiB + 0 bytes, occupied by LUKS header)
[I]: Starting copying of data, read offset 50331648, write offset 0
[I]:  0:00:   0.0%       336 MiB / 1907712 MiB    57.0 MiB/s   Left: 1907376 MiB  9:17 h:m
[I]:  0:00:   0.0%       480 MiB / 1907712 MiB    43.7 MiB/s   Left: 1907232 MiB 12:06 h:m
[I]:  0:00:   0.0%       624 MiB / 1907712 MiB    38.5 MiB/s   Left: 1907088 MiB 13:46 h:m
[I]:  0:00:   0.0%       768 MiB / 1907712 MiB    36.0 MiB/s   Left: 1906944 MiB 14:42 h:m
[I]:  0:00:   0.1%       960 MiB / 1907712 MiB    35.2 MiB/s   Left: 1906752 MiB 15:03 h:m
[I]:  0:00:   0.1%      1152 MiB / 1907712 MiB    35.4 MiB/s   Left: 1906560 MiB 14:57 h:m
[I]:  0:00:   0.1%      1344 MiB / 1907712 MiB    34.4 MiB/s   Left: 1906368 MiB 15:23 h:m
[I]:  0:00:   0.1%      1536 MiB / 1907712 MiB    33.9 MiB/s   Left: 1906176 MiB 15:37 h:m
[I]:  0:00:   0.1%      1680 MiB / 1907712 MiB    33.2 MiB/s   Left: 1906032 MiB 15:56 h:m
[I]:  0:00:   0.1%      1872 MiB / 1907712 MiB    33.0 MiB/s   Left: 1905840 MiB 16:03 h:m
[I]:  0:01:   0.1%      2016 MiB / 1907712 MiB    32.6 MiB/s   Left: 1905696 MiB 16:13 h:m
[I]:  0:01:   0.1%      2208 MiB / 1907712 MiB    32.3 MiB/s   Left: 1905504 MiB 16:22 h:m
[I]:  0:01:   0.1%      2352 MiB / 1907712 MiB    31.9 MiB/s   Left: 1905360 MiB 16:34 h:m
[I]:  0:01:   0.1%      2496 MiB / 1907712 MiB    31.7 MiB/s   Left: 1905216 MiB 16:41 h:m
[I]:  0:01:   0.1%      2688 MiB / 1907712 MiB    31.5 MiB/s   Left: 1905024 MiB 16:47 h:m
[I]:  0:01:   0.1%      2832 MiB / 1907712 MiB    31.3 MiB/s   Left: 1904880 MiB 16:53 h:m
[I]:  0:01:   0.2%      2976 MiB / 1907712 MiB    31.2 MiB/s   Left: 1904736 MiB 16:57 h:m
johndoe31415 commented 5 years ago

Hi Bernhard, sounds good to me. I've released v0.05 just now which includes the fix for your issue. I've updated the docs as well but included a hint that cryptsetup-reencrypt might be the better alternative (it's maintained upstream which means that when they change their internal LUKS header format, they'll update all tooling as well -- you won't run into issues like this one with their official tools). Out of curiousity, why did you use luksipc instead of cryptsetup-reencrypt? Is there some feature that reencrypt does not have? Or was luksipc just the first thing you found?

haumacher commented 5 years ago

Thanks for your help! This was the first time I fiddled with hard disk encryption - and yes, luksipc was the first tool I stumbled across...