Create sparse output files

[SteveAyre]

Would it be possible to create the output file as a sparse file with the disk size as its apparent size (perhaps as a command line option)?

I'm not familiar with Go so can't say how to create such a file, but with dd it would be as follows:

$ dd if=/dev/zero of=file.img bs=1 count=0 seek=512M 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00149225 s, 0.0 kB/s $ du -h file.img 0 file.img $ du -h --apparent-size file.img 512M file.img

This would not immediately allocate the space on disk so wouldn't affect the time to decrypt the drive, but would make it appear as if it is the correct size.

This would be useful to try mounting the drive read-only for inspection while the decryption is still running. Without being a sparse file you'll see an dmesg error like EXT4-fs (loop0): bad geometry: block count 244018688 exceeds size of device (30502080 blocks) but with the correct apparent size the drive can be mounted successfully with the correct options (for example mount -o loop,ro,noload,offset=$OFFSET,errors=continue $IMAGE $MOUNTPOINT). How much is recoverable at that stage would depend on the decryption progress, but potentially allows recovering particular files much quicker. It also allows you to verify the resulting file is a mountable image early on.

[andlabs]

This kind of pre-allocation would probably be an OS-specific thing, but that's fine. I can look into it.

(Note: while OS X supports sparse files, the HFS+ filesystem that it uses by default doesn't.)

[themaddoctor]

Steve, since you're on linux, you can mount the drive with cryptsetup without writing an image. The details depend on which encryption chip you have. If you know the key, I can explain how.

[SteveAyre]

@themaddoctor That would be useful as it's quite a large drive (1TB) but is probably only about 5% used. Taking quite a while to decrypt on its own. :)

[themaddoctor]

I just need to know if you have the key, and which encryption chip it has.

[SteveAyre]

Initio and I have the DEK

[themaddoctor]

The initio chip changes the order of bytes before and after decryption. This requires a (very small) kernel module. If you have Ubuntu 16.04.2, I already have the driver ready. If not, you need to build it. It's very easy, I promise.

[themaddoctor]

Here is the C code for it. To build, you need the build requirement for your kernel. On Ubuntu, you get them with this: sudo apt-get build-dep linux-image-$(uname -r) Create a makefile: echo “obj-m := permute.o” > Makefile Build: make -C /lib/modules/uname -r/build M=$PWD Insert: sudo insmod ./permute.ko

/*

• Cryptographic API.
• very simple cipher to permute each block of 16 bytes
• Thomas Kaeding thomas.a.kaeding@gmail.com, 20170313 */

include <linux/module.h>

include <linux/init.h>

include <linux/types.h>

include <linux/errno.h>

include <linux/crypto.h>

include <asm/byteorder.h>

static u8 encvec[16], decvec[16];

int permute_setkey(struct crypto_tfm tfm, const u8 in_key, unsigned int key_len) { int i; for (i=0;i<key_len;i++) { encvec[2i] = in_key[i] >> 4; encvec[2i+1] = in_key[i] & 0x0f; } for (i=0;i<16;i++) decvec[encvec[i]] = (u8)i; return 0; }

static void permute_encrypt(struct crypto_tfm tfm, u8 out, const u8 *in) { int i; u8 temp[16]; for (i=0;i<16;i++) temp[i] = in[i]; for (i=0;i<16;i++) out[i] = temp[encvec[i]]; return; }

static void permute_decrypt(struct crypto_tfm tfm, u8 out, const u8 *in) { int i; u8 temp[16]; for (i=0;i<16;i++) temp[i] = in[i]; for (i=0;i<16;i++) out[i] = temp[decvec[i]]; return; }

static struct crypto_alg permute_alg = { .cra_name = "permute", .cra_driver_name = "permute", .cra_priority = 100, .cra_flags = CRYPTO_ALG_TYPE_CIPHER, .cra_blocksize = 16, .cra_ctxsize = 0, .cra_alignmask = 3, .cra_module = THIS_MODULE, .cra_u = { .cipher = { .cia_min_keysize = 8, .cia_max_keysize = 8, .cia_setkey = permute_setkey, .cia_encrypt = permute_encrypt, .cia_decrypt = permute_decrypt } } };

static int __init permute_init(void) { return crypto_register_alg(&permute_alg); }

static void __exit permute_fini(void) { crypto_unregister_alg(&permute_alg); }

module_init(permute_init); module_exit(permute_fini);

MODULE_DESCRIPTION("permutes the bytes of each 16-byte block"); MODULE_LICENSE("GPL"); MODULE_ALIAS("permute");

[themaddoctor]

If you build it, would you please send me a copy (email in a comment at the top of the code) plus the name and version of the distribution you use. Thank you.

Setting up the decryption filter uses cryptsetup:

echo "32107654ba98fedc" | xxd -p -r | sudo cryptsetup \ -d - --hash=plain -c permute-ecb --key-size=64 \ create wd-layer1 /dev/sdc echo "XXXXXXXXXXXXXXXXXXXXXXXX" | xxd -p -r | sudo cryptsetup -d - --hash=plain \ --key-size=256 -c aes-ecb create wd-layer2 /dev/mapper/wd-layer1 echo "32107654ba98fedc" | xxd -p -r | sudo cryptsetup \ -d - --hash=plain -c permute-ecb --key-size=64 \ create wd /dev/mapper/wd-layer2

Replace sdc with the appropriate device file for your WD disk, and replace XXXXXXX with the hex digits of your key.

Check: sudo file -sL /dev/mapper/wd If you get "DOS/MBR boot sector" it worked. If not, contact me.

[SteveAyre]

Unfortunately that's proving tricky - it's a Netgear ReadyNAS 102 system (ARM armada 370 based) and Netgear haven't released a linux-headers package. They've released a zip of the linux source they used and what appears to be the config they used, but having tried building a few kernel modules it always seems to give module_layout symbol errors in dmesg. They haven't provided Module.symvers and the one from building the kernel doesn't seem to match theirs. If I get it working I'll let you know.

[themaddoctor]

Do you have a directory called /lib/modules/4.4...../build ? (where 4.4..... is replaced with your kernel version)

[SteveAyre]

Nope, no build directory, just GPL sources in a zip file that netgear cross compiled. They do give their .config at least (assuming it's the same one they built with) but so far haven't built a version where the module is loadable.

[SteveAyre]

Good news, I managed to crosscompile it on 32bit Ubuntu Trusty using their GPL sources. Seems either something about the marvell crosscompile toolchain or changing CONFIG_CROSS_COMPILE changes the module_layout/Module.symvers. So you have to cross compile, you can't build the kernel modules locally to the system. The version I built loads without an error so will let you know how I get on with it tonight.

[themaddoctor]

Will you send me a copy of the module? Thank you.

[SteveAyre]

It worked! I'll send you a copy and some system details shortly. :) Thank you!

[themaddoctor]

YW. Glad to help.