Performance impact on RPi4b?

[xanoni]

Thank you for this great guide. Do you have any info on how full disk encryption (adiantum) impacts read/write speeds with modern SSDs? Also, what would CPU utilization look like at maximum read/write speeds?

[keks24]

Hello,

I am glad to hear, that my guide is useful for you!

Do you have any info on how full disk encryption (adiantum) impacts read/write speeds with modern SSDs?

I guess, that modern SSDs means SATA SSDs.

Unfortunately, I do not have any SATA SSDs available, where I could run some tests. Also, depending on manufacturer, newer M.2 SSDs might have slower ICs from Samsung[1].

So, you could only test this on your own:

1. Configure the bootloader of your Raspberry Pi, so it boots from USB first
2. Set up Raspberry Pi OS on SSD
3. Connect the SSD to the Raspberry Pi and boot from it
4. Run some sophisticated benchmarks:

   $ for cipher in "xchacha12,aes-adiantum-plain64" "xchacha20,aes-adiantum-plain64" "aes-xts-plain64"; do cryptsetup benchmark --cipher="${cipher}"; done
   # Tests are approximate using memory only (no storage IO).
   #            Algorithm |       Key |      Encryption |      Decryption
   xchacha12,aes-adiantum        256b       106.8 MiB/s       112.3 MiB/s
   # Tests are approximate using memory only (no storage IO).
   #            Algorithm |       Key |      Encryption |      Decryption
   xchacha20,aes-adiantum        256b        98.6 MiB/s        99.0 MiB/s
   # Tests are approximate using memory only (no storage IO).
   # Algorithm |       Key |      Encryption |      Decryption
   aes-xts        512b        61.9 MiB/s        56.1 MiB/s

   $ sync && echo "3" > /proc/sys/vm/drop_caches
   $ dd if=/dev/zero of=deleteme.img bs=1M count=10240 conv=fdatasync status=progress
   $ echo "3" > /proc/sys/vm/drop_caches
   $ dd if=deleteme.img of=/dev/null bs=1M status=progress
   $ rm deleteme.img

   $ hdparm -Tt /dev/sdx

5. Take notes of read and write speeds
6. Set up encrypted Raspberry Pi OS on SSD
7. Connect the SSD to the Raspberry Pi and boot from it
8. Run tests as in step 4.
9. Take notes of read and write speeds
10. Compare the values

You can also test this in a somewhat real scenario:

1. Format and partition the SSD without any encryption
2. Connect and mount it via USB
3. Run the tests with dd and hdparm
4. Take notes of the values.
5. Encrypt the SSD via cryptsetup and partition it
6. Repeat steps 2 to 4.

Also, what would CPU utilization look like at maximum read/write speeds?

I tested this on a class 10 SD card, so the write speed is about 17 MB/s. The average CPU utilisation is about 23 %. But it will differ, if the data transfer is higher.

[1] I guess, this could also apply to SATA SSDs: https://www.youtube.com/watch?v=K07sEM6y4Uc

[xanoni]

I've run some tests on a ~900GB encrypted volume. It's indeed very resource intensive and slows things down quite a bit ;-) Anything that could be tweaked without surrendering a reasonable level of security?

Without encryption: ~200MB/s write and ~250MB/s read. With encryption: ~85MBs/s write and ~55MB/s read. (WRITE FASTER THAN READ)

I used this for the test: cryptsetup --cipher="xchacha20,aes-adiantum-plain64" --key-size="256" luksFormat "/dev/sdb3"

Benchmark script used: https://gist.github.com/xanoni/eabae6ef5c8cf4ffaccb9df9ddbd4929

R/W SPEEDS: Logs WITHOUT encryption: https://gist.github.com/xanoni/08a70bf939bf5227d352c3cbd3506804

R/W SPEEDS: Logs WITH encryption (xchacha20): https://gist.github.com/xanoni/f73b78a8d008e4223cb24bec53ad8bac

CPU LOAD (encrypted): Here's what happens during the dd benchmark:

CPU LOAD (encrypted): Screenshot also showing the kernel threads:

CPU LOAD (encrypted): Even during full idle the threads keep creeping around:

[xanoni]

By the way, similar story with xchacha12 (cryptsetup --cipher="xchacha12,aes-adiantum-plain64" --key-size=256 luksFormat "/dev/sdb3"). I don't see a big difference. Any other ciphers I should try?

Detailed benchmark: https://gist.github.com/xanoni/7e84bd60c0ff6c8a7a24604996d6eff3

[xanoni]

And now the surprise.... AES-XTS 512bit wins the dd benchmark (115 MB/sec write and 75 MB/sec read) but performs worse in the hdparm benchmark ... as suggested with key size 512 given it is cut in half ... same 400% utilization so I'm sparing you the screenshots.

cryptsetup --cipher="aes-xts-plain64" --key-size="512" luksFormat "/dev/sdb3"

Any idea why we see this?

Logs: https://gist.github.com/xanoni/0bc17e42350bb65b43f70e841ad535ec

[xanoni]

And the winner is 256bit AES-XTS (~128bit AES cbc essiv).... 125MB/sec write and ~90MB/sec read.

cryptsetup --cipher="aes-xts-plain64" --key-size="256" luksFormat "/dev/sdb3"

Logs: https://gist.github.com/xanoni/c37503630e9ce101a618f27ef6d443cf

Maybe the answer is XTS with 384bit? EDIT: Here is the benchmark (120MB/sec write + 80MB/sec read) -> https://gist.github.com/xanoni/69030c9d13954f3319fb35c056aa2703

[keks24]

Nice benchmarks!

Anything that could be tweaked without surrendering a reasonable level of security?

I would still stick to xchacha12 or xchacha20, since these encryption methods are fast and software-based. The ARM CPU does not support hardware-accelerated AES encryption:

$ lscpu | grep "Flags"
Flags:               half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32

You could also compare the load values of 5 and 15 minutes across all three ciphers. I would say, that aes-xts-plain64 would cause the highest load.

This article states, that there is a performance gain, if the encrypted partition has a sector size of 4096 bytes:

To be clear, Adiantum is only faster on sufficiently long messages; the 30% number is for 4096-byte messages. But that's fine for the intended use cases. To get good performance some dm-crypt users will need to use 4096-byte sectors rather than the default of 512, but they really ought to be doing so anyway.

I only tested this on hard drives with a physical sector size of 4096 bytes and I got the impression, that it does indeed increase write performance. If I remember correctly, mine was at 40%, but I also experimented with RAID 5 and RAID10 at the same time, where the latter was my final choice.

This would be worth a try, but it requires LUKS 2 to set the sector size:

$ parted /dev/sdx
(parted) mklabel
New disk label type? gpt
Warning: The existing disk label on /dev/sdx will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
Error: Partition(s) 1 on /dev/sdx have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  As a result, the old partition(s) will remain in use.  You should reboot now before
making further changes.
Ignore/Cancel? i
(parted) mkpart
Partition name?  []? deleteme
File system type?  [ext2]? ext4
Start? 2
End? -1
(parted) unit s
(parted) print
Model: Generic Flash Disk (scsi)
Disk /dev/sdx: 7864320s
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start  End       Size      File system  Name      Flags
 1      4096s  7862271s  7858176s  ext4         deleteme

(parted) align-check
alignment type(min/opt)  [optimal]/minimal?
Partition number? 1
1 aligned
(parted) quit
$ cryptsetup --cipher="xchacha20,aes-adiantum-plain64" --key-size="256" --sector-size="4096" luksFormat "/dev/sdx1"
WARNING: Device /dev/sdx1 already contains a 'crypto_LUKS' superblock signature.

WARNING!
========
This will overwrite data on /dev/sdx1 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for /dev/sdx1: 1234
Verify passphrase: 1234
$ cryptsetup open /dev/sdx1 cryptdeleteme
$ cryptsetup status cryptdeleteme | grep "sector size"
  sector size:  4096
$ parted /dev/mapper/cryptdeleteme "unit s print"
Model: Unknown (unknown)
Disk /dev/mapper/cryptdeleteme: 978176s
Sector size (logical/physical): 4096B/4096B
Partition Table: loop
Disk Flags:

Number  Start  End      Size     File system  Flags
 1      0s     978175s  978176s  ext4

This article and this article are very interesting to read:

Android dropping Speck [...] In any case, every attack so far appears to hit a wall at 8 rounds, with 12 rounds---the recommended eSTREAM round number for Salsa20---seeming to offer a reasonable security margin, still somewhat better than that of the AES.

Adiantum is a construction, not a primitive. Its security is reducible to that of XChaCha12 and AES-256, subject to a security bound; the proof is in Section 5 of our paper. Therefore, one need not "trust" Adiantum; they only need trust XChaCha12 and AES-256.

So, everything above chacha8 in combination with AES-256 can be considered secure.

By the way, similar story with xchacha12 (cryptsetup --cipher="xchacha12,aes-adiantum-plain64" --key-size=256 luksFormat "/dev/sdb3"). I don't see a big difference. Any other ciphers I should try?

I cannot think of any. Maybe there are some more software-based ciphers, but you would have to do your research. :)

And now the surprise.... AES-XTS 512bit wins the dd benchmark (115 MB/sec write and 75 MB/sec read) but performs worse in the hdparm benchmark ... as suggested with key size 512 given it is cut in half ... same 400% utilization so I'm sparing you the screenshots. [...] Any idea why we see this?

This is indeed weird. Have you cleared the PageCache, dentries and inodes after each test?:

$ sync && echo "3" > /proc/sys/vm/drop_caches

And the winner is 256bit AES-XTS (~128bit AES cbc essiv).... 125MB/sec write and ~90MB/sec read.

I am not so sure about aes-cbc-essiv, because of the Padding Oracle Attack. You would have to estimate this for yourself, if this is a security risk or not.

Maybe the answer is XTS with 384bit?

I cannot give you a proper answer to this. If the CPU does support hardware-accelerated AES, I would stick to aes-xts-plain64 and if not, I would stick to xchacha20,aes-adiantum-plain64.

About the benchmarks: You may also want to test with random values to cover a somewhat real life scenario, but using /dev/urandom for this is very inaccurate, since the Raspberry Pi has to generate the random values, which causes extra load.

Maybe you can test it like this. I am not sure, if this would be a correct test. Be aware, that this writes a file to RAM:

$ mount /dev/sdx1 /mnt
$ findmnt /dev/shm
TARGET   SOURCE FSTYPE OPTIONS
/dev/shm tmpfs  tmpfs  rw,nosuid,nodev
$ cd /dev/shm
$ dd if=/dev/urandom of=deleteme_urandom.img bs=1M count=5120 conv=fdatasync status=progress
$ rsync --info=progress2 deleteme_urandom.img /mnt

[xanoni]

This is indeed weird. Have you cleared the PageCache, dentries and inodes after each test?:

Here is my benchmark script, based on what you shared: https://gist.github.com/xanoni/eabae6ef5c8cf4ffaccb9df9ddbd4929

Between each of my posts (cipher changes) I unmounted the drive, did cryptsetup close, changed the cipher via cryptsetup luksFormat, and reformatted the drive with mkfs.ext4.

Missing anything?

I cannot give you a proper answer for this. If the CPU does support hardware-accelerated AES, I would stick to aes-xts-plain64 and if not, to xchacha20,aes-adiantum-plain64.

Okay, so I gotta figure out why AES-XTS gives me better performance than Adiantum.

By the way, here is the benchmark for AES-XTS-384bit (120MB/sec write + 80MB/sec read) -> https://gist.github.com/xanoni/69030c9d13954f3319fb35c056aa2703

[xanoni]

I am not so sure about aes-cbc-essiv, because of the Padding Oracle Attack. You would have to estimate this for yourself, if this is a security risk or not.

Just to clarify, I didn't use aes-cbc-essiv, I used aes-xts-plain64. I'm just saying that my 256bit key used should be equivalent to 128bit non-XTS AES. Do you have any insights on how secure aes-xts-plain64 with 256bit would be, and what the common attack vectors are?

[xanoni]

I went looking for some benchmarks ... seems the aes-xts-plain64 with 256bit is about to become insecure (in theory), while 384bit should be OK.

I'll see if I can get different results with increased block sizes and/or random data, but if not I'll choose aes-xts-plain64 with 384 bits.

128 bit (~256 bit XTS):

192 bit (~384 bit XTS):

256 bit (~512 bit XTS):

Source: https://www.keylength.com/en/compare/

[keks24]

Here is my benchmark script, based on what you shared: https://gist.github.com/xanoni/eabae6ef5c8cf4ffaccb9df9ddbd4929

Nice! I like these kind of first step of automation!

Missing anything?

Maybe you could do some additional tests with fio or bonnie++, which are a bit more complex to configure. Also, there is a --direct parameter for hdparm, which bypasses the kernel cache:

https://linuxreviews.org/HOWTO_Test_Disk_I/O_Performance

By the way, here is the benchmark for AES-XTS-384bit (120MB/sec write + 80MB/sec read) -> https://gist.github.com/xanoni/69030c9d13954f3319fb35c056aa2703

This is just a guess: Maybe your SSD has some caching mechanism for write operations or the flash controller does some vodoo, which increases the write performance. But this does not explain the slower read speed...

Just to clarify, I didn't use aes-cbc-essiv, I used aes-xts-plain64. I'm just saying that my 256bit key used should be equivalent to 128bit non-XTS AES.

Oh OK, I misunderstood, since you talked about it.

Do you have any insights on how secure aes-xts-plain64 with 256bit would be, and what the common attack vectors are? I went looking for some benchmarks ... seems the aes-xts-plain64 with 256bit is about to become insecure (in theory), while 384bit should be OK.

I actually never looked that up. I remember, that a network administrator of my ex-company told me to make sure, that I use a key size of 512, instead of 256 bit. He clearly demonstrated, that he is well-informed and since then I trust that statement.

This answer on StackExchange might be a good orientation to estimate the risks of a chosen cipher.

I'll see if I can get different results with increased block sizes and/or random data, but if not I'll choose aes-xts-plain64 with 384 bits.

Sounds like a good plan!

[xanoni]

--direct indeed makes a huge difference ... think I need to re-run everything

EDIT: seems it won't change the conclusion, though.

[xanoni]

The tests look more plausible now. Write speed is closer to read speed, and for smaller block sizes the read speed by far exceeds the write speed.

Conclusion didn't change, AES-XTS is 30-50% faster depending on block size. Besides, the load is lower ... with AES I stay below 3 whereas with Adiantum I almost hit 4 in the peak.

I was not able to test the 4096-bit alignment, though. Probably don't have the right LUKS: "Device size is not aligned to requested sector size.".

[keks24]

Conclusion didn't change, AES-XTS is 30-50% faster depending on block size. Besides, the load is lower ... with AES I stay below 3 whereas with Adiantum I almost hit 4 in the peak.

Interesting.

I would expect, that the former cipher would cause more load. It is still a mystery to me, why it is faster.

I was not able to test the 4096-bit alignment, though. Probably don't have the right LUKS: "Device size is not aligned to requested sector size.".

You need LUKS 2 for this and you need to align the partition with an offset of 2 MiB (4096 bytes) and not 1 MiB (2048 bytes) like I described above.

---

I also did some tests (dd) with a LUKS sector size of 4096 bytes on SD card and it only increased the read performance (zeroes) by estimated 17 % (~37 MB/s -> ~45.5 MB/s); write performance broke down by estimated 52 % (~17 MB/s -> 8 MB/s). So I will stick to the default 512 bytes in this case. But this is not comparable with your case.

[xanoni]

# fdisk /dev/sda

Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): p
Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: Portable SSD T5
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33553920 bytes
Disklabel type: gpt
Disk identifier: <SANITIZED>

Command (m for help): n
Partition number (1-128, default 1): 1
First sector (34-1953525134, default 65535): 4096
Last sector, +/-sectors or +/-size{K,M,G,T,P} (4096-1953525134, default 1953525134): +1953517567

Created a new partition 1 of type 'Linux filesystem' and of size 931.5 GiB.

Command (m for help): p
Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: Portable SSD T5
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33553920 bytes
Disklabel type: gpt
Disk identifier: <SANITIZED>

Device     Start        End    Sectors   Size Type
/dev/sda1   4096 1953521663 1953517568 931.5G Linux filesystem

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

# cryptsetup --cipher="xchacha20,aes-adiantum-plain64" --sector-size=4096 --key-size=256 luksFormat /dev/sda1

WARNING!
========
This will overwrite data on /dev/sda1 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase for /dev/sda1:
Verify passphrase:
Device size is not aligned to requested sector size.

[xanoni]

Second attempt:

# fdisk /dev/sda

Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): n
Partition number (1-128, default 1): 1
First sector (34-1953525134, default 65535): 4096
Last sector, +/-sectors or +/-size{K,M,G,T,P} (4096-1953525134, default 1953525134):

Created a new partition 1 of type 'Linux filesystem' and of size 931.5 GiB.

Command (m for help): p
Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: Portable SSD T5
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33553920 bytes
Disklabel type: gpt
Disk identifier: <SANITIZED>

Device     Start        End    Sectors   Size Type
/dev/sda1   4096 1953525134 1953521039 931.5G Linux filesystem

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Now it fails here:

# cryptsetup --cipher="xchacha20,aes-adiantum-plain64" --sector-size=4096 --key-size=256 luksFormat /dev/sda1

WARNING!
========
This will overwrite data on /dev/sda1 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase for /dev/sda1:
Verify passphrase:

# cryptsetup open /dev/sda1 cryptssd
Enter passphrase for /dev/sda1:
device-mapper: reload ioctl on   failed: Invalid argument

# dmesg | tail
[139970.549523]  sda: sda1
[140041.259761] device-mapper: table: 254:0: start=65535 not aligned to h/w logical block size 4096 of sda1
[140041.259773] device-mapper: core: Cannot calculate initial queue limits
[140041.259779] device-mapper: ioctl: unable to set up device queue for new table.

[keks24]

Reading this issue, one states, that end sector minus start sector must be divisible by 8. In your case:

1953525134 - 4096 = 1953521038
1953521038 / 8 = 244190129.75

The result must be an integer. So, if the end sector would be 1953525128 bytes, it should work:

1953525128 - 4096 = 1953521032
1953521032 / 8 = 244190129.00

The parameter --verbose of cryptsetup might also be helpful. :)

[xanoni]

Reading this issue, one states, that end sector minus start sector must be divisible by 8. In your case:

I had already tried this, which also didn't work :(

Device     Boot Start        End    Sectors   Size Id Type
/dev/sda1       65536 1953521664 1953456129 931.5G 83 Linux

Result:

[...]
Are you sure? (Type uppercase yes): YES
Enter passphrase for /dev/sda1:
Verify passphrase:
Device size is not aligned to requested sector size.

EDIT: I also tried decreasing the end by 1 sector to get a sector count that's divisible by 4096, but that also didn't work.

EDIT2: and -v only adds this output: Command failed with code -1 (wrong or missing parameters).

[keks24]

This is weird. What version have you installed? Mine is 2.3.4:

$ cryptsetup --version
cryptsetup 2.3.4
$ eix cryptsetup
[I] sys-fs/cryptsetup
     Available versions:  2.3.4-r1(0/12) ~2.3.6(0/12) {+argon2 gcrypt kernel nettle nls +openssl pwquality reencrypt static static-libs +udev urandom KERNEL="linux"}
     Installed versions:  2.3.4-r1(0/12)(13:26:06 01/05/21)(argon2 openssl udev -gcrypt -kernel -nettle -nls -pwquality -reencrypt -static -static-libs -urandom KERNEL="linux")
     Homepage:            https://gitlab.com/cryptsetup/cryptsetup/blob/master/README.md
     Description:         Tool to setup encrypted devices with dm-crypt

[xanoni]

The default that ships with Raspbian, and which is supposed to support LUKS2 ....

# cryptsetup --version
cryptsetup 2.1.0

[keks24]

Oh yeah, of course. I was wrong. I also have version 2.1.0 on Raspbian.

This is really weird. I cannot think of any solutions right now.

[xanoni]

Okay I need to do something at this point ... AES-XTS 384bit it is ... but this was a good learning exercise :) ... I'm sure I'll have more Qs as I proceed through the guide. THANK YOU!

[keks24]

Sure thing!

[xanoni]

For everyone's reference, here are the latest benchmarks with --direct enabled ... I also included the scripts that I used.

https://gist.github.com/xanoni/af4612e882c1d1bf22c7063258eb160a

I did not benchmark AES-XTS-PLAIN64 512bit this time, given it was not an option for me.

[xanoni]

Guess this can be closed now. I think performance will be sufficient, but will let you know if my Pi melts down, ha ha.

[keks24]

Sure! :)

[xanoni]

I figured out how to get the 4k sectors ... it's a bit weird

Let's say I want a 256 MiB LUKS device, then I have to increase the partition size by an extra 32 MiB (65536 sectors x 512 bytes) and then execute luksFormat with parameter --offset=65536 ... haven't tested speed diff yet

[xanoni]

Good news, @keks24

With 4k blocks, both AES and ChaCha have increased in speed. I'm now getting up to 185 MiB/s read speeds in some benchmarks, write speeds up to 125 MiB/s.

XChaCha speeds went up more than 120% in some categories ..

ChaCha write speeds are now ~10% higher than AES and read speed are 30-45% higher than AES.

There is no big difference between XChaCha12 and XChaCha20. (Haven't compared CPU utilization.)

Here's my partition table. I tested on the second partition (sdd2). I had to leave some ~32 MiB buffers to optimize the disk layout but still get 4k alignment.

(parted) p free
Model: Samsung Portable SSD T5 (scsi)
Disk /dev/sdd: 3907029168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start        End          Size         File system  Name  Flags
        34s          65534s       65501s       Free Space
 1      65535s       589822s      524288s
        589823s      655349s      65527s       Free Space
 2      655350s      84541429s    83886080s
        84541430s    84605684s    64255s       Free Space
 3      84605685s    3907027700s  3822422016s
        3907027701s  3907029134s  1434s        Free Space

Here is how I created the LUKS volumes:

#! /usr/bin/env -S bash -ex

DRIVE="/dev/sdd2"
LABEL="croot"
#OFFSET=65536 # for 1st partition
OFFSET=65528
ITER_TIME=4000
CIPHER="xchacha12,aes-adiantum-plain64"
#CIPHER="xchacha20,aes-adiantum-plain64"
KEY_SIZE=256
#CIPHER="aes-xts-plain64"
#KEY_SIZE=512

cryptsetup \
    --cipher="${CIPHER}" --label="${LABEL}" --key-size="${KEY_SIZE}" \
    --hash="sha512" --sector-size="4096" --offset="${OFFSET}" \
    --iter-time="${ITER_TIME}" --debug \
    luksFormat "${DRIVE}"

cryptsetup \
    --persistent --allow-discards --debug \
    luksOpen "${DRIVE}" "${LABEL}"

mkfs.ext4 -b 4096 "/dev/mapper/${LABEL}"

mkdir "/mnt/${LABEL}" || true

mount "/dev/mapper/${LABEL}" "/mnt/${LABEL}"

[keks24]

ChaCha write speeds are now ~10% higher than AES and read speed are 30-45% higher than AES.

Ah ha! See? :)

I am glad, that you could tweak the read and write speeds.

Here's my partition table. I tested on the second partition (sdd2). I had to leave some ~32 MiB buffers to optimize the disk layout but still get 4k alignment.

I still do not understand this.

It is actually sufficient to start the first sector of the first partition at 4096 bytes and every other partition should be aligned to 4k, as I have mentioned here.

Maybe try parted instead of fdisk for another test? :)

[xanoni]

@keks24 warning, dense info and conspiracies below.

I gained a better understanding yesterday but was too tired to update everything here before falling asleep. But I was just happy and relieved and wanted to share the success before fully understanding the "why" :)

It is actually sufficient to start the first sector of the first partition at 4096 bytes and every other partition should be aligned to 4k, as I have mentioned here.

I certainly tried that when we first met here in early July, but the problem is that the first sector that's available to me in parted is 65535. If I go lower, it throws an error. Also, if I create just a 256 MiB partition and then run luksFormat, only 256-32=224 MiB are made available to the mapped device. The likely reason is further down in this message, see "Why all this confusion and trouble?". TL;DR: it's probably an USB SCSI firmware bug.

I have not tried to lower --offset for partition 1 (given it will be unencrypted), but I decreased the partition back to 256 MiB and made sure it's optimally aligned (as per parted). I then used --offset $((1024*8)) for partitions 2 and 3. Not specifying an --offset or using --offset 0 would cause the errors that I reported in July.

Why? I think I know, because I now have more debug info. cryptsetup --debug talks a lot, whereas the -v that we tried earlier just doesn't do much. So what: LUKS wants to write its headers and keys into the offset space, thus if you keep it 0 that will case trouble.

If you choose a value that's smaller than what I used, but not too small, LUKS will warn you (in the --debug output) that there's not much space to store additional keys, but it will work despite the warnings. I used --offset $((1024*8)), which I think was twice (or more) of what it took to suppress the warning. But why not have a bit of extra space, it's virtually free.

FYI, there is no measurable performance difference between my initial ~32 MiB and the reduced offsets, but you probably didn't expect that.

Below my current partition table. parted classifies it as fully aligned ("optimal"). The 1434s gap at the end is needed or cryptsetup luksFormat will fail, as the size wouldn't be a multiple of 4096B.

Unit?  [compact]? s
(parted) p free
Model: Samsung Portable SSD T5 (scsi)
Disk /dev/sdd: 3907029168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start        End          Size         File system  Name  Flags
        34s          65534s       65501s       Free Space
 1      65535s       589822s      524288s      fat16
        589823s      655349s      65527s       Free Space
 2      655350s      84541429s    83886080s
        84541430s    84605684s    64255s       Free Space
 3      84605685s    3907027700s  3822422016s
        3907027701s  3907029134s  1434s        Free Space

Why all this confusion and trouble?

It seems that our confusion is caused by the fact that certain USB SCSI disks report erroneous/misleading sector information, especially if UAS mode is enabled (which my model requires to send the SCSI commands for fstrim / discard over USB). I hypothesize that the 65535 start sector that I'm seeing in fdisk -l /dev/sdd is actually just a display bug. There is no reason why the drive would need 32 MiB of free space at the beginning.

# cat /sys/block/sdd/queue/optimal_io_size
33553920

33553920B/512B=65535, and 65535+1 = 2^16, which is the error value -1 if we assume a 16-bit signed binary notation (11111111 11111111 in Two's Complement). Suspicious, right?

One problem that I haven't solved is that I'm seeing this in dmesg (first line for partition 2, second line for partition 3):

device-mapper: table: 254:1: adding target device (start sect 0 len 83877888) caused an alignment inconsistency
device-mapper: table: 254:2: adding target device (start sect 0 len 3822413824) caused an alignment inconsistency

You'll notice that 83877888 = 83886080s - $((1024*8)) and 3822413824 = 3822422016s - $((1024*8)) (the numbers are the partition sizes from parted above and my --offset value).

I'm not 100% sure, but what I found by searxing suggested that it's just "fake news", i.e., the Linux Kernel trusting the crappy sector information that the USB SCSI drive reports and thus complaining that the sector ratios are off. Let me know if you think it's more serious.

[xanoni]

Updated LUKS volume creation script:

#! /usr/bin/env -S bash -ex

#DRIVE="/dev/sdd2"
#LABEL="croot"

DRIVE="/dev/sdd3"
LABEL="cblocks"

OFFSET=$((1024*8))
BS=4096
#BS=512

CIPHER="xchacha12,aes-adiantum-plain64"
#CIPHER="xchacha20,aes-adiantum-plain64"
KEY_SIZE=256

#CIPHER="aes-xts-plain64"
#KEY_SIZE=512

HASH="sha512"
ITER_TIME=4000

umount "/mnt/${LABEL}" || true
cryptdisks_stop "${LABEL}"

cryptsetup \
    --cipher="${CIPHER}" --label="${LABEL}" --key-size="${KEY_SIZE}" \
    --hash="${HASH}" --sector-size="${BS}" --offset="${OFFSET}" \
    --iter-time="${ITER_TIME}" --debug \
    luksFormat "${DRIVE}"

cryptsetup \
    --persistent --allow-discards --debug \
    luksOpen "${DRIVE}" "${LABEL}"

#mkfs.ext4 "/dev/mapper/${LABEL}"
mkfs.ext4 -b "${BS}" "/dev/mapper/${LABEL}"

mkdir "/mnt/${LABEL}" || true

mount "/dev/mapper/${LABEL}" "/mnt/${LABEL}"

[keks24]

I totally forgot to answer this. My mind is currently omnipresent.

I just stumbled across this issue again, while I was thinking of the 4096 Bytes sector size offset, which I have adapted recently.

LUKS wants to write its headers and keys into the offset space, thus if you keep it 0 that will case trouble.

Interesting. This explains that small offset, which parted shows me:

$ parted --list
Model: Linux device-mapper (crypt) (dm)
[...]

Number  Start  End     Size    File system  Flags
 1      0.00B  15.6GB  15.6GB  ext4

Model: SD SC16G (sd/mmc)
[...]

Number  Start   End     Size    Type     File system  Flags
[...]
 2      273MB   15.9GB  15.7GB  primary

[...] I now have more debug info. cryptsetup --debug talks a lot

Good to know! I wish, that there would be some kind of convention, that every programm uses multiple -v parameters to increase verbosity. Oh well...

I now have set it as default parameter. :)

TL;DR: it's probably an USB SCSI firmware bug.

Yeah, I think so, too. This weird behaviour cries for a firmware bug.

What happens, if you partition an image file on your current filesystem and encrypt it? High verbosity for comparison:

$ cd "$(mktemp --directory)"
$ dd if="/dev/zero" of="partitionme.img" bs="1M" count="2048" conv="fdatasync" status="progress"

$ parted "partitionme.img"
GNU Parted 3.4
Using /tmp/tmp.orSuZwz0bA/partitionme.img
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit mib
(parted) print
Error: /tmp/tmp.orSuZwz0bA/partitionme.img: unrecognised disk label
Model:  (file)
Disk /tmp/tmp.orSuZwz0bA/partitionme.img: 2048MiB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:
(parted) mklabel gpt
(parted) mkpart
Partition name?  []? boot
File system type?  [ext2]? fat32
Start? 1
End? 1025
(parted) mkpart
Partition name?  []? rootfs
File system type?  [ext2]? ext4
Start? 1025
End? -1
(parted) print
Model:  (file)
Disk /tmp/tmp.orSuZwz0bA/partitionme.img: 2048MiB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start    End      Size     File system  Name    Flags
 1      1.00MiB  1025MiB  1024MiB  fat32        boot
 2      1025MiB  2047MiB  1022MiB  ext4         rootfs

(parted) quit

$ parted "partitionme.img" "unit s print"
Model:  (file)
Disk /tmp/tmp.orSuZwz0bA/partitionme.img: 4194304s
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start     End       Size      File system  Name    Flags
 1      2048s     2099199s  2097152s               boot    msftdata
 2      2099200s  4192255s  2093056s               rootfs
$ losetup
$ losetup --offset="$(( 512 * 2099200 ))" "/dev/loop1" "partitionme.img"
$ losetup
NAME       SIZELIMIT     OFFSET AUTOCLEAR RO BACK-FILE                           DIO LOG-SEC
/dev/loop1         0 1074790400         0  0 /tmp/tmp.orSuZwz0bA/partitionme.img   0     512

$ cryptsetup --debug --cipher="xchacha20,aes-adiantum-plain64" --key-size="256" --sector-size="4096" luksFormat "/dev/loop1"
# cryptsetup 2.3.6 processing "cryptsetup --debug --cipher=xchacha20,aes-adiantum-plain64 --key-size=256 --sect
or-size=4096 luksFormat /dev/loop1"
# Running command luksFormat.
# Locking memory.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device /dev/loop1.
# Trying to open and read device /dev/loop1 with direct-io.
# Initialising device-mapper backend library.

WARNING!
========
This will overwrite data on /dev/loop1 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
# Interactive passphrase entry requested.
Enter passphrase for /tmp/tmp.orSuZwz0bA/partitionme.img: 1234
Verify passphrase: 1234
# Crypto backend (OpenSSL 1.1.1l  24 Aug 2021) initialized in cryptsetup library version 2.3.6.
# Detected kernel Linux 5.10.61-gentoo x86_64.
# PBKDF argon2i, time_ms 2000 (iterations 0), max_memory_kb 1048576, parallel_threads 4.
# Formatting device /dev/loop1 as type LUKS2.
# dm version   [ opencount flush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# Detected dm-ioctl version 4.43.0.
# Detected dm-crypt version 1.22.0.
# Device-mapper backend running with UDEV support enabled.
# Topology: IO (512/0), offset = 0; Required alignment is 1048576 bytes.
# Checking if cipher xchacha20,aes-adiantum-plain64 is usable.
# Using userspace crypto wrapper to access keyslot area.
# Formatting LUKS2 with JSON metadata area 12288 bytes and keyslots area 16744448 bytes.
# Creating new digest 0 (pbkdf2).
# Setting PBKDF2 type key digest 0.
# Running pbkdf2(sha256) benchmark.
# PBKDF benchmark: memory cost = 0, iterations = 1092266, threads = 0 (took 30 ms)
# PBKDF benchmark: memory cost = 0, iterations = 1379705, threads = 0 (took 380 ms)
# PBKDF benchmark: memory cost = 0, iterations = 1539759, threads = 0 (took 681 ms)
# Benchmark returns pbkdf2(sha256) 1539759 iterations, 0 memory, 0 threads (for 256-bits key).
# Segment 0 assigned to digest 0.
# Wiping LUKS areas (0x000000 - 0x1000000) with zeroes.
# Wiping keyslots area (0x008000 - 0x1000000) with random data.
# Reusing open rw fd on device /dev/loop1
# Device size 1072693248, offset 16777216.
# Acquiring write lock for device /dev/loop1.
# Opening lock resource file /run/cryptsetup/L_7:1
# Verifying lock handle for /dev/loop1.
# Device /dev/loop1 WRITE lock taken.
# Trying to write LUKS2 header (16384 bytes) at offset 0.
# Reusing open rw fd on device /dev/loop1
# Checksum:f5db56b94688ac637bb5c588b250621062ea8ef54df824c0f42b21b0f80f3330 (in-memory)
# Trying to write LUKS2 header (16384 bytes) at offset 16384.
# Reusing open rw fd on device /dev/loop1
# Checksum:976d5696895f3893b04269916267fb58f9ddc586b5944cff237773738189788e (in-memory)
# Device /dev/loop1 WRITE lock released.
# Adding new keyslot -1 using volume key.
# Adding new keyslot -1 with volume key assigned to a crypt segment.
# Selected keyslot 0.
# Keyslot 0 assigned to digest 0.
# Trying to allocate LUKS2 keyslot 0.
# Found area 32768 -> 163840
# Running argon2i() benchmark.
# PBKDF benchmark: memory cost = 32, iterations = 4, threads = 4 (took 8 ms)
# PBKDF benchmark: memory cost = 512, iterations = 4, threads = 4 (took 1 ms)
# PBKDF benchmark: memory cost = 8192, iterations = 4, threads = 4 (took 14 ms)
# PBKDF benchmark: memory cost = 146285, iterations = 4, threads = 4 (took 220 ms)
# PBKDF benchmark: memory cost = 166232, iterations = 4, threads = 4 (took 222 ms)
# PBKDF benchmark: memory cost = 187198, iterations = 4, threads = 4 (took 256 ms)
# PBKDF benchmark: memory cost = 1048576, iterations = 5, threads = 4 (took 1813 ms)
# Benchmark returns argon2i() 5 iterations, 1048576 memory, 4 threads (for 256-bits key).
# Calculating attributes for LUKS2 keyslot 0.
# Acquiring write lock for device /dev/loop1.
# Opening lock resource file /run/cryptsetup/L_7:1
# Verifying lock handle for /dev/loop1.
# Device /dev/loop1 WRITE lock taken.
# Checking context sequence id matches value stored on disk.
# Reusing open ro fd on device /dev/loop1
# Updating keyslot area [0x8000].
# Reusing open rw fd on device /dev/loop1
# Device size 1072693248, offset 16777216.
# Device /dev/loop1 WRITE lock already held.
# Trying to write LUKS2 header (16384 bytes) at offset 0.
# Reusing open rw fd on device /dev/loop1
# Checksum:009399d73df98858f952fcff8cac26a2b1cf31a0642d67233707456771cbcaf5 (in-memory)
# Trying to write LUKS2 header (16384 bytes) at offset 16384.
# Reusing open rw fd on device /dev/loop1
# Checksum:ffa2904111af4f5a940c9c86f37773ce49927326ed9dfd0c01c760469f73553b (in-memory)
# Device /dev/loop1 WRITE lock released.
Key slot 0 created.
# Releasing crypt device /dev/loop1 context.
# Releasing device-mapper backend.
# Closing read only fd for /dev/loop1.
# Closing read write fd for /dev/loop1.
# Unlocking memory.
Command successful.

$ cryptsetup --debug open "/dev/loop1" cryptimage
# cryptsetup 2.3.6 processing "cryptsetup --debug open /dev/loop1 cryptimage"
# Running command open.
# Locking memory.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device /dev/loop1.
# Trying to open and read device /dev/loop1 with direct-io.
# Initialising device-mapper backend library.
# Trying to load any crypt type from device /dev/loop1.
# Crypto backend (OpenSSL 1.1.1l  24 Aug 2021) initialized in cryptsetup library version 2.3.6.
# Detected kernel Linux 5.10.61-gentoo x86_64.
# Loading LUKS2 header (repair disabled).
# Acquiring read lock for device /dev/loop1.
# Opening lock resource file /run/cryptsetup/L_7:1
# Verifying lock handle for /dev/loop1.
# Device /dev/loop1 READ lock taken.
# Trying to read primary LUKS2 header at offset 0x0.
# Opening locked device /dev/loop1
# Veryfing locked device handle (bdev)
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:009399d73df98858f952fcff8cac26a2b1cf31a0642d67233707456771cbcaf5 (on-disk)
# Checksum:009399d73df98858f952fcff8cac26a2b1cf31a0642d67233707456771cbcaf5 (in-memory)
# Trying to read secondary LUKS2 header at offset 0x4000.
# Reusing open ro fd on device /dev/loop1
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:ffa2904111af4f5a940c9c86f37773ce49927326ed9dfd0c01c760469f73553b (on-disk)
# Checksum:ffa2904111af4f5a940c9c86f37773ce49927326ed9dfd0c01c760469f73553b (in-memory)
# Device size 1072693248, offset 16777216.
# Device /dev/loop1 READ lock released.
# PBKDF argon2i, time_ms 2000 (iterations 0), max_memory_kb 1048576, parallel_threads 4.
# Activating volume cryptimage using token -1.
# Interactive passphrase entry requested.
Enter passphrase for /tmp/tmp.orSuZwz0bA/partitionme.img: 1234
# Activating volume cryptimage [keyslot -1] using passphrase.
# dm version   [ opencount flush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# Detected dm-ioctl version 4.43.0.
# Detected dm-crypt version 1.22.0.
# Device-mapper backend running with UDEV support enabled.
# dm status cryptimage  [ opencount noflush ]   [16384] (*1)
# Keyslot 0 priority 1 != 2 (required), skipped.
# Trying to open LUKS2 keyslot 0.
# Reading keyslot area [0x8000].
# Acquiring read lock for device /dev/loop1.
# Opening lock resource file /run/cryptsetup/L_7:1
# Verifying lock handle for /dev/loop1.
# Device /dev/loop1 READ lock taken.
# Reusing open ro fd on device /dev/loop1
# Device /dev/loop1 READ lock released.
# Verifying key from keyslot 0, digest 0.
# Loading key (32 bytes, type logon) in thread keyring.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm status cryptimage  [ opencount noflush ]   [16384] (*1)
# Calculated device size is 2062336 sectors (RW), offset 32768.
# DM-UUID is CRYPT-LUKS2-655c479f8cea416786721523fc1101a0-cryptimage
# Udev cookie 0xd4d5ed9 (semid 6) created
# Udev cookie 0xd4d5ed9 (semid 6) incremented to 1
# Udev cookie 0xd4d5ed9 (semid 6) incremented to 2
# Udev cookie 0xd4d5ed9 (semid 6) assigned to CREATE task(0) with flags DISABLE_LIBRARY_FALLBACK         (0x20)
# dm create cryptimage CRYPT-LUKS2-655c479f8cea416786721523fc1101a0-cryptimage [ opencount flush ]   [16384] (*1)
# dm reload cryptimage  [ opencount flush securedata ]   [16384] (*1)
# dm resume cryptimage  [ opencount flush securedata ]   [16384] (*1)
# cryptimage: Stacking NODE_ADD (253,3) 0:0 0600 [trust_udev]
# cryptimage: Stacking NODE_READ_AHEAD 256 (flags=1)
# Udev cookie 0xd4d5ed9 (semid 6) decremented to 1
# Udev cookie 0xd4d5ed9 (semid 6) waiting for zero
# Udev cookie 0xd4d5ed9 (semid 6) destroyed
# cryptimage: Skipping NODE_ADD (253,3) 0:0 0600 [trust_udev]
# cryptimage: Processing NODE_READ_AHEAD 256 (flags=1)
# cryptimage (253:3): read ahead is 256
# cryptimage: retaining kernel read ahead of 256 (requested 256)
Key slot 0 unlocked.
# Releasing crypt device /dev/loop1 context.
# Releasing device-mapper backend.
# Closing read only fd for /dev/loop1.
# Unlocking memory.
Command successful.

$ cryptsetup --debug status "/dev/mapper/cryptimage"
# cryptsetup 2.3.6 processing "cryptsetup --debug status /dev/mapper/cryptimage"
# Running command status.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Initialising device-mapper backend library.
# dm version   [ opencount flush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# Detected dm-ioctl version 4.43.0.
# Detected dm-crypt version 1.22.0.
# Device-mapper backend running with UDEV support enabled.
# dm status cryptimage  [ opencount noflush ]   [16384] (*1)
# Releasing device-mapper backend.
/dev/mapper/cryptimage is active.
# Allocating crypt device context by device /dev/mapper/cryptimage.
# Initialising device-mapper backend library.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm status cryptimage  [ opencount noflush ]   [16384] (*1)
# Releasing device-mapper backend.
# Trying to open and read device /dev/loop1 with direct-io.
# Allocating context for crypt device /dev/loop1.
# Trying to open and read device /dev/loop1 with direct-io.
# Initialising device-mapper backend library.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm table cryptimage  [ opencount flush securedata ]   [16384] (*1)
# Trying to open and read device /dev/loop1 with direct-io.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm deps cryptimage  [ opencount flush ]   [16384] (*1)
# Crypto backend (OpenSSL 1.1.1l  24 Aug 2021) initialized in cryptsetup library version 2.3.6.
# Detected kernel Linux 5.10.61-gentoo x86_64.
# Reloading LUKS2 header (repair disabled).
# Acquiring read lock for device /dev/loop1.
# Opening lock resource file /run/cryptsetup/L_7:1
# Verifying lock handle for /dev/loop1.
# Device /dev/loop1 READ lock taken.
# Trying to read primary LUKS2 header at offset 0x0.
# Opening locked device /dev/loop1
# Veryfing locked device handle (bdev)
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:009399d73df98858f952fcff8cac26a2b1cf31a0642d67233707456771cbcaf5 (on-disk)
# Checksum:009399d73df98858f952fcff8cac26a2b1cf31a0642d67233707456771cbcaf5 (in-memory)
# Trying to read secondary LUKS2 header at offset 0x4000.
# Reusing open ro fd on device /dev/loop1
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:ffa2904111af4f5a940c9c86f37773ce49927326ed9dfd0c01c760469f73553b (on-disk)
# Checksum:ffa2904111af4f5a940c9c86f37773ce49927326ed9dfd0c01c760469f73553b (in-memory)
# Device size 1072693248, offset 16777216.
# Device /dev/loop1 READ lock released.
# PBKDF argon2i, time_ms 2000 (iterations 0), max_memory_kb 1048576, parallel_threads 4.
  type:    LUKS2
# dm versions   [ opencount flush ]   [16384] (*1)
# dm table cryptimage  [ opencount flush securedata ]   [16384] (*1)
# Trying to open and read device /dev/loop1 with direct-io.
  cipher:  xchacha20,aes-adiantum-plain64
  keysize: 256 bits
  key location: keyring
  device:  /dev/loop1
  loop:    /tmp/tmp.orSuZwz0bA/partitionme.img
  sector size:  4096
  offset:  32768 sectors
  size:    2062336 sectors
  mode:    read/write
# Releasing crypt device /dev/loop1 context.
# Releasing device-mapper backend.
# Closing read only fd for /dev/loop1.
Command successful.

$ mkfs.ext4 "/dev/mapper/cryptimage"
mke2fs 1.46.2 (28-Feb-2021)
Creating filesystem with 257792 4k blocks and 64512 inodes
Filesystem UUID: f9925f34-a420-400e-a3ad-95d099f8632b
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376

Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

$ mount "/dev/mapper/cryptimage" "/mnt/"
$ df --human-readable --print-type "/mnt/"
Filesystem             Type  Size  Used Avail Use% Mounted on
/dev/mapper/cryptimage ext4  973M   24K  907M   1% /mnt