search

JsBergbau / Verschiedenes

Sammlung verschiedener Dinge
3 stars 0 forks source link

Finding optimum encrypted RAID5 settings for NAS with RaspberryPI 4 #1

Open JsBergbau opened 4 years ago

JsBergbau commented 4 years ago

To get the optimum performance out of a RaspberryPI4 NAS with openmediavault, I did some benchmarks. Purpose was to find the setup with highest throughput performance for encrypted HDDs with RAID Level 5. Because a lot of measurements were done I wanted to share my results here so others safe a lot of time. Additionally I didn't expect so many performance differences dedending on the sequence the RAID and encryption were built.

TL;DR

Build your encrypted RAID 5 with to get the maximum performance on a RPI4. This gives about 190 MB/s read speed and 100 MB/s write speed.

sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdb /dev/sdc /dev/sdd -c 4096

sudo cryptsetup -v --type luks2 --cipher aes-xts-plain64 --pbkdf argon2id --key-size 512 --hash sha256 --iter-time 4000 --verify-passphrase --use-random luksFormat /dev/md0

sudo cryptsetup open /dev/md0 map

sudo mkfs.ext4 /dev/mapper/map -m 0 -Jsize=100 -O sparse_super -T largefile4

sudo mount /dev/mapper/map test

Long version:

Setup:

cryptsetup benchmark @1,5 GHz (just for reference not used in this testcase) grafik

cryptsetup benchmark @2 GHz grafik

BTRFS

First try was the BTRFS filesystem. sudo cryptsetup -v --type luks2 --cipher aes-xts-plain64 --pbkdf argon2id --key-size 512 --hash sha256 --iter-time 4000 --verify-passphrase --use-random luksFormat /dev/sdb same for sdc and sdd opened via sudo cryptsetup open /dev/sda sdb same for sdc and sdd

mounted via sudo mount /dev/sdb test Note: lsblk shows wrong size with BTRFS

Write performance unencrypted: grafik

Write performance encrypted grafik

Note: As I was not 100% sure if these values were measured overclocked, I've done the measurement again. Then I had only about 50-60 MB/s write performance and I didn't know why. After a reboot the above write performance was restored. So if you notice bad performances, please reboot. All other tests following have also been done again to ensure that not this strange speed reduction was the cause for the bad performance.

Read performance encrypted grafik

RAID5 via MDADM

sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdb /dev/sdc /dev/sdd Before all following benchmarks wait until cat /proc/mdstatshows no recovery process anymore. This recovery with Raid 5 direclty after creation is a normal behaviour. It has to be done to get correct checksums.

Unencrypted benchmark, default Chunksize 512K grafik

Unencrypted 4K Chunksize grafik

Unencrypted 4096K Chunksize grafik

Unencrypted 16384K Chunksize grafik

Unencrypted 8192K Chunksize grafik

So 4096K was the fastest Chunksize for unencrypted scenario.

Unencryted reading with 512K Chunksize grafik

Unencrypted reading with 4096K Chunksize grafik

Encrypted RAID5 and MDADM

Encryption at HDD Level

Let's get encrypted: Now every single HDD was encrypted with luks, like above example with BTRFS. Then the RAID was build with the /dev/mapper devices as input, also same like the BTRFS example. Performance was quite bad, CPU Usage varied extreme between 25 and 66%. That explains the bad performance.

4096K Chunksize grafik

512K Chunksize grafik

Encrypt the whole RAID

In the next tests raid was built with the 3 physical devices

sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdb /dev/sdc /dev/sdd

sudo cryptsetup -v --type luks2 --cipher aes-xts-plain64 --pbkdf argon2id --key-size 512 --hash sha256 --iter-time 4000 --verify-passphrase --use-random luksFormat /dev/md0

sudo cryptsetup open /dev/md0 map

sudo mkfs.ext4 /dev/mapper/map -m 0 -Jsize=100 -O sparse_super -T largefile4

sudo mount /dev/mapper/map test

512K Chunksize, write: grafik

512K Chunksize, read: grafik

Best read performance with good write performance

4096K Chunksize, write: grafik Writing is a bit slower with with 4096K chunksize

However reading is now insane fast, only a little smaller than unencrypted. grafik I even wrote from /dev/urandom to testfile to prevent any buffers or stuff like that. Still we get almost doubled read speed compared to write speed.

RAID5 with LVM

Next test was RAID5 with LVM:

sudo lvcreate --type raid5 -l 100%FREE --nosync -n testlv testvg
sudo mkfs.ext4 /dev/testvg/testlv -m 0 -Jsize=100 -O sparse_super -T largefile4

Resulting write speed: grafik

Note: This bad performance is without encryption. Because of that bad performance without encrytpion no further tests with LVM-RAID5 were made.

MarcS1975 commented 3 years ago

Great comparison! thanks. One question re our setup: You say Note: You should overclock before installing OpenMediaVault otherwise COU speed was even reduced. Just double check if your CPU really runs overclock when you overlock afterwards.

Why would instlling OMVreduce the overclokcing speed of the Rspberry?

JsBergbau commented 3 years ago

You're welcome.

Why would instlling OMVreduce the overclokcing speed of the Rspberry?

OMV uses or installs its own CPU governor and seems to store available values on setup. So when not overlock, OMV doesn't recognise CPU can go faster now. However I think I've remembered something in the changelog that they've changed this behaviour. So it might be outdated in the meantime.

MarcS1975 commented 3 years ago

understood. I have this problem whereby I am overclocking the Pi4 via config.txt (to 2000) but when checking the MHz after booting OMV it always remains at 1500. Do you know where I can check if OMV is overwriting CPU speed somwhere?

On Sun, Jan 10, 2021 at 11:02 PM JsBergbau notifications@github.com wrote:

You're welcome.

Why would instlling OMVreduce the overclokcing speed of the Rspberry?

OMV uses or installs its own CPU governor and seems to store available values on setup. So when not overlock, OMV doesn't recognise CPU can go faster now. However I think I've remembered something in the changelog that they've changed this behaviour. So it might be outdated in the meantime.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/JsBergbau/Verschiedenes/issues/1#issuecomment-757559673, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMNNHGSZMQR74QUCCMU7X4LSZIWZXANCNFSM4K7VB73A .

JsBergbau commented 3 years ago

That looks exactly like the problem I've encountered. Sorry I don't know what exactly to do here. I suggest you ask in an OMV forum or open an OMV issue. Can you please report your solution to fix this problem here?

MarcS1975 commented 3 years ago

Solution from OMV Forum. It worked for me !

It is setting cpufrequtils settings. So, change OMV_CPUFREQUTILS_MAXSPEED in /etc/default/openmediavault and MAX_SPEED in /etc/default/cpufrequtils. Then do:

omv-salt stage run prepare

omv-salt deploy run cpufrequtils

On Mon, Jan 11, 2021 at 7:31 AM JsBergbau notifications@github.com wrote:

That looks exactly like the problem I've encountered. Sorry I don't know what exactly to do here. I suggest you ask in an OMV forum or open an OMV issue. Can you please report your solution to fix this problem here?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/JsBergbau/Verschiedenes/issues/1#issuecomment-757674997, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMNNHGRZDNAK7Y42ZQ7X3JDSZKSMJANCNFSM4K7VB73A .

bobafetthotmail commented 3 years ago

Hi, can you tell me if the SMART commands work on disks installed in your IB-3740-C31? for example sudo smartctl --all /dev/sda This command is used to see health status of the hard drives, I think it is also used by OMV when you check the hard drive status/health from web interface.

JsBergbau commented 3 years ago

I don't have the Icy Box IB-3740-C31 case anymore. It consumed too much idle power (9 W, more than 3 times more than the raspberry PI when all disks where in standby / not spinning. You have to measure, when case is connected to USB, because then it consumes much more power) and after power outage you had to press the on button by hand. Not very clever.

But I think SMART worked, however can't tell exactly. What I remember and definitively worked was sudo hdparm -C /dev/sda. This command is to check if disk is spinning or not. This command works only on a very few external HDD enclosures correct. So it should work on Icy Box IB-3740-C31. Perhaps it is required to to do some storage quirks kernel parameters. Use lsusb to get ID. Then you can google for it.

bobafetthotmail commented 3 years ago

Thanks, I did not buy it yet, I am doing some research on what is the best device to buy for a similar job.

You said you do not have it anymore, did you find a better enclosure?

JsBergbau commented 3 years ago

I did test a lot of HDD enclosures to get very little standby power consumption.

Concerning power consumption I use 4 seperate single HDD enclosures. Cheap and lowest powerconsumption has Icy Box IB-AC705-6G. However it has USB back power. You need an active hub with multiple devices, because despite of its PSU it draws a lot from USB port depending of the device state.

I'm using also Sharkoon Swift Case Pro USB 3.0. With HDD spun down 1,7 watt power consumption.

I also use ICY Box IB-377U3. But you should flash another firmware from here https://forum.odroid.com/viewtopic.php?f=53&t=41926 to get real drive serial number. Power consumption with HDD spun down 1,3 watt. However sudo hdparm -C /dev/sda to show whether disk is spinning is not working. It will always wake up device. Only Sharkoon case it where it works. All other HDD enclosures in this review don't support it correctly.

martinvanmiddelkoop commented 2 years ago

This is really great, thanks a lot for doing this and publishing it!

Would this RAID-5 with encryption of the whole RAID volume work well with 3 USB 3.0 external 2.5" disks attached through one powered USB 3.0 hub, would you reckon? I will connect the Raspberry Pi to the network using the 1Gbit port.

MarcS1975 commented 2 years ago

I have 8x 2.5'' sata HDs encrypted (2x usb hubs) and some via raid connected and no problems.

JsBergbau commented 2 years ago

Thanks for appreciation.

When using 2.5" HDDs it depends of your hubs. When the power supply is strong enough three USB 3.0 2.5" disks are no problem.

MarcS1975 commented 2 years ago

For my 8x hds I needed external power. Anything more than 2x Hds via USB I would highly recommend additional power because the PI4 does not provide sufficient juice.