Open JsBergbau opened 4 years ago
Great comparison! thanks. One question re our setup: You say Note: You should overclock before installing OpenMediaVault otherwise COU speed was even reduced. Just double check if your CPU really runs overclock when you overlock afterwards.
Why would instlling OMVreduce the overclokcing speed of the Rspberry?
You're welcome.
Why would instlling OMVreduce the overclokcing speed of the Rspberry?
OMV uses or installs its own CPU governor and seems to store available values on setup. So when not overlock, OMV doesn't recognise CPU can go faster now. However I think I've remembered something in the changelog that they've changed this behaviour. So it might be outdated in the meantime.
understood. I have this problem whereby I am overclocking the Pi4 via config.txt (to 2000) but when checking the MHz after booting OMV it always remains at 1500. Do you know where I can check if OMV is overwriting CPU speed somwhere?
On Sun, Jan 10, 2021 at 11:02 PM JsBergbau notifications@github.com wrote:
You're welcome.
Why would instlling OMVreduce the overclokcing speed of the Rspberry?
OMV uses or installs its own CPU governor and seems to store available values on setup. So when not overlock, OMV doesn't recognise CPU can go faster now. However I think I've remembered something in the changelog that they've changed this behaviour. So it might be outdated in the meantime.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/JsBergbau/Verschiedenes/issues/1#issuecomment-757559673, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMNNHGSZMQR74QUCCMU7X4LSZIWZXANCNFSM4K7VB73A .
That looks exactly like the problem I've encountered. Sorry I don't know what exactly to do here. I suggest you ask in an OMV forum or open an OMV issue. Can you please report your solution to fix this problem here?
Solution from OMV Forum. It worked for me !
It is setting cpufrequtils settings. So, change OMV_CPUFREQUTILS_MAXSPEED in /etc/default/openmediavault and MAX_SPEED in /etc/default/cpufrequtils. Then do:
omv-salt stage run prepare
omv-salt deploy run cpufrequtils
On Mon, Jan 11, 2021 at 7:31 AM JsBergbau notifications@github.com wrote:
That looks exactly like the problem I've encountered. Sorry I don't know what exactly to do here. I suggest you ask in an OMV forum or open an OMV issue. Can you please report your solution to fix this problem here?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/JsBergbau/Verschiedenes/issues/1#issuecomment-757674997, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMNNHGRZDNAK7Y42ZQ7X3JDSZKSMJANCNFSM4K7VB73A .
Hi, can you tell me if the SMART commands work on disks installed in your IB-3740-C31? for example sudo smartctl --all /dev/sda This command is used to see health status of the hard drives, I think it is also used by OMV when you check the hard drive status/health from web interface.
I don't have the Icy Box IB-3740-C31 case anymore. It consumed too much idle power (9 W, more than 3 times more than the raspberry PI when all disks where in standby / not spinning. You have to measure, when case is connected to USB, because then it consumes much more power) and after power outage you had to press the on button by hand. Not very clever.
But I think SMART worked, however can't tell exactly. What I remember and definitively worked was sudo hdparm -C /dev/sda
. This command is to check if disk is spinning or not. This command works only on a very few external HDD enclosures correct. So it should work on Icy Box IB-3740-C31. Perhaps it is required to to do some storage quirks kernel parameters. Use lsusb
to get ID. Then you can google for it.
Thanks, I did not buy it yet, I am doing some research on what is the best device to buy for a similar job.
You said you do not have it anymore, did you find a better enclosure?
I did test a lot of HDD enclosures to get very little standby power consumption.
Concerning power consumption I use 4 seperate single HDD enclosures. Cheap and lowest powerconsumption has Icy Box IB-AC705-6G. However it has USB back power. You need an active hub with multiple devices, because despite of its PSU it draws a lot from USB port depending of the device state.
I'm using also Sharkoon Swift Case Pro USB 3.0. With HDD spun down 1,7 watt power consumption.
I also use ICY Box IB-377U3. But you should flash another firmware from here https://forum.odroid.com/viewtopic.php?f=53&t=41926 to get real drive serial number. Power consumption with HDD spun down 1,3 watt. However sudo hdparm -C /dev/sda
to show whether disk is spinning is not working. It will always wake up device. Only Sharkoon case it where it works. All other HDD enclosures in this review don't support it correctly.
This is really great, thanks a lot for doing this and publishing it!
Would this RAID-5 with encryption of the whole RAID volume work well with 3 USB 3.0 external 2.5" disks attached through one powered USB 3.0 hub, would you reckon? I will connect the Raspberry Pi to the network using the 1Gbit port.
I have 8x 2.5'' sata HDs encrypted (2x usb hubs) and some via raid connected and no problems.
Thanks for appreciation.
When using 2.5" HDDs it depends of your hubs. When the power supply is strong enough three USB 3.0 2.5" disks are no problem.
For my 8x hds I needed external power. Anything more than 2x Hds via USB I would highly recommend additional power because the PI4 does not provide sufficient juice.
To get the optimum performance out of a RaspberryPI4 NAS with openmediavault, I did some benchmarks. Purpose was to find the setup with highest throughput performance for encrypted HDDs with RAID Level 5. Because a lot of measurements were done I wanted to share my results here so others safe a lot of time. Additionally I didn't expect so many performance differences dedending on the sequence the RAID and encryption were built.
TL;DR
Build your encrypted RAID 5 with to get the maximum performance on a RPI4. This gives about 190 MB/s read speed and 100 MB/s write speed.
Long version:
Setup:
cryptsetup benchmark @1,5 GHz (just for reference not used in this testcase)![grafik](https://user-images.githubusercontent.com/37013344/75684082-12e0f480-5c90-11ea-92e9-58b8e11d9000.png)
cryptsetup benchmark @2 GHz![grafik](https://user-images.githubusercontent.com/37013344/75684140-1f654d00-5c90-11ea-86b2-1463222e736e.png)
BTRFS
First try was the BTRFS filesystem.
sudo cryptsetup -v --type luks2 --cipher aes-xts-plain64 --pbkdf argon2id --key-size 512 --hash sha256 --iter-time 4000 --verify-passphrase --use-random luksFormat /dev/sdb
same for sdc and sdd opened viasudo cryptsetup open /dev/sda sdb
same for sdc and sddsudo mkfs.btrfs --metadata raid5 --data raid5 /dev/mapper/sdb /dev/mapper/sdc /dev/mapper/sdd
mounted via
sudo mount /dev/sdb test
Note:lsblk
shows wrong size with BTRFSWrite performance unencrypted:![grafik](https://user-images.githubusercontent.com/37013344/75683895-d3b2a380-5c8f-11ea-982a-56408bded61e.png)
Write performance encrypted![grafik](https://user-images.githubusercontent.com/37013344/75684257-53d90900-5c90-11ea-8d70-5658cf5e9746.png)
Note: As I was not 100% sure if these values were measured overclocked, I've done the measurement again. Then I had only about 50-60 MB/s write performance and I didn't know why. After a reboot the above write performance was restored. So if you notice bad performances, please reboot. All other tests following have also been done again to ensure that not this strange speed reduction was the cause for the bad performance.
Read performance encrypted![grafik](https://user-images.githubusercontent.com/37013344/75684270-5d627100-5c90-11ea-8b01-b3ca41b00738.png)
RAID5 via MDADM
sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdb /dev/sdc /dev/sdd
Before all following benchmarks wait untilcat /proc/mdstat
shows no recovery process anymore. This recovery with Raid 5 direclty after creation is a normal behaviour. It has to be done to get correct checksums.Unencrypted benchmark, default Chunksize 512K![grafik](https://user-images.githubusercontent.com/37013344/75685364-5c324380-5c92-11ea-8170-5d11d2bb7e58.png)
Unencrypted 4K Chunksize![grafik](https://user-images.githubusercontent.com/37013344/75685442-766c2180-5c92-11ea-8f22-f5ed2f096f67.png)
Unencrypted 4096K Chunksize![grafik](https://user-images.githubusercontent.com/37013344/75685475-83891080-5c92-11ea-8690-a3e8e726c5e1.png)
Unencrypted 16384K Chunksize![grafik](https://user-images.githubusercontent.com/37013344/75685588-b3381880-5c92-11ea-9747-747408659cee.png)
Unencrypted 8192K Chunksize![grafik](https://user-images.githubusercontent.com/37013344/75685615-bdf2ad80-5c92-11ea-8588-59c6e27a8c06.png)
So 4096K was the fastest Chunksize for unencrypted scenario.
Unencryted reading with 512K Chunksize![grafik](https://user-images.githubusercontent.com/37013344/75686067-889a8f80-5c93-11ea-8234-41b62aabf4c1.png)
Unencrypted reading with 4096K Chunksize![grafik](https://user-images.githubusercontent.com/37013344/75987442-d7406780-5ee7-11ea-9105-b666cca7f4c5.png)
Encrypted RAID5 and MDADM
Encryption at HDD Level
Let's get encrypted: Now every single HDD was encrypted with luks, like above example with BTRFS. Then the RAID was build with the /dev/mapper devices as input, also same like the BTRFS example. Performance was quite bad, CPU Usage varied extreme between 25 and 66%. That explains the bad performance.
4096K Chunksize![grafik](https://user-images.githubusercontent.com/37013344/75686415-1eceb580-5c94-11ea-9cb3-a9da2f9c14ae.png)
512K Chunksize![grafik](https://user-images.githubusercontent.com/37013344/75686435-255d2d00-5c94-11ea-8029-079e337fed83.png)
Encrypt the whole RAID
In the next tests raid was built with the 3 physical devices
512K Chunksize, write:![grafik](https://user-images.githubusercontent.com/37013344/75688526-8b977f00-5c97-11ea-85c4-ca5e071c2c32.png)
512K Chunksize, read:![grafik](https://user-images.githubusercontent.com/37013344/75688547-93572380-5c97-11ea-96eb-613680af3950.png)
Best read performance with good write performance
4096K Chunksize, write:
Writing is a bit slower with with 4096K chunksize
However reading is now insane fast, only a little smaller than unencrypted.
I even wrote from /dev/urandom to testfile to prevent any buffers or stuff like that. Still we get almost doubled read speed compared to write speed.
RAID5 with LVM
Next test was RAID5 with LVM:
Resulting write speed:![grafik](https://user-images.githubusercontent.com/37013344/75688923-36a83880-5c98-11ea-93fe-ff051f1a52e8.png)
Note: This bad performance is without encryption. Because of that bad performance without encrytpion no further tests with LVM-RAID5 were made.