search

veracrypt / VeraCrypt

Disk encryption with strong security based on TrueCrypt
https://www.veracrypt.fr
Other
6.9k stars 946 forks source link

Using volumes via network (SMB,iSCSI) makes it slow - way slower than regular SMB,iSCSI access #1277

Open kstepyra opened 10 months ago

kstepyra commented 10 months ago

I am connected to 8 drive NAS via 10gbe, when i copy to it i can top out >1GB/s of transfer. Whenever i try to copy anything to/from VC container it tops out at about 100-150MB/s. Changing NTFS block size doesn't change anything. Tried changing everything - network stack on TrueNAS, all filesystem types/block sizes on VC container, system on main machine - nothing helps. Last thing i didn't check is newer VC version - but i don't see anything related in changelog.

Expected behavior

Veracrypt shouldn't slow down drastically when accessing VC container via network (10% of possible speed here)

Observed behavior

Copying files from/to mounted VC over SMB/iSCSI is very slow

Steps to reproduce

  1. Write/read from NAS (SMB/iSCSI) to benchmark its speed
  2. Create VC container (any filesystem type) on NAS via SMB/iSCSI, with AES for best enc/dec speed on modern CPU
  3. Write/Read files from it on Windows 10 or Ubuntu 22 server machine
  4. Don't benefit :)

Your Environment

NAS server is on TrueNAS ZFS array (raidz1) with 128KB block size, connected using 10gbit nic Main machine is either ubuntu or windows 10, both with 5950x, 128GB of RAM and samsung 970 evo plus 2TB drive.

VeraCrypt version: 1.25.9 x64

Operating system and version: Windows 10 version 22H2 build 19045.3693

System type: 64-bit

nerai commented 10 months ago

If you have an SSD in the machine, can you test the R/W speed with a VC container or partition there? This should help to determine if the medium being a RAID is a factor.

kstepyra commented 10 months ago

Directly on TrueNAS server, without client machine and SMB?

thany commented 8 months ago

Same problem here. It oftenly grinds to below 1MB/s even though the target server has modern disks that are otherwise idle and capable of pulling 200MB/s. It's also on 2.5Gb ethernet that is also otherwise idle. Safe to say, in my case, the server or my connection to it, is absolutely not the problem.

I did notice that VC makes the server work REALLY HARD, by somehow writing super slowly but maybe in random places or something. Not a problem for SSDs, but a huge pita for HDDs. Don't know if this is the case, but again, dragging a huge regular file to the same SMB share writes it at full blast without problems.

Moreover, while this grinding to a halt is happening, the local pc running VC also grinds. Many I/O based operations are delayed, leading to huge lags all over the system. Even something as simple as opening task manager took almost a minute. This is absolutely unacceptable on the kind of system I have, and it's completely VC's fault, because the pc is otherwise as fast as a mortal person can have in this day and age.

And the fact that VC happily "bursts" or "buffers" writes well in excess of the target volume's maximum speed, makes the problem even worse. This means it will keep grinding and grinding and grinding well after having sucessfully cancelled a copy operation to the VC volume.

So yeah, I don't know what's going on with VC or its driver or whatever, but there's a seriously bad problem in there somewhere. Probably inherited from TrueCrypt, but it's there nonetheless.

nerai commented 8 months ago

The relevant points, as far as I can tell:

The important part seems to be the combination of VC, SMB and RAID. If any of these factors is omitted, it works fine. I've confirmed that TrueCrypt behaves identically in these cases.

It would be great if others could confirm if this is the case for them, too, or if their problem requires other/fewer factors.

It would perhaps also be interesting to see which type of RAID (hardware, mdadm, zfs, ...), which level (0/1/5/6/...), which block size etc. I've tried settings that should be advantageous for VC but the improvements were insufficient.

thany commented 8 months ago

It would perhaps also be interesting to see which type of RAID (hardware, mdadm, zfs, ...), which level (0/1/5/6/...), which block size etc. I've tried settings that should be advantageous for VC but the improvements were insufficient.

For me it's Unraid. My VC containers are on the main array, consisting of 4 disks, one of which is parity. This is not exactly RAID (that's the whole point of Unraid), but it does keep parity. So perhaps it's something to do with the block sizes that VC writes, or block alignment, or something along those lines. Probably the reason why reading performs normal, afaict, but writing speed degrades pretty quickly.

But this still doesn't address VC's over-willingness to burst writes, to sort of "take" a large chunk of write operations up to a point, at incredible speed, well in excess of whatever the physical volume file is stored on. This bursting behaviour certainly doesn't help in this scenario.

amadboss commented 8 months ago

The relevant points, as far as I can tell:

  • VC on a local machine is fast, for both SSD and HDD (100 MB/s or more). This rules out the CPU as a culprit.
  • VC on a remote drive that is NOT in a RAID, accessed via SMB, is fast (?) or at least fast-ish (50+ MB/s for large file transfers). This rules out the RAID as a definite/unambiguous bottleneck.
  • VC on a remote drive that IS in a RAID of some kind, accessed via SMB, can be slow (3 MB/s or so).
  • The same remote drive in a RAID, accessed via SMB, WITHOUT VC, is fast (100+ MB/s). This rules out the connection and the top speed of the remote drive as culprits.

The important part seems to be the combination of VC, SMB and RAID. If any of these factors is omitted, it works fine. I've confirmed that TrueCrypt behaves identically in these cases.

It would be great if others could confirm if this is the case for them, too, or if their problem requires other/fewer factors.

It would perhaps also be interesting to see which type of RAID (hardware, mdadm, zfs, ...), which level (0/1/5/6/...), which block size etc. I've tried settings that should be advantageous for VC but the improvements were insufficient.

I personaly use OMV with 4 disk in raid 5 (they use mdadm for RAID) and i have exactely the same issue as mentioned. With Raid + smb + vc my netword speed drop drop ~1Gbps (with raid + smb) to 40Mbps (raid+smb+vc). I test pretty mush all settings on vc container like buffer size, file system or encryption algorithm but nothing change.

Does anyone have some news about this issue on how to solve and why is this happening ?

thany commented 8 months ago

Does anyone have some news about this issue on how to solve and why is this happening ?

Yeah I was hoping the veracrypt folks would have replied by now. At least something like "we're on it" would be more affirming than nothing at all.

nerai commented 5 months ago

Does anyone have some news about this issue on how to solve and why is this happening ?

Yeah I was hoping the veracrypt folks would have replied by now. At least something like "we're on it" would be more affirming than nothing at all.

This issue is quite hard to replicate because it requires specific hardware and, apart from that, I'm sure they have a life of their own, so this may take some time. We should wait patiently :)

thany commented 5 months ago

@nerai We should, but it can't hurt if the developer could post at least a quick status update, rather than leaving us "in the dark", so to speak. That way, at the very least, it's an indication that it's on their todo list.

thany commented 2 months ago

It's 3 months later, and an update would be both helpful and greatly appreciated.

I'm seriously not sure what the solution is. I don't even know how to diagnose any of this, as it's quite "low level" and therefor hard to "get to" from userland.

I'm happy to try out some stuff, but I don't know what 😞

vedimh commented 2 months ago

Hi, I can confirm this issue and add some new details. I have four computers in local network (1Gbps) - two Windows 10 and two Debian 12 (dual boot Windows 10).

When I mount Veracrypt file from NTFS shared drive on Debian 12 and try to copy some larger file to it (upload from Windows 10 to Debian), transfer speed never exceeds 60-70mbps. But, if copy something in opposite direction (download from Debian to Windows 10), I get around 700Mbps. This happens with both Debian PCs.

Also, if I reboot those two back to Windows 10 and then mount Veracrypt over shares again, I get over 600Mbps in both directions.

I should also mention that copying files between Windows and Debian (no Veracrypt mounts) works with full 1Gbps in both directions.

Any help greatly appreciated :)

IntCleastwood commented 2 months ago

Same here in combination with MacOS Sonoma + Unraid NAS connected via 2.5 Gbit. I am connected via WiFi6 and sitting near to the router, where iperf3 shows me approx 2 - 2,1 GBit/s for general network speed between NAS + Macbook.

Following different threads, especially samba + MacOS seems to be an unlucky combination, but i can gain up to 180 MByte/s for a general file copy from my Macbook to the NAS share (no encryption involved).

Creating a container directly on the samba share drops to 100 MByte/s which is okayish/expected, since its only a one-timer.

But the mounted container via SMB share is painfully slow, about 20-30 MByte/s which makes it unusable for example for photo editing.

thany commented 2 months ago

I've moved a container on Unraid from the main array (which is spinning rust) to a RAID-5 ZFS array (which is SSD all the way). Mounting that on Windows via an SMB share yields in the same performance, which doesn't make any sense. Surely, 4 SSD's raided together should easily be able to saturate the 2.5Gb wired networking between both.

And also, Veracrypts bursting behaviour makes it even slower, by faking high speed first and then having to write back all that data at reduced "real" speed.

nerai commented 2 months ago

What happens if you access local storage via SMB instead of directly? Does that result in great slowdown, too?

vedimh commented 2 months ago

@nerai No, at least not for me. For me, the problem appears only when mounting Veracrypt volume hosted on Linux to Windows PC. In this case transfer from Windows to Linux (copying file to Veracrypt volume) is extremely slow, but transfer in opposite direction is just fine. Also, all other mounting combinations work without issues - local mounts, Windows to Windows, Linux to Linux and Linux to Windows. It is really,really weird.

thany commented 1 month ago

I wonder if we can disable write bursting, or whatever it's called, so that copy operations can be tested for speed more easily and genuinely.

With VC bursting writes, it's really difficult to see if a copy to a volume is genuinely fast, or if it'll grind down after "taking" a few hundred MB.

nerai commented 1 month ago

A few 100MB? For me the burst buffer seems to be about 10GB. I have 128GB RAM, maybe that's influencing it.