Performance - NTFS vs WinBtrfs

theChaosCoder commented 7 years ago

I couldn't find any benchmarks or any information about the performance of WinBtrfs, so I made a quick test with CrystalDiskMark 5. It seems that winbtrfs has better random writes/reads but lower seq. write/read performance.

Have you made similar observations?

OS : Windows 10 Professional [10.0 Build 16299] (x64) WinBtrfs v1.0.1 on HDD

WinBTRFS with LZO Sequential Read (Q= 32,T= 1) : 119.619 MB/s Sequential Write (Q= 32,T= 1) : 93.616 MB/s Random Read 4KiB (Q= 32,T= 1) : 1.381 MB/s [ 337.2 IOPS] Random Write 4KiB (Q= 32,T= 1) : 1.874 MB/s [ 457.5 IOPS] Sequential Read (T= 1) : 117.870 MB/s Sequential Write (T= 1) : 122.909 MB/s Random Read 4KiB (Q= 1,T= 1) : 0.672 MB/s [ 164.1 IOPS] Random Write 4KiB (Q= 1,T= 1) : 4.304 MB/s [ 1050.8 IOPS]

NTFS Sequential Read (Q= 32,T= 1) : 152.757 MB/s Sequential Write (Q= 32,T= 1) : 145.346 MB/s Random Read 4KiB (Q= 32,T= 1) : 0.951 MB/s [ 232.2 IOPS] Random Write 4KiB (Q= 32,T= 1) : 1.098 MB/s [ 268.1 IOPS] Sequential Read (T= 1) : 154.369 MB/s Sequential Write (T= 1) : 146.380 MB/s Random Read 4KiB (Q= 1,T= 1) : 0.395 MB/s [ 96.4 IOPS] Random Write 4KiB (Q= 1,T= 1) : 1.153 MB/s [ 281.5 IOPS]

btrfs

maharmstone commented 7 years ago

Thank you, that's interesting to know!

My guess is that the sequential reads and writes are slower because unlike NTFS Btrfs does checksumming. Have you tried running it in a directory with the nocsum flag turned on?

theChaosCoder commented 7 years ago

I can't find such a flag in btrfs properties. I guess I need btrfs-tools for that!?

And CrystalDiskMark creates a random folder in the root dir everytime... can I set nocsum for the whole drive?

maharmstone commented 7 years ago

If you have the shell extension installed, it should be a checkbox on the Btrfs properties tab. And yes, from memory I think you can set it on the root directory.

theChaosCoder commented 7 years ago

I can see Disable Copy-on-Write (= nocsum?).

I used dd now for testing seq speed dd if=/dev/zero of=asd bs=1M count=2024 conv=fdatasync,notrunc

With cow disabled the speed is more or less the same. With cow enabled it is about 30-40mb/s slower.

theChaosCoder commented 7 years ago

Now with my ssd

btrfs_ssd

claunia commented 6 years ago

@theChaosCoder what about disabling lzo but leaving cow? lzo makes btrfs be more cpu dependent and takes speed away (how much depends on your cpu)

theChaosCoder commented 6 years ago

SSD - WinBtrfs 1.0.1 (Debug) @maharmstone Could the debug version of winbtrfs affect the write rate significantly?

divinity76 commented 6 years ago

actually the compression should make the /dev/zero test much FASTER, as zeroes compress extremely well

Could the debug version of winbtrfs affect the write rate significantly? - idk, but it could definitely have a significant impact on the compression

@maharmstone are you sure its called nocsum ? because in linux its called nodatasum

daiaji commented 6 years ago

@theChaosCoder Cow seems to have a big impact on write performance The native btrfs of the Linux kernel may perform better? May NTFS compression be worse than Btrfs's lzo performance? CrystalDiskMark may be able to work on Linux mono

theChaosCoder commented 6 years ago

According to this benchmark https://quixdb.github.io/squash-benchmark/ the ms-compress:LZNT1 is indeed slower then lzo. Since windows8.1 you can compress files also with xpress and lzx. But I couldn't find any usefull benchmarks.

Wimlib implemented both of them https://wimlib.net/compression.html

daiaji commented 6 years ago

@theChaosCoder It's interesting! I saw lz4 with amazing compression speed and pressure! In terms of revenue, this is far more applicable to transparent compression of file systems than Lzo.

I saw lz4 with amazing compression speed and pressure! In terms of revenue, this is far more applicable to transparent compression of file systems than Lzo.

But they don't seem to care lz4😥

Btrfs seems to be showing poor 4K write performance. Is this perhaps a reason for non-native support?

mirh commented 5 years ago

It would be also cool to test how performance compares with the "original" linux.

corngood commented 5 years ago

I was just testing git status (using cygwin git), or a pretty large repo:

NTFS: git status -s 4.40s user 23.98s system 257% cpu 11.034 total BTRFS 1.1: git status -s 6.84s user 417.06s system 104% cpu 6:45.84 total

So approximately 40x slower. BTRFS seems to be highly CPU limited. The git process has many threads all in sync primitives, for example:

0   ntoskrnl.exe!KeSynchronizeExecution+0x5be6
1   ntoskrnl.exe!KeWaitForSingleObject+0x12e6
2   ntoskrnl.exe!KeWaitForSingleObject+0xadb
3   ntdll.dll!ZwClose+0x14
4   KERNELBASE.dll!CloseHandle+0x62
5   cygwin1.dll!setpassent+0xa87
6   cygwin1.dll!umask+0xf97
7   cygwin1.dll!acl_get_perm+0x930a

or

0   ntoskrnl.exe!KeSynchronizeExecution+0x5be6
1   ntoskrnl.exe!KeWaitForSingleObject+0x12e6
2   ntoskrnl.exe!KeWaitForSingleObject+0xadb
3   ntoskrnl.exe!KeWaitForSingleObject+0x1ff
4   ntoskrnl.exe!IofCompleteRequest+0x748
5   ntoskrnl.exe!KeLeaveCriticalRegion+0x37
6   btrfs.sys+0xa502
7   ntoskrnl.exe!IofCallDriver+0x59
8   FLTMGR.SYS!FltIsCallbackDataDirty+0x2e7
9   FLTMGR.SYS!FltDecodeParameters+0x406
10  ntoskrnl.exe!IofCallDriver+0x59
11  cbfs6.sys+0x2e3b
12  cbfs6.sys+0x10da
13  ntoskrnl.exe!IofCallDriver+0x59
14  ntoskrnl.exe!SeQueryInformationToken+0x2e14
15  ntoskrnl.exe!NtDeviceIoControlFile+0x1e40
16  ntoskrnl.exe!ObfDereferenceObjectWithTag+0xc6
17  ntoskrnl.exe!SeLockSubjectContext+0x899
18  ntoskrnl.exe!NtClose+0xcd
19  ntoskrnl.exe!setjmpex+0x6f83
20  ntdll.dll!ZwClose+0x14
21  KERNELBASE.dll!CloseHandle+0x62
22  cygwin1.dll!setpassent+0xa87
23  cygwin1.dll!umask+0xf97
24  cygwin1.dll!acl_get_perm+0x930a

I haven't yet dug into the code enough to understand why this would be so much worse than NTFS, but it's bad enough that I'll have to at least investigate before using it for day-to-day work.

corngood commented 5 years ago

That cbfs6.sys looked suspicious to me, so I disabled it. I'm not sure how I ended up with it installed.

The status time was reduced a lot: git status -s 4.48s user 64.51s system 121% cpu 56.931 total. Now it's only 6x slower. There was no noticeable effect on NTFS.

On NTFS, git uses ~22% of a 16-thread ryzen CPU, but on BTRFS it uses ~7% (one core). I'm still getting similar call stacks:

ntoskrnl.exe!KeSynchronizeExecution+0x5be6
ntoskrnl.exe!KeWaitForSingleObject+0x12e6
ntoskrnl.exe!KeWaitForSingleObject+0xadb
ntoskrnl.exe!KeWaitForSingleObject+0x1ff
ntoskrnl.exe!KeCheckProcessorGroupAffinity+0x98a
ntoskrnl.exe!ExIsResourceAcquiredSharedLite+0x7cb
ntoskrnl.exe!KiCheckForKernelApcDelivery+0x27
ntoskrnl.exe!KeLeaveCriticalRegion+0x37
btrfs.sys+0xa502
ntoskrnl.exe!IofCallDriver+0x59
FLTMGR.SYS!FltIsCallbackDataDirty+0x2e7
FLTMGR.SYS!FltDecodeParameters+0x406
ntoskrnl.exe!IofCallDriver+0x59
ntoskrnl.exe!SeQueryInformationToken+0x2e14
ntoskrnl.exe!NtDeviceIoControlFile+0x1e40
ntoskrnl.exe!ObfDereferenceObjectWithTag+0xc6
ntoskrnl.exe!SeLockSubjectContext+0x899
ntoskrnl.exe!NtClose+0xcd
ntoskrnl.exe!setjmpex+0x6f83
ntdll.dll!ZwClose+0x14
KERNELBASE.dll!CloseHandle+0x62
cygwin1.dll!setpassent+0xa87
cygwin1.dll!umask+0xf97
cygwin1.dll!acl_get_perm+0x930a

maharmstone commented 5 years ago

Thanks David. btrfs.sys+0xa502 is the call to FsRtlExitFileSystem at the end of drv_close, which makes sense given the call stack.

One characteristic of Git is that it has lots of small files... my hunch is that you have a lot of data stored as inline files, and it's choking on the per-subvolume metadata lock. I'll give it a proper look when I get the chance.

corngood commented 5 years ago

Yeah, I'm running git status on a clean working tree with 174k files. So it should just be a ton of metadata reads.

Could you point me at the lock you're describing? I'm guessing somewhere under drv_create?

maharmstone commented 5 years ago

Well, it turned out to be a whole number of things. It's nowhere near done, and it still needs a lot of testing, but on my machine the most recent patches bring git status on the Linux kernel down from over 15 minutes to under 8 seconds. That's the same as doing the same thing on Linux, so it's probably more or less optimal speed-wise.

corngood commented 5 years ago

Wow, that's awesome. I'll try to find some time to build master (I assume this is the stuff you pushed recently) and run my benchmarks again.

Thanks.

Atemu commented 5 years ago

That sounds amazing, thank you for your work!

Have you also tested how sequential performance with relatively few files was affected?

corngood commented 5 years ago

@maharmstone I gave it a try. I was able to create a new volume, subvolume, directory, snapshot, but the first time I tried to write something to a file I got a crash:

FYI I built it with VS2017, so this may not be relevant to you.

*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

KERNEL_SECURITY_CHECK_FAILURE (139)
A kernel component has corrupted a critical data structure.  The corruption
could potentially allow a malicious user to gain control of this machine.
Arguments:
Arg1: 0000000000000003, A LIST_ENTRY has been corrupted (i.e. double remove).
Arg2: ffff94855b0ed940, Address of the trap frame for the exception that caused the bugcheck
Arg3: ffff94855b0ed898, Address of the exception record for the exception that caused the bugcheck
Arg4: 0000000000000000, Reserved

Debugging Details:
------------------

KEY_VALUES_STRING: 1

STACKHASH_ANALYSIS: 1

TIMELINE_ANALYSIS: 1

DUMP_CLASS: 1

DUMP_QUALIFIER: 401

BUILD_VERSION_STRING:  17134.1.amd64fre.rs4_release.180410-1804

SYSTEM_MANUFACTURER:  MSI

SYSTEM_PRODUCT_NAME:  MS-7A33

SYSTEM_SKU:  To be filled by O.E.M.

SYSTEM_VERSION:  3.0

BIOS_VENDOR:  American Megatrends Inc.

BIOS_VERSION:  5.50

BIOS_DATE:  09/20/2017

BASEBOARD_MANUFACTURER:  MSI

BASEBOARD_PRODUCT:  X370 GAMING PLUS (MS-7A33)

BASEBOARD_VERSION:  3.0

DUMP_TYPE:  1

BUGCHECK_P1: 3

BUGCHECK_P2: ffff94855b0ed940

BUGCHECK_P3: ffff94855b0ed898

BUGCHECK_P4: 0

TRAP_FRAME:  ffff94855b0ed940 -- (.trap 0xffff94855b0ed940)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=ffffbf07850ba2e8 rbx=0000000000000000 rcx=0000000000000003
rdx=ffffbf0784f89220 rsi=0000000000000000 rdi=0000000000000000
rip=fffff805b5500194 rsp=ffff94855b0edad0 rbp=0000000000000080
 r8=ffffbf0784cda7a8  r9=000000000000001b r10=ffffe303c99024c0
r11=ffffbf0784fa5180 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl nz ac pe cy
btrfs!RtlFailFast+0x5 [inlined in btrfs!reap_fcb+0x374]:
fffff805`b5500194 cd29            int     29h
Resetting default scope

EXCEPTION_RECORD:  ffff94855b0ed898 -- (.exr 0xffff94855b0ed898)
ExceptionAddress: fffff805b5500194 (btrfs!RtlFailFast+0x0000000000000005)
   ExceptionCode: c0000409 (Security check failure or stack buffer overrun)
  ExceptionFlags: 00000001
NumberParameters: 1
   Parameter[0]: 0000000000000003
Subcode: 0x3 FAST_FAIL_CORRUPT_LIST_ENTRY

CPU_COUNT: 10

CPU_MHZ: bb8

CPU_VENDOR:  AuthenticAMD

CPU_FAMILY: 17

CPU_MODEL: 1

CPU_STEPPING: 1

BLACKBOXBSD: 1 (!blackboxbsd)

BLACKBOXPNP: 1 (!blackboxpnp)

BUGCHECK_STR:  0x139

PROCESS_NAME:  System

CURRENT_IRQL:  0

DEFAULT_BUCKET_ID:  FAIL_FAST_CORRUPT_LIST_ENTRY

ERROR_CODE: (NTSTATUS) 0xc0000409 - The system detected an overrun of a stack-based buffer in this application. This overrun could potentially allow a malicious user to gain control of this application.

EXCEPTION_CODE: (NTSTATUS) 0xc0000409 - The system detected an overrun of a stack-based buffer in this application. This overrun could potentially allow a malicious user to gain control of this application.

EXCEPTION_CODE_STR:  c0000409

EXCEPTION_PARAMETER1:  0000000000000003

ANALYSIS_SESSION_HOST:  DAVIDM

ANALYSIS_SESSION_TIME:  04-15-2019 10:44:02.0736

ANALYSIS_VERSION: 10.0.17763.1 amd64fre

LAST_CONTROL_TRANSFER:  from fffff80354a63f69 to fffff80354a53380

STACK_TEXT:  
ffff9485`5b0ed618 fffff803`54a63f69 : 00000000`00000139 00000000`00000003 ffff9485`5b0ed940 ffff9485`5b0ed898 : nt!KeBugCheckEx
ffff9485`5b0ed620 fffff803`54a64310 : 00000000`00000000 00000000`00000000 ffff3ef1`fccc8d42 00000000`00000000 : nt!KiBugCheckDispatch+0x69
ffff9485`5b0ed760 fffff803`54a6291f : 00000000`00000000 ffffe303`c9799840 ffff9680`4e11c140 fffff803`54b942d4 : nt!KiFastFailDispatch+0xd0
ffff9485`5b0ed940 fffff805`b5500194 : ffffbf07`84fa5358 ffffbf07`84fa53b0 00000000`00000080 ffffbf07`84fa53c0 : nt!KiRaiseSecurityCheckFailure+0x2df
ffff9485`5b0edad0 fffff805`b5500225 : ffffbf07`850ba2f8 00000000`00000000 ffffbf07`845f45a8 fffff803`548dc56a : btrfs!reap_fcb+0x374 [c:\users\davidm\src\btrfs\src\btrfs.c @ 1539] 
ffff9485`5b0edb00 fffff805`b55595a4 : ffffe303`c9901498 00000000`00000080 00000000`00000000 ffffe303`c9901ed8 : btrfs!reap_fcbs+0x35 [c:\users\davidm\src\btrfs\src\btrfs.c @ 1566] 
ffff9485`5b0edb30 fffff805`b552231c : ffffe303`c9900180 ffffffff`00000000 ffffe303`c9901498 00000000`00000000 : btrfs!free_trees+0x294 [c:\users\davidm\src\btrfs\src\treefuncs.c @ 830] 
ffff9485`5b0edbb0 fffff803`5491d4f7 : 00000000`00000080 fffff805`b55221e0 ffffe303`c9799700 ffffe303`c9900030 : btrfs!flush_thread+0x13c [c:\users\davidm\src\btrfs\src\flushthread.c @ 7417] 
ffff9485`5b0edc10 fffff803`54a5a7f6 : ffff9680`4eb48180 ffffe303`c9799700 fffff803`5491d4b0 00000000`00000000 : nt!PspSystemThreadStartup+0x47
ffff9485`5b0edc60 00000000`00000000 : ffff9485`5b0ee000 ffff9485`5b0e8000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16

THREAD_SHA1_HASH_MOD_FUNC:  818543b709ef8d2737be9d90f2fa14bbad26c6c9

THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  6344d02533c08a46dd2133fb140b0c34a3378718

THREAD_SHA1_HASH_MOD:  ed006da84d64fd1acd771fbd8008f29200185557

FOLLOWUP_IP: 
btrfs!reap_fcb+374 [c:\users\davidm\src\btrfs\src\btrfs.c @ 1539]
fffff805`b5500194 cd29            int     29h

FAULT_INSTR_CODE:  8b4829cd

FAULTING_SOURCE_LINE:  c:\users\davidm\src\btrfs\src\btrfs.c

FAULTING_SOURCE_FILE:  c:\users\davidm\src\btrfs\src\btrfs.c

FAULTING_SOURCE_LINE_NUMBER:  1539

FAULTING_SOURCE_CODE:  
 11060:     )
 11061: 
 11062: {
 11063: 
>11064:     __fastfail(Code);
 11065: }
 11066: 
 11067: #endif // _MSC_VER
 11068: 
 11069: //

SYMBOL_STACK_INDEX:  4

SYMBOL_NAME:  btrfs!reap_fcb+374

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: btrfs

IMAGE_NAME:  btrfs.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  5cb480a6

STACK_COMMAND:  .thread ; .cxr ; kb

BUCKET_ID_FUNC_OFFSET:  374

FAILURE_BUCKET_ID:  0x139_3_CORRUPT_LIST_ENTRY_btrfs!reap_fcb

BUCKET_ID:  0x139_3_CORRUPT_LIST_ENTRY_btrfs!reap_fcb

PRIMARY_PROBLEM_CLASS:  0x139_3_CORRUPT_LIST_ENTRY_btrfs!reap_fcb

TARGET_TIME:  2019-04-15T13:37:01.000Z

OSBUILD:  17134

OSSERVICEPACK:  0

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK:  272

PRODUCT_TYPE:  1

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

OSEDITION:  Windows 10 WinNt TerminalServer SingleUserTS

OS_LOCALE:  

USER_LCID:  0

OSBUILD_TIMESTAMP:  2018-09-14 23:18:09

BUILDDATESTAMP_STR:  180410-1804

BUILDLAB_STR:  rs4_release

BUILDOSVER_STR:  10.0.17134.1.amd64fre.rs4_release.180410-1804

ANALYSIS_SESSION_ELAPSED_TIME:  818

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:0x139_3_corrupt_list_entry_btrfs!reap_fcb

FAILURE_ID_HASH:  {e9082920-3c50-4ffc-06c2-baea55bd1ea1}

Followup:     MachineOwner
---------

Zero3K commented 5 years ago

That's not good. I hope that it can be fixed.

Chaython commented 4 years ago

NTFS

CrystalDiskMark 6.0.2 x64 (C) 2007-2018 hiyohiyo Crystal Dew World : https://crystalmark.info/

MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 2815.107 MB/s Sequential Write (Q= 32,T= 1) : 1615.286 MB/s Random Read 4KiB (Q= 8,T= 8) : 1231.131 MB/s [ 300569.1 IOPS] Random Write 4KiB (Q= 8,T= 8) : 970.043 MB/s [ 236826.9 IOPS] Random Read 4KiB (Q= 32,T= 1) : 439.156 MB/s [ 107215.8 IOPS] Random Write 4KiB (Q= 32,T= 1) : 225.975 MB/s [ 55169.7 IOPS] Random Read 4KiB (Q= 1,T= 1) : 44.427 MB/s [ 10846.4 IOPS] Random Write 4KiB (Q= 1,T= 1) : 110.803 MB/s [ 27051.5 IOPS]

Test : 500 MiB [C: 58.7% (55.4/94.4 GiB)] (x1) [Interval=5 sec] Date : 2020/03/28 23:28:04 OS : Windows 10 Enterprise [10.0 Build 18363] (x64)

BTRFS

CrystalDiskMark 6.0.2 x64 (C) 2007-2018 hiyohiyo Crystal Dew World : https://crystalmark.info/

MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 2302.475 MB/s Sequential Write (Q= 32,T= 1) : 130.982 MB/s Random Read 4KiB (Q= 8,T= 8) : 727.899 MB/s [ 177709.7 IOPS] Random Write 4KiB (Q= 8,T= 8) : 1.350 MB/s [ 329.6 IOPS] Random Read 4KiB (Q= 32,T= 1) : 399.801 MB/s [ 97607.7 IOPS] Random Write 4KiB (Q= 32,T= 1) : 2.295 MB/s [ 560.3 IOPS] Random Read 4KiB (Q= 1,T= 1) : 40.229 MB/s [ 9821.5 IOPS] Random Write 4KiB (Q= 1,T= 1) : 1.204 MB/s [ 293.9 IOPS]

Test : 500 MiB [F: 26.5% (206.7/781.3 GiB)] (x1) [Interval=5 sec] Date : 2020/03/28 23:30:32 OS : Windows 10 Enterprise [10.0 Build 18363] (x64)

Same drive two different partitions, may be unfair. But 30-99% slower? There are much more serious issues than speed however. This is with default settings.

XPG SX8200 1tb i76700k 2900mhz ram

maharmstone commented 4 years ago

Thanks for this. I've not done any rigorous testing yet, but like you I found that write speeds on Btrfs were worse than read speeds, whereas on NTFS they are about the same.

The difference in your read speeds is probably due to the checksumming that Btrfs does.

maharmstone commented 4 years ago

Having had a look at what it's doing, I'm not sure that the figures for random writes tell you anything meaningful at all - certainly it's not indicative of a bug that needs to be fixed. It's slow because it has to traverse a list of thousands of extents, whereas NTFS does everything in place.

The only time you're likely to encounter random writes in practice is with databases and VM images, and the official advice for these has always been to make sure they're marked nocow.

Chaython commented 4 years ago

Different software, different run, much faster but innacurate[?] disks wouldn't let me run a write test while the drive was active...a Fio script I was recommended gave errors creating a tmp dir to benchmark... But this read speed is much more accurate to the device's specifications than in windows. Screenshot from 2020-03-29 13-33-48

alucryd commented 3 years ago

Adding to the performance testimonies, here are some crystaldiskmark results on a 960 EVO NVMe with 2 separate partitions.

NTFS

------------------------------------------------------------------------------
CrystalDiskMark 8.0.1 x64 (C) 2007-2021 hiyohiyo
                                  Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
  SEQ    1MiB (Q=  8, T= 1):  2375.063 MB/s [   2265.0 IOPS] <  3447.45 us>
  SEQ    1MiB (Q=  1, T= 1):  1388.444 MB/s [   1324.1 IOPS] <   754.82 us>
  RND    4KiB (Q= 32, T= 1):   620.412 MB/s [ 151467.8 IOPS] <   204.52 us>
  RND    4KiB (Q=  1, T= 1):    46.323 MB/s [  11309.3 IOPS] <    88.29 us>

[Write]
  SEQ    1MiB (Q=  8, T= 1):  1510.362 MB/s [   1440.4 IOPS] <  5539.39 us>
  SEQ    1MiB (Q=  1, T= 1):  1507.275 MB/s [   1437.4 IOPS] <   695.16 us>
  RND    4KiB (Q= 32, T= 1):   525.530 MB/s [ 128303.2 IOPS] <   242.68 us>
  RND    4KiB (Q=  1, T= 1):   203.812 MB/s [  49758.8 IOPS] <    20.00 us>

Profile: Default
   Test: 1 GiB (x5) [C: 67% (86/127GiB)]
   Mode: [Admin]
   Time: Measure 5 sec / Interval 5 sec 
   Date: 2021/02/03 19:26:04
     OS: Windows 10  [10.0 Build 19041] (x64)

BTRFS (xxHash, no compression)

------------------------------------------------------------------------------
CrystalDiskMark 8.0.1 x64 (C) 2007-2021 hiyohiyo
                                  Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
  SEQ    1MiB (Q=  8, T= 1):  2030.902 MB/s [   1936.8 IOPS] <  4120.17 us>
  SEQ    1MiB (Q=  1, T= 1):   852.324 MB/s [    812.8 IOPS] <  1229.54 us>
  RND    4KiB (Q= 32, T= 1):   772.471 MB/s [ 188591.6 IOPS] <   164.25 us>
  RND    4KiB (Q=  1, T= 1):    41.672 MB/s [  10173.8 IOPS] <    98.17 us>

[Write]
  SEQ    1MiB (Q=  8, T= 1):   697.329 MB/s [    665.0 IOPS] < 11957.07 us>
  SEQ    1MiB (Q=  1, T= 1):   341.835 MB/s [    326.0 IOPS] <  3064.89 us>
  RND    4KiB (Q= 32, T= 1):     4.059 MB/s [    991.0 IOPS] < 32151.50 us>
  RND    4KiB (Q=  1, T= 1):     5.613 MB/s [   1370.4 IOPS] <   729.17 us>

Profile: Default
   Test: 1 GiB (x5) [E: 64% (68/105GiB)]
   Mode: [Admin]
   Time: Measure 5 sec / Interval 5 sec 
   Date: 2021/02/03 19:32:40
     OS: Windows 10  [10.0 Build 19041] (x64)

BTRFS (xxHash, zstd:3 compression)

------------------------------------------------------------------------------
CrystalDiskMark 8.0.1 x64 (C) 2007-2021 hiyohiyo
                                  Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
  SEQ    1MiB (Q=  8, T= 1):  2024.762 MB/s [   1931.0 IOPS] <  4133.89 us>
  SEQ    1MiB (Q=  1, T= 1):   992.458 MB/s [    946.5 IOPS] <  1056.03 us>
  RND    4KiB (Q= 32, T= 1):   758.915 MB/s [ 185282.0 IOPS] <   167.19 us>
  RND    4KiB (Q=  1, T= 1):    41.257 MB/s [  10072.5 IOPS] <    99.15 us>

[Write]
  SEQ    1MiB (Q=  8, T= 1):   230.245 MB/s [    219.6 IOPS] < 36239.66 us>
  SEQ    1MiB (Q=  1, T= 1):   338.777 MB/s [    323.1 IOPS] <  3093.65 us>
  RND    4KiB (Q= 32, T= 1):     5.213 MB/s [   1272.7 IOPS] < 25038.44 us>
  RND    4KiB (Q=  1, T= 1):     5.735 MB/s [   1400.1 IOPS] <   713.74 us>

Profile: Default
   Test: 1 GiB (x5) [L: 64% (68/105GiB)]
   Mode: [Admin]
   Time: Measure 5 sec / Interval 5 sec 
   Date: 2021/02/03 19:46:45
     OS: Windows 10  [10.0 Build 19041] (x64)

Read speeds are still lower overall, but very usable, write speeds however are somewhat of a letdown, and compression totally destroys performance, but that was to be expected.

All in all those are trade-offs that I'm willing to make since my workload of choice will be gaming so this shouldn't matter much. My steam library sits on a ntfs volume striped across 2 SATA SSDs, mounting this on linux requires libldm and I'd like to get rid of it, I'll give raid0 btrfs a try and see how that goes.

lapineige commented 3 years ago

Does that benchmark take into account the fact that compressed data results in more "raw" (uncompressed) data wrote/read ?

mirh commented 3 years ago

Paragon's ntfs driver doesn't require any extra dependencies, on top of possibly being even faster than ext4.

alucryd commented 3 years ago

Does that benchmark take into account the fact that compressed data results in more "raw" (uncompressed) data wrote/read ?

That is a good question I'm afraid I can't answer. Will see how real world workloads fare, that's what matters after all.

Paragon's ntfs driver doesn't require any extra dependencies, on top of possibly being even faster than ext4.

The whole point is to get rid of NTFS and use a superior filesystem (checksum, native raid, better transparent compression). Speed is hardly the deciding factor for me, and I'd rather have windows lag behind than having linux being limited by windows. Plus the paragon driver is still relatively new and hasn't landed in the kernel yet, so there's that.

lapineige commented 3 years ago

That is a good question I'm afraid I can't answer. Will see how real world workloads fare, that's what matters after all.

Maybe if you can measure the time to open/write a very big file (set) (be cautious about caching and so on, it can bias the result, a lot) you can have an estimation 🤔

I'm asking because I'm surprised that compression really destroy the performances, on my setup when I compared btrfs with and without compression (on linux) I saw no impact of compression on read and write speed (but thanks to the compression the "real" speed was better).

maharmstone commented 3 years ago

Thanks @alucryd. It's interesting that sequential writes are so much worse than NTFS - I'll have to look into that.

@lapineige:

Does that benchmark take into account the fact that compressed data results in more "raw" (uncompressed) data wrote/read ?

It'll be measuring the uncompressed size, userspace has no idea that compression is even going on.

@alucryd:

compression totally destroys performance, but that was to be expected.

You've got a really good drive - if you had a mechanical HDD, it's possible that compressed I/O would be faster than uncompressed, because there's less data to transfer.

I'd also have to see what Crystal DiskMark is doing, but I suspect it's reading and writing random data, which is by definition incompressible. It's possible that it's only slower because the driver is spending its time running the data through the compression algorithm, finding it's larger, and discarding it...

lapineige commented 3 years ago

It'll be measuring the uncompressed size, userspace has no idea that compression is even going on.

Yes, but you can then check what's the compressed size. Then you know the speed to read/write a compressed file, and what's the "real" speed that you get for the whole uncompressed data. This way I suppose you can know if there is any compression benefit on read/write speed.

if you had a mechanical HDD, it's possible that compressed I/O would be faster than uncompressed, because there's less data to transfer.

I did not compare that to NTFS (only to ext4), but in my case that was the result, +50% read speed and +30% write speed and a HDD. (I can provide the details if you want) That should be the same for SATA SSD, has most of them are capped to ~550MB/s because of SATA limitations, then if your CPU can compress at least as much data/s, you write 550MB compressed data which equals in more uncompressed data. If a benchmark (like CrystalMark) does only measure uncompressed data read/write speed, it won't be able to show that result.

but I suspect it's reading and writing random data, which is by definition incompressible

That's a very good point. And also very unrepresentative from common real life usage, where big files/folders (the ones that matters for a significant real life difference) are much easier to compress.

It's possible that it's only slower because the driver is spending its time running the data through the compression algorithm, finding it's larger, and discarding it...

Not that if you're using zstd (and maybe other algorithms) it has a built-in heuristic that will not try to compress data that can't be compressed (if the first chunk is not compressible), so the performance cost should not be a big deal, except maybe on small CPUs… :thinking: (In my case, ryzen 3600, the performance cost is not noticeable on a 15GB write benchmark, less than 1%)

alucryd commented 3 years ago

Another piece of interesting data, this is on a raid 0 made of 2 SATA SSDs, with xxhash checksum and zstd:3 compression:

------------------------------------------------------------------------------
CrystalDiskMark 8.0.1 x64 (C) 2007-2021 hiyohiyo
                                  Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
  SEQ    1MiB (Q=  8, T= 1):  1102.484 MB/s [   1051.4 IOPS] <  7592.76 us>
  SEQ    1MiB (Q=  1, T= 1):   875.420 MB/s [    834.9 IOPS] <  1197.39 us>
  RND    4KiB (Q= 32, T= 1):   375.557 MB/s [  91688.7 IOPS] <   337.80 us>
  RND    4KiB (Q=  1, T= 1):    38.893 MB/s [   9495.4 IOPS] <   105.21 us>

[Write]
  SEQ    1MiB (Q=  8, T= 1):   753.214 MB/s [    718.3 IOPS] < 11086.90 us>
  SEQ    1MiB (Q=  1, T= 1):   754.814 MB/s [    719.8 IOPS] <  1388.28 us>
  RND    4KiB (Q= 32, T= 1):     5.697 MB/s [   1390.9 IOPS] < 22921.66 us>
  RND    4KiB (Q=  1, T= 1):     6.018 MB/s [   1469.2 IOPS] <   680.22 us>

Profile: Default
   Test: 1 GiB (x5) [G: 80% (749/932GiB)]
   Mode: [Admin]
   Time: Measure 5 sec / Interval 5 sec 
   Date: 2021/02/04 18:05:34
     OS: Windows 10  [10.0 Build 19041] (x64)

We're maxing out the theoretical read speed of the SSDs in the first sequential read tests, and sequential write speeds actually outperform the NVMe SSD. FWIW, the NVMe SSD has a theoretical sequential write speed of 1900MB/s, while each SATA SSD is at 520MB/s.

These are great results, and I'm happy to report that gaming on BTRFS on Windows works like a charm. No hiccup whatsoever in a 1h session of AC Valhalla maxed out at 4K. Great job on the driver!

theChaosCoder commented 3 years ago

You could disable COW for you gaming ssd / gaming folder. This will make writes much faster.

lapineige commented 3 years ago

What CPU do you use ? I'm really surprised of the performance for the NVMe SSD, considering that mine had got a +30% boost thanks to the compression (compared to btrfs without compression, on linux), and it has similar performances (1.5-2GB/s read and write speed)

You even have more than twice the NVMe speed with your 2 SSD… I would have supposed that raid 0 would provide a maximum of 2 times the speed…

alucryd commented 3 years ago

You could disable COW for you gaming ssd / gaming folder. This will make writes much faster.

Indeed, although writes will not generally be made when speed is critical, cow doesn't make much sense here and disabling it should help a lot.

What CPU do you use ? I'm really surprised of the performance for the NVMe SSD, considering that mine had got a +30% boost thanks to the compression (compared to btrfs without compression, on linux), and it has similar performances (1.5-2GB/s read and write speed)

You even have more than twice the NVMe speed with your 2 SSD… I would have supposed that raid 0 would provide a maximum of 2 times the speed…

Ryzen 5800x. I'd have assumed the same, also chose xxhash for checksuming specifically after running the btrfs-progs hashpeed bench because it was the fastest of the bunch and since dedup is not there yet I'm not really afraid of collisions. I have yet to run any comparison bench on Linux though, will probably do that tomorrow.

lapineige commented 3 years ago

Ryzen 5800x.

Ok so there is noooo reason that your CPU could be slowing you down (ryzen 3600 here, no sign of CPU bottleneck at all). I don't understand these results…

Could you test coping a few (big) files maybe ? Personally I adapted that script (for linux) https://www.reddit.com/r/btrfs/comments/hyra46/benchmark_of_btrfs_decompression/, that could be a good point of comparison.

Additional note: I had better speed result with zstd compression level 1, and no compression ratio difference. I've found similar results on the internet.

TheMadHau5 commented 3 years ago

@alucryd:

Plus the paragon driver is still relatively new and hasn't landed in the kernel yet, so there's that.

Just wanted to point out, but the Paragon driver appears to be reasonably stable, and also commercially supported. The only reason it it not in the kernel is because it is paid, and therefore proprietary (and so incompatible with GPL2).

Atemu commented 3 years ago

Paragon is actually in the process of contributing their NTFS driver to the kernel:

https://lore.kernel.org/lkml/20201225135119.3666763-1-almaz.alexandrovich@paragon-software.com/

TheMadHau5 commented 3 years ago

That takes me by surprise, although I am glad to see it.

alucryd commented 3 years ago

Paragon is actually in the process of contributing their NTFS driver to the kernel:

https://lore.kernel.org/lkml/20201225135119.3666763-1-almaz.alexandrovich@paragon-software.com/

That's exactly what I meant by "has not landed yet", but it's not relevant to the discussion here. This thread is about performance of NTFS vs BTRFS on Windows, not Linux. BTRFS performance on Linux is also a valid comparison point, but NTFS performance on Linux hardly matters. I really don't see how a fast kernel driver will make a very limited filesystem relevant again, if it has ever been on Linux anyway. There are and have been so many better options, and the real big deal here is that one of them is now available on Windows. So can we please stay on topic?

maharmstone commented 3 years ago

Bear in mind that AFAIK the new NTFS driver doesn't do anything with POSIX metadata, so it's probably not much use as a general-purpose Linux filesystem.

lss4 commented 3 years ago

Ummm... I'm not sure if it's possible to create indexes in particular folders so programs can list the files there quickly...

One of the game seems slow to list the save games when the save game folder's contents become too many. With NTFS, it seems the list could be memorized as only the first time when the folder is accessed (per program run) would take time, but with Btrfs, every time when the folder is accessed would take time... I don't know how the game works to list the save game folder, but this seems to imply something is not working efficiently. (NOTE: This is on a Samsung 980 Pro SSD, though the files originally resided on a Sandisk SATA one, but I'm not sure if this could be relevant...)

EDIT: Another question. I don't know what parameter WinBtrfs uses by default as I just realized my Linux distro (Manjaro) is using zstd compression for Btrfs volumes according to my fstab... although when I looked at the files' properties from Windows the compression checkbox is not checked...

Chaython commented 3 years ago

The compression checkbox doesn't always work... Please refer to the readme [ https://github.com/maharmstone/btrfs#mount-options ] - regedit: HKLM\SYSTEM\CurrentControlSet\Services\btrfs in relation to compression... For me on a SX8200 [NVME SSD] every directory, BTRFS or NTFS load immediately, NTFS is using Compact and in BTRFS I'm using ZSTD. If I use a program like "files" https://github.com/files-community/Files it takes a much longer time than the native file explorer. Though it too, is regardless of the FS in use. But it sure is a sexy GUI. Anyways with the newest revision, BTRFS is almost as fast in Windows as Linux. BTRFS is still a bit off from NTFS in Windows, but in some runs, it actually beats in 4k random reads. Some performance degradation is to be expected, not being a native FS and having to work around MS BS. Top is BTRFS, bottom is NTFS, IDK if 4k measuring is broken, but the completion time was still fast, so I assume its measuring is just bugged.

Atemu commented 3 years ago

With NTFS, it seems the list could be memorized as only the first time when the folder is accessed (per program run) would take time, but with Btrfs, every time when the folder is accessed would take time...

NTFS being faster the second time is due to that information being cached in RAM by the OS.
I'm not sure to what extent WinBtrfs does that yet but I'd open a separate issue about that because it definitely should be doing the same and it's a different aspect of performance than what is being discussed here.

A simple reproducer would also be helpful. How many files/dir are we talking?

I don't know what parameter WinBtrfs uses by default as I just realized my Linux distro (Manjaro) is using zstd compression for Btrfs volumes according to my fstab... although when I looked at the files' properties from Windows the compression checkbox is not checked...

Compression is a per-mount option.

If it's set, compression is enabled on all files for that mount regardless of the separate per-file compression setting.

WinBtrfs' mount options can be tweaked in the registry, see the README.

lapineige commented 3 years ago

Compression is a per-mount option.

And that also means that WinBtrfs could mount it without compression, and Linux with compression. Both works fine (you can even unmount and remount with(out) compression). It's just that file written from windows won't be compressed.

lss4 commented 3 years ago

NTFS being faster the second time is due to that information being cached in RAM by the OS. I'm not sure to what extent WinBtrfs does that yet but I'd open a separate issue about that because it definitely should be doing the same and it's a different aspect of performance than what is being discussed here.

A simple reproducer would also be helpful. How many files/dir are we talking?

It's a mod of an old game, and it's rather difficult that a lot of save games are needed while playing... At the time of writing my last comment, I had about 1490 files in the Saved Games folder, each contains about 4-5MB of information, with a total size of 6.9GB.

It seems the game needs to list all the files and read a little bit of the information from each file to create the list. With NTFS it takes less time, and the information bits can be cached so it won't take as much time when entering the Save/Load screen the rest of the times for this program run. But with Btrfs, it appears the game needs to read those information each time the Save/Load screen is entered, and the time needed at that point was at least 30 seconds per listing.

To work around I've moved the old save games (no longer relevant to my current progress) to a different folder. By keeping the files count low I could keep the game's Save/Load screen responsive.

As you said, it's possible that the OS (or maybe just the game) might not be able to conduct the caching on Btrfs (maybe some configuration needed) the way it could do on NTFS partitions...

maharmstone commented 3 years ago

@lss4 - What game and what mod?

lss4 commented 3 years ago

@lss4 - What game and what mod?

If you're interested, I was talking about Mental Omega, a RA2:YR mod that's still active and very interesting.

It's just that when saving and loading games the issue happens, but otherwise it works well. Actually the modding engine it uses (Ares) did not support saving and loading until some recent versions so it's possible these functions work but not quite optimal.

At one time I had much more save games (while on a NTFS partition) and noticed it took a while to open the Saved Games folder for the first time, but subsequent saves/loads don't take time so I didn't really bother as it's a minor annoyance at this point. It's just when using Btrfs did I notice that it's taking time reading the Saved Games folder every time and this starts to become a problem.

Just checked, the mod's launcher doesn't appear to have the issue, so the files were probably being read in a different way there. It's just the ingame save/load function has the slow loading issue.

Guess I'll take a look and see if there are any other system options helpful about this.

maharmstone commented 3 years ago

Thanks. When I get some time, I'll try it out and try to figure out what it's doing.

maharmstone / btrfs