Can't copy from NTFS to BTRFS in Windows

[kenny-w1]

Hello, I tried copying World of Warcraft 3.3.5a (16-17GB) from my NTFS windows drive to BTRFS(non-filesystem, backup storage harddrive), going from my primary harddrive (SSD) to my secondary harddrive (HDD), but it froze up & when I tried exiting it it ended up messing up my computer really bad making the explorer not respond, signing out & back in did not fix the problem but restarting did.

I seem to be able to copy from BTRFS to NTFS on Windows, but not from NTFS to BTRFS on Windows. Really like to see that someone is developing a BTRFS driver for Windows, makes things far more convenient, hope you guys can fix it up so copying stuff won't mess things up. Thank you.

If I have missed any relevant information please let me know & I will do what I can to provide you with what you need.

[maharmstone]

What you're describing is a deadlock, and sounds very much like a bug I quashed the other day. Can you please try the version I've just released, see if that improves things for you?

[gileshuang]

I just upgraded to 1.0.2, and this bug still hapend. I copy a directory in NTFS which contain a game (30GB). While about 4.8GB copied, the Explorer frozen. Source: NTFS on SSD. Target: btrfs on HDD.

[gileshuang]

This is since the same issue with #58

[RobertBoganKang]

I cannot copy large files too. It stuck forever, and I force my system to restart by pressing power button. However, I have a very good solution for this. I use fastcopy with no problem at all. However, the performance is slower than NTFS in windows, thus I still use NTFS for storage purpose.

[NatoBoram]

Maybe this can help you? https://www.reddit.com/r/btrfs/comments/8ydu47

Context

Driver is WinBtrfs v1.0.2. https://github.com/maharmstone/btrfs

I used Ubuntu 18.04 to access the rsync command because robocopy is shit. https://www.microsoft.com/store/productId/9N9TNGVNDL3Q

I can actually launch League of Legends and play it from the Btrfs drive!

The process was started with robocopy but it crashed my system every time I used it. I decided to try rsync. The command itself crashed maybe about twice, but after restarting it and being patient, it completed the operation! Some files couldn't by synced like System Volume Information, so it returned some errors a few times. Also, Windows permissions are shit.

robocopy

@ECHO OFF
REM      /ZB :: use restartable mode; if access denied use Backup mode.
REM       /J :: copy using unbuffered I/O (recommended for large files).
REM /COPYALL :: COPY ALL file info (equivalent to /COPY:DATSOU).
REM     /MIR :: MIRror a directory tree (equivalent to /E plus /PURGE).
REM  /MT[:n] :: Do multi-threaded copies with n threads (default 8). n must be at least 1 and not greater than 128. This option is incompatible with the /IPG and /EFSRAW options. Redirect output using /LOG option for better performance.
REM     /R:n :: number of Retries on failed copies: default 1 million.
REM     /W:n :: Wait time between retries: default is 30 seconds.
REM     /ETA :: show Estimated Time of Arrival of copied files.
ROBOCOPY D: E: /ZB /J /COPYALL /MIR /MT /R:5 /W:5 /ETA
PAUSE

rsync

# -c, --checksum              skip based on checksum, not mod-time & size
# -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
# -r, --recursive             recurse into directories
# -u, --update                skip files that are newer on the receiver
# -p, --perms                 preserve permissions
#     --delete-during         receiver deletes during the transfer
#     --force                 force deletion of directories even if not empty
#     --stats                 give some file-transfer stats
# -h, --human-readable        output numbers in a human-readable format
#     --progress              show progress during transfer

rsync -caruph --delete-during --force --stats --progress /mnt/d/ /mnt/e/

--stats

Number of files: 559,658 (reg: 490,317, dir: 69,328, link: 13)
Number of created files: 191,860 (reg: 179,877, dir: 11,983)
Number of deleted files: 0
Number of regular files transferred: 180,075
Total file size: 283.88G bytes
Total transferred file size: 93.10G bytes
Literal data: 93.10G bytes
Matched data: 0 bytes
File list size: 7.60M
File list generation time: 0.058 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 93.16G
Total bytes received: 3.62M

sent 93.16G bytes  received 3.62M bytes  4.17M bytes/sec
total size is 283.88G  speedup is 3.05

Keep in mind that I didn't transfer everything at once because both robocopy and rsync are unstable and crashed more than once. But, in the end, everything is here!

Conclusion

It works!

• NTFS uses 117% the space Btrfs uses.
• Btrfs uses 85% the space NTFS uses.

Later, I reinstalled Windows and formatted my whole HDD to Btrfs. I also installed absolutely everything on it, leaving only Windows on my SSD. I actually play League of Legends from the Btrfs drive!

I think the problem is Microsoft. If I use their tools like robocopy or the Explorer, the computer freezes and I have to reboot using the power button. However, if I use any tool that's not from Microsoft (in this case, rsync), the copy is done properly.

[maharmstone]

I agree with both of Atemu12's comments in that thread - I'm very sceptical that something hasn't gone wrong there. Inline extents might account for it, but I very much doubt you have 40GB worth of tiny files there. Are you certain you don't have any sort of compression turned on?

If robocopy freezes on Btrfs but not NTFS, that's very much an issue with the driver...

[gileshuang]

1. I use Microsoft Explorer, not robocopy/rsync, and this bug happens.
2. I do not enable any compression on btrfs volume.
3. I copy a fold from NTFS to btrfs volume, which about 30GB with a game. I am very sure that btrfs frozen, not NTFS. While it frozen, I can not access the btrfs volume from Explorer, but can access NTFS volume.
4. I also mount this btrfs volume on Linux, without any compression. On Linux, this btrfs volume do not any issue.

[gileshuang]

By the way, I am using version 1.0.2, not the git version. With Windows 10 1803(x64)

[maharmstone]

I've not been able to reproduce this, and I've copied gigabytes and gigabytes. Is your filesystem clean - what's the output of btrfs check?

[gileshuang]

I think the filesystem is clean. There is the output of btrfs check

[root@Xeon-Linux ~] <2018-11-18 20:53:29>
# btrfs check /dev/sdc2
Opening filesystem to check...
Checking filesystem on /dev/sdc2
UUID: e09c0c2c-2586-4c6c-bff5-4a6bcd08e1b5
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 37507375104 bytes used, no error found
total csum bytes: 36486568
total tree bytes: 134643712
total fs tree bytes: 90226688
total extent tree bytes: 3309568
btree space waste bytes: 20392402
file data blocks allocated: 37446782976
 referenced 37455122432

There is the subvolumes of the btrfs devices

[root@Xeon-Linux ~] <2018-11-18 21:03:14>
# btrfs subvolume list /media/Data-Btrfs/
ID 257 gen 9530 top level 5 path Data-Btrfs
ID 503 gen 9514 top level 5 path Build
ID 821 gen 8449 top level 5 path Arch-Snap/Data-Btrfs-2018-05-28
ID 822 gen 8473 top level 5 path Arch-Snap/Data-Btrfs-2018-06-04
ID 823 gen 8495 top level 5 path Arch-Snap/Data-Btrfs-2018-06-20
[root@Xeon-Linux ~] <2018-11-18 21:03:27>
# btrfs subvolume show /media/Data-Btrfs/
Data-Btrfs
    Name:           Data-Btrfs
    UUID:           5ccdfdca-2d47-d944-9267-ff3fd335c4c3
    Parent UUID:        -
    Received UUID:      -
    Creation time:      2017-11-06 00:38:28 +0800
    Subvolume ID:       257
    Generation:         9530
    Gen at creation:    8
    Parent ID:      5
    Top level ID:       5
    Flags:          -
    Snapshot(s):
                Arch-Snap/Data-Btrfs-2018-05-28
                Arch-Snap/Data-Btrfs-2018-06-04
                Arch-Snap/Data-Btrfs-2018-06-20

The Data-Btrfs is the subvolume which mounted on Windows. The hibernate of the OS(Windows and Linux) is disabled, and swap is disabled on Linux too.

After next version released, I will be happy to test it again.

[maharmstone]

Thank you. I think I've sorted this now - it was actually a couple of different issues, all of which only manifested when copying large files at high speeds (> 100 MB/s).

https://github.com/maharmstone/btrfs/commit/2c0fcdb5f32b3c6c43664a276d27d33e1babf064 https://github.com/maharmstone/btrfs/commit/86ca3defe355863ee70a96de8cd170cbac784adf

[rox4d]

Strange, It is quite easy reproducible with disk in usb2 enclosure for me (Win7 x64), is there are plans to release a binary build with these fixes any time soon?

[gileshuang]

Since version 1.1 released, I tried copy two BIG folder from NTFS to BTRFS, using Windows Explorer. One of the folder have 40GB data, contained both large files and small files. One have 100GB data, with so many small files. Both succeed, without any freeze.

I will keep using this driver. If there are some interesting new things, I will let you known.

[rox4d]

V1.1 seems to work here (win7-amd64), just copied about 100Gb over esata and about 40Gb over usb2 without locks (v1.0.2 was locked almost immediately in both cases), it is great, thanks!

[maharmstone]

Excellent, thanks for letting me know!

[G-Bigo]

Just copied 15TB to an external BTRFS RAID array with TotalCommander. WinBTRFS v1.1, Windows v1809 x64. Worked fine.

[Zero3K]

I think this issue should be closed since it was fixed.

[maharmstone]

Closing old issues