Corrupt files when copying

[jdenman03]

qBittorrent & operating system versions

MAC OS 12.3.1 qBittorrent 4.4.2 x64

What is the problem?

After downloading a file, there is a size discrepancy upon finishing. I have posted screenshots of the issue. What's odd is that the file will play perfectly fine on the same HDD it is downloaded to, but once I copy the file to an external drive or to my NAS the file becomes corrupt.

HOWEVER, if I copy the file and paste it to the same HDD, the file sizes get corrected and then I can copy the file to my external HDD or to my NAS without any problems.

I initially thought that it was a MAC OS problem, but I upgraded to 12.3.1 and the problem is still there.

Steps to reproduce

No response

Additional context

No response

Log(s) & preferences file(s)

No response

[glassez]

After downloading a file, there is a size discrepancy upon finishing. I have posted screenshots of the issue

The screenshots above are identical, no?

I initially thought that it was a MAC OS problem, but I upgraded to 12.3.1 and the problem is still there.

Are you seriously sure that this update solves all the problems that existed before?

[jdenman03]

My screenshots are NOT identical. Look at the total disk space. Before the copy shows 1.92 and after the copy shows 1.98.

And no it does NOT solve the problem at all. I concluded the sentence out by saying "and the problem is still there."

[glassez]

Before the copy shows 1.92 and after the copy shows 1.98.

But the sizes in bytes are the same... Also I can't see what kind of copying you're talking about. Maybe I didn't notice something again, but both files are in the same folder, and the full file name is not visible.

And no it does NOT solve the problem at all. I concluded the sentence out by saying "and the problem is still there."

Sorry, your sentence sounds for me like "since the problem wasn't solved by macOS upgrade then it wasn't the problem with macOS".

[glassez]

After downloading a file, there is a size discrepancy upon finishing. I have posted screenshots of the issue. What's odd is that the file will play perfectly fine on the same HDD it is downloaded to, but once I copy the file to an external drive or to my NAS the file becomes corrupt.

Maybe some files aren't completely flushed to disk at the time you encountered this issue.

[jdenman03]

Before the copy shows 1.92 and after the copy shows 1.98.

But the sizes in bytes are the same... Also I can't see what kind of copying you're talking about. Maybe I didn't notice something again, but both files are in the same folder, and the full file name is not visible.

And no it does NOT solve the problem at all. I concluded the sentence out by saying "and the problem is still there."

Sorry, your sentence sounds for me like "since the problem wasn't solved by macOS upgrade then it wasn't the problem with macOS".

Byte sizes are correct yes but the total size on the disk is wrong initially. If I copy that file to an external HDD or a NAS, the file becomes corrupt and the movie won't play correctly or be heavily pixelated. If I proceed to copy the file instead to the SAME HDD, the size on the disk then matches up correctly to the byte sizes and then I can copy the files to an external HDD or a NAS without any issues and the file isn't corrupt and my md5 checksum is the same.

I was just stating that the problem wasn't resolved by upgrading to the newest macOS. I don't have another mac to try, so I can't tell. I guess I could downgrade the qBittorrent to see if the problem fixes.

[jdenman03]

After downloading a file, there is a size discrepancy upon finishing. I have posted screenshots of the issue. What's odd is that the file will play perfectly fine on the same HDD it is downloaded to, but once I copy the file to an external drive or to my NAS the file becomes corrupt.

Maybe some files aren't completely flushed to disk at the time you encountered this issue.

It's every single download that is larger than 1GB. Here's what the videos do when I copy them to a external HDD or NAS (without copying them first to the initial HDD).

[glassez]

It's every single download that is larger than 1GB. Here's what the videos do when I copy them to a external HDD or NAS (without copying them first to the initial HDD).

Could you test how it behaves if you close qBittorrent after downloading of some affected file is done? Also how it behaves if you reboot your Mac after downloading is done and before copying to the external HDD or NAS?

[jdenman03]

It's every single download that is larger than 1GB. Here's what the videos do when I copy them to a external HDD or NAS (without copying them first to the initial HDD).

Could you test how it behaves if you close qBittorrent after downloading of some affected file is done? Also how it behaves if you reboot your Mac after downloading is done and before copying to the external HDD or NAS?

Yes I will try that out next and let you know. I have a file that should be finishing in the next 30 minutes and will try that.

[jdenman03]

It's every single download that is larger than 1GB. Here's what the videos do when I copy them to a external HDD or NAS (without copying them first to the initial HDD).

Could you test how it behaves if you close qBittorrent after downloading of some affected file is done? Also how it behaves if you reboot your Mac after downloading is done and before copying to the external HDD or NAS?

So quitting qbittorrent didn't do anything. However after rebooting the machine (without copying to internal HDD first), the file size corrected itself after the reboot. I have attached screenshots for reference. However another file just completed after the reboot and the problem is still there.

So currently the only way to rectify the issue right now is to either a) reboot the machine after downloads or b) copy the file to the HDD first and then copy to the NAS/external HDD.

[ghost]

somewhat duplicate of https://github.com/qbittorrent/qBittorrent/issues/16787

[jdenman03]

somewhat duplicate of #16787

I saw that one, but mine isn’t 10x at all and while his is more, mine is less.

[ghost]

Do you use preallocate disk space?

[jdenman03]

Do you use preallocate disk space?

I edited my previous comment. But no I do not use preallocated space.

[glassez]

However after rebooting the machine (without copying to internal HDD first), the file size corrected itself after the reboot.

I think it is yet another drawback of MMAP based disk IO subsystem of libtorrent 2.0. Most likely the buffers are not explicitly flushed to disk before closing the file, so the OS does not do it immediately. @arvidn, ping.

[jdenman03]

However after rebooting the machine (without copying to internal HDD first), the file size corrected itself after the reboot.

I think it is yet another drawback of MMAP based disk IO subsystem of libtorrent 2.0. Most likely the buffers are not explicitly flushed to disk before closing the file, so the OS does not do it immediately. @arvidn, ping.

This make sense. I am testing out a solution now to see if it fixes it. I will post an update after my next few downloads.

[jdenman03]

Also just a note and it looks like this has already been well discussed here on the forum, if I download directly to my NAS, qBittorrent will freeze and eventually hang up my entire mac causing me to hard reboot the machine.

[jdenman03]

My test solution didn't work. I tried using a separate path for incomplete torrents, but that didn't fix it.

[jdenman03]

just as an update to this post. I reverted back to a really old qbit build and everything works fine. I'll update back when there is a newer build and I will try it.

[kheer1]

just as an update to this post. I reverted back to a really old qbit build and everything works fine. I'll update back when there is a newer build and I will try it.

I'm having the same issue, what old build did you revert to? I'm thinking of doing the same thing.

[jdenman03]

just as an update to this post. I reverted back to a really old qbit build and everything works fine. I'll update back when there is a newer build and I will try it.

I'm having the same issue, what old build did you revert to? I'm thinking of doing the same thing.

Just the old one that was already downloaded on my drive. It's 4.3.8

[arvidn]

I believe this is fundamentally caused by a bug in Mac OS. Writing to a memory mapped file is supposed to end up in the same page cache normal file I/O use as disk cache.

That said, the most recent libtorrent release attempts to hint the OS to be more aggressive in flushing data to disk, which may mitigate this issue.

[jdenman03]

I believe this is fundamentally caused by a bug in Mac OS. Writing to a memory mapped file is supposed to end up in the same page cache normal file I/O use as disk cache.

That said, the most recent libtorrent release attempts to hint the OS to be more aggressive in flushing data to disk, which may mitigate this issue.

while you are probably right and more advanced than me at this, why is that 4.3.8 doesn't have this issue and everything works correctly?

[arvidn]

while you are probably right and more advanced at me at this, why is that 4.3.8 doesn't have this issue and everything works correctly?

I don't believe it's using the new version of libtorrent. the new version of libtorrent uses memory mapped files for disk I/O

[glassez]

while you are probably right and more advanced at me at this, why is that 4.3.8 doesn't have this issue and everything works correctly?

I don't believe it's using the new version of libtorrent.

It uses libtorrent-1.2.

[ArthurGuez]

I'm having roughly the same issue. I download torrents directly onto my external hard drive but I have had a lot of corrupted files for the last 2 weeks or so. I'm not even able to install an older version of qBittorrent. I hope this will be fixed soon :)

[jdenman03]

I'm having roughly the same issue. I download torrents directly onto my external hard drive but I have had a lot of corrupted files for the last 2 weeks or so. I'm not even able to install an older version of qBittorrent. I hope this will be fixed soon :)

How are you not able to install an older version?

FYI, if you first copy the file to the same HDD it was downloaded on, it will flush the cache and it should be fine. That's what I resorted to until I installed 4.3.8

[arvidn]

@ArthurGuez what operating system are you using?

[ArthurGuez]

@jdenman03 I don't know. I've tried countless times but it just crashes everytime I open the app. I've been able to reinstall the latest version but I still have the corrupted files issue described before. I'll try your workaround, thanks.

@arvidn macOS Big Sur 11.6.5 (my Macbook Air is too old for Monterey).

[loekf]

I am Monterrey 12.3.1. I also see the issue that downloaded video files are corrupted. I went back to 4.3.9 and that one is working fine.

[ghost]

The aggressive flushing to disk is enabled by default on Windows only. Maybe it should’ve been enabled for all OS.

Btw @jdenman03 could you test to see what happens when you go to contents tab and right click on the files and click open destination or open the file itself? Does it become corrupted when you copy afterwards?

[mshick]

Same here, MacOS 12.3.1, qBittorrent 4.4.2, and all video files copied to my external HDD are corrupt. I've upgraded and downgraded (to 4.3.9) twice in the past month, curious whether I had some other transient issue causing the corruption, but both times all video files downloaded after the update get corrupted.

In my case I download incomplete files to a very fast, USB-C NVMe SSD and move completed d/ls to a slower spinning disk.

I do not preallocate disk space.

[jdenman03]

The aggressive flushing to disk is enabled by default on Windows only. Maybe it should’ve been enabled for all OS.

Btw @jdenman03 could you test to see what happens when you go to contents tab and right click on the files and click open destination or open the file itself? Does it become corrupted when you copy afterwards?

I can open the original file and it will play just fine, even like you suggested. It’s only when I decide to copy the file to another location will it become corrupt. If I copy the file to the SAME HDD it plays just fine and my MD5 checksum is then correct also. And then I can copy to another HDD without it becoming corrupt.

[ghost]

I mean have you tried copying the file AFTER opening it through content tab?

[blevins0321]

I just discovered this issue on a recently-new install of 4.4.2 on an M1 Mac Studio. My setup is an Incomplete folder on my local SSD, then copying to an external Mac Disk Utility-build Raid array when complete.

Found a possible workaround. If I enable the advanced setting to recheck upon completion, the torrents will go through a couple rounds of checking and partial re-downloading, but will eventually pass hash check. Bit unfortunate to have the data moving around so much, but it seems to work until this issue is actually fixed.

[loekf]

I just discovered this issue on a recently-new install of 4.4.2 on an M1 Mac Studio. My setup is an Incomplete folder on my local SSD, then copying to an external Mac Disk Utility-build Raid array when complete.

Found a possible workaround. If I enable the advanced setting to recheck upon completion, the torrents will go through a couple rounds of checking and partial re-downloading, but will eventually pass hash check. Bit unfortunate to have the data moving around so much, but it seems to work until this issue is actually fixed.

Turning that on indeed fixes having corrupt files after I copy the downloaded files to my NAS.

So I am back to 4.4.2 now with this advanced setting enabled.

I see hardly additional downloading though, it does checking, but perhaps the files are small (under 500 MB) and my connection is fast.

[ArthurGuez]

I just discovered this issue on a recently-new install of 4.4.2 on an M1 Mac Studio. My setup is an Incomplete folder on my local SSD, then copying to an external Mac Disk Utility-build Raid array when complete. Found a possible workaround. If I enable the advanced setting to recheck upon completion, the torrents will go through a couple rounds of checking and partial re-downloading, but will eventually pass hash check. Bit unfortunate to have the data moving around so much, but it seems to work until this issue is actually fixed.

Turning that on indeed fixes having corrupt files after I copy the downloaded files to my NAS.

So I am back to 4.4.2 now with this advanced setting enabled.

I see hardly additional downloading though, it does checking, but perhaps the files are small (under 500 MB) and my connection is fast.

Didn't work for me. I still had some corrupted files in the end

[loekf]

You are correct. At first I thought there was a fix or workaround, but yesterday I got corrupted files again.

So the recheck on completion and also manual forcing a re-check does not solve it.

Back to 4.3.9

[ciegetank]

Can anyone verify if 4.4.3 resolves the corruption issue?

[glassez]

Can anyone verify if 4.4.3 resolves the corruption issue?

Is it mentioned as fixed in 4.4.3 release info?

[jdenman03]

Can anyone verify if 4.4.3 resolves the corruption issue?

Is it mentioned as fixed in 4.4.3 release info?

I don’t see it mentioned, but I’ll download and try later today.

[arvidn]

I can't reproduce the issue, so I can't know whether it's fixed or not. It might be helpful to pull more information from the files that are corrupt for you. Things like how many extents there are, last modification time, whether they are corrected eventually. I'm suspecting that there's some cache coherence issue where the data has made it into some write cache, but some other ways of accessing the file circumvents that cache and only considers what's actually hit the disk.

[loekf]

I can confirm it still occurs on 4.4.3 and 4.4.3..1. Not all files, maybe the file size has something to do with it.

I see hiccups when playing downloaded files. Video stops, then resumes or recovers after a few seconds.

Very annoying, back to 4.3.9

Not related probably, but I see that 4.3.9 also downloads much faster. 4.4.3.x takes ages to download metadata and actually start.

[jdenman03]

I can confirm it still occurs on 4.4.3 and 4.4.3..1. Not all files, maybe the file size has something to do with it.

I see hiccups when playing downloaded files. Video stops, then resumes or recovers after a few seconds.

Very annoying, back to 4.3.9

Not related probably, but I see that 4.3.9 also downloads much faster. 4.4.3.x takes ages to download metadata and actually start.

Yes it is still happening for me as well even with the newest update.

[ghost]

@jdenman03 can you please test with disabling OS cache? (Untick the enable OS cache from advanced settings)

[thuglifem]

If i may answer - results are the same with OS cache disabled. Still got corrupt files when copy downloaded files to other drives. Tried everything suggested above and my observations are exactly the same as OP. 4.3.9 works with no issues. (12.4 Monterey on mba m1)

[jdenman03]

If i may answer - results are the same with OS cache disabled. Still got corrupt files when copy downloaded files to other drives. Tried everything suggested above and my observations are exactly the same as OP. 4.3.9 works with no issues. (12.4 Monterey on mba m1)

You are correct. Results are the same for myself as well even with the OS cache disabled. I meant to reply back, but forgot. Thanks for chiming in.

[hany]

Just found this issue. I thought I was going crazy, but finally traced the problem down to qBittorrent. Freshly downloaded files will play file locally, but when copied to an external NAS, they are corrupted. Things started working again after a reboot for previously downloaded files. I've downgraded to 4.3.9 for now.

[Steve-Agnew-NZ]

I've been having the same problem for a couple of months - copying the file with Finder in OSX 12.4 corrupts the target file - the MD5 hashes of the source and target are different. Copying using rsync works OK. There is some more analysis and speculation over here that you might find helpful:

https://discussions.apple.com/thread/253864981

I've been using qBitTorrent 4.3.1 so you may find downgrading doesn't fix it.

[hany]

Just FYI, since downgrading to 4.3.9 (from 4.4.3.1 and 4.4.2), the issue is completely gone. I've been running md5's on all downloaded files on the source and destination. Previously every single copied file would be corrupted, and now I have 100% success. I've been copying using Finder this entire time. My source is Catalina 10.15.7 and destination is a NAS.

Before stumbling on this issue, to "workaround" the problem, I would often download alternative versions of the file. These versions were zip or rar archives. I would always unpack the archive on the source system, then copy the newly generated/unpacked file using Finder. This would always work since the file being copied was not the originally downloaded file.

[dldingfls]

Downgrading to 4.3.9 worked perfectly for me too. I lost a lot of hair using version 4.4.3, with glitchy files that pixelated and with image freeze and sometimes total file shut-down. Totally sorted now!