add a "trust me, it's all at 100%" option to skip the data check for seeding

[jbruchon]

I recently had to remove and re-add a 120GB torrent that I created and that I'm seeding. When I point the re-loaded torrent to the data folder containing the data, all the data is re-checked. I'd like the ability to skip this re-check and tell the program that all the data is fully intact and at 100% completion.

[FranciscoPombal]

Doesn't the "skip hash check" checkbox in the add torrent dialog achieve what you want?

[jbruchon]

No. I don't have an "add torrent" dialog. I may add a hundred torrents in a few minutes, so that is off. There are several things I'd like to see available outside of that dialog that are part of that dialog, such as the ability to download magnet metadata without starting the actual data download, but I don't want drill down into settings twice to turn that box back on and then back off every time I have one particular torrent I'd like to make that exception for. I didn't know that was in that box because I always have it off, so it is effectively dead to me. Edit: might have come across harsh, but not meant that way, sorry

[R-Adrian]

i'm opposed to skipping the data check for seeding in the official qBittorrent releases. If someone wants to do this they should have to recompile the client from source and then the client should announce itself with a radically different peer_id that doesn't claim to be qBittorrent in any way, shape or form.

This feature can be heavily abused to seed fake/poisoned torrents or files that have been intentionally changed and corrupted to send malicious data specially crafted to crash some remote clients or conduct various remote exploit attacks (including, but not limited to, potentially doing a remote code execution in the remote client). This "feature" is also a relatively common tactic used by anti-piracy enforcement companies.

Well-designed clients will drop any bad checksum data without crashing (or executing remote code while crashing), but poisoning torrents with bad checksummed files is enough of a low-effort way to waste the time of thousands of other users.

if qBittorrent implements in the official releases such features that contribute to torrent poisoning, it will then probably become blacklisted / banned from pretty much every tracker out there.

[jbruchon]

@R-Adrian Well according to @FranciscoPombal it apparently already has the feature (tucked away in a dialog box that some users prefer to disable) and it's not blacklisted everywhere, so...maybe read the thread prior to adding your two cents? Just a thought.

[R-Adrian]

true, it has the checkbox for when adding a torrent file... which can have its contents be 0,001% finished or 99% finished downloading. That one can be useful when resuming a download.

But the client then does another full data check anyway when it finishes downloading and starting to seed. This check is what's at issue: full torrents being seeded with bad data. Partial torrents... meh.

[jbruchon]

You don't seem to understand how useless your argument is. All I have to do to poison the data in your scenario is have the legit data on hand first, then in-place overwrite the parts I want to taint after the data check finishes. Alternatively, the fastresume can be modified offline to contain the desired state, bypassing the check entirely. Also, the client doesn't do a full data check when it finishes downloading unless you check a box that explicitly forces that to happen.

You are arguing that the program should police the owner of the machine, much like DRM and software activation. This is an unacceptable stance, especially in a piece of open source software, and doubly so when the anti-user provision is hilariously easy to bypass. The only reason I don't do it instead of adding a feature request is because it's merely a matter of (in)convenience for me; I can wait for 121GB to transfer over a 2.5G ethernet connection from a RAID array, but I don't think I should have to when I'm the origin seeder and my copies are literally all of the source data for the torrent.

[R-Adrian]

ok.. for particular cases where there are lots of unique data, it might make sense to have a checkbox for skipping all checks somewhere under tools -> options -> advanced ... preferably behind a warning that says "beware...you risk making a mess of things. Also, there's a leopard in here." Bad torrent data will happen anyway, either in official or custom-compiled builds, but its better to not make it too easy to happen (Murphy and his laws...) And it's not exactly useless.. it raises the knowledge bar a bit, being probably just enough of an inconvenience so that we don't see it everywhere. Not many people will venture to modifying those program configuration files.

Also, about the second data check, on finish... i didn't realize that... i probably enabled that setting long ago and got so used to it that i forgot about it... though i still like its use, especially when qBittorrent crashes randomly. (last such crash that i had was in... november last year... i had to redo all my qBittorrent settings.)

[jbruchon]

A possible way to mitigate your concerns is to have the program run a data integrity check on a piece before it is sent out. That not only prevents sending out bad data, but also catches any sort of corruption, intentional or otherwise, and brings it to the user's attention. That would actually be a nice thing.

[R-Adrian]

indeed, a delayed just-in-time data check only when a piece is first requested seems useful

just like with manufacturing, instead of checking the entire stock before production, check it during production, when needed. https://en.wikipedia.org/wiki/Just-in-time_manufacturing

this might actually solve the problem of checking large files.

[FranciscoPombal]

@R-Adrian @jbruchon

Guys, the situation is really not that bad. @an0n666 has done some work recently to prevent "fake" seeding and torrent poisoning (theses fixes made it to 4.3.0): https://github.com/qbittorrent/qBittorrent/pull/13206 and https://github.com/qbittorrent/qBittorrent/pull/12037. From my understanding, as soon as a piece is requested from a file that doesn't exist, the torrent will go into "missing files" state. Also, I'm pretty sure libtorrent doesn't let you seed bad pieces on purpose already (for example, if you create files with the same name but with random/bogus data for seeding). A sufficiently motivated attacker can of course modify the code to allow for this, but at least mainstream libtorrent-based clients have peer-banning heuristics built-in to defend against this.

So I really don't see a problem in adding a way to add multiple torrents with "skip hash check" at once. But please, @an0n666 @arvidn correct me if I'm wrong here.

[as-muncher]

I personally would want to make sure the data was at 100% before seeding. I know that I have downloaded some stuff, that obviously was seeded as a new torrent, but had only completed to maybe 99.5% from its original torrent. I currently sucks as of 4.3.0.1 with the torrents and rechecking and all that, because that's happened to me a lot, too, so I can see why you're so upset that you had to sit there, waiting for your 120 GB data to be checked.

[jbruchon]

@as-muncher My concern is when I know I have the data at 100%, particularly because I'm the person who made the torrent. In such a situation, being able to skip the hash checking process makes a lot of sense.

[as-muncher]

Yeah, I guess that makes sense if you're making a new torrent like that. I suppose qbittorrent might have to do something if there were, for some odd reason, already a torrent with the same name. Maybe it's making a hash of the torrent or something.

[smiba]

Sorry to bump this, but I would really love this feature.

Once a year or so a hick-up (or error on my part) causes the network share to disconnect resulting in it having to "check" over 3TB of torrent files which takes a good half a day or so.

I know for sure the data is correct, it's stored on ZFS which means it will auto-heal. Also the network storage itself never went down, just the mount, so no corruption could've happened to torrents that were at 100% already.

To be sure, this is not the "Skip hash check" at the torrent adding screen, the torrents have already been in my client, finished downloading sometimes as much as 2 years ago and just got marked as requires recheck because the storage got disconnected.

[as-muncher]

@smiba Have you tried perhaps BiglyBT, or maybe a different torrent client? I know with some of these torrent clients, they're supposed to just check the data that was in question at the time of a crash or what not, not the entire thing.

[smiba]

Switching is not really an option, I specifically choose qbittorrent so I could use qbittorrent-nox on linux and access it from the webbrowser. I used Transmission previously but that was just a miserable experience, qbittorrent does exactly what I want and has a ton of options, but this is the only real issue I'm struggling with.

[arvidn]

Libtorrent is a lot less sensitive (and less trigger happy) to require a full check than it would as a few years ago. I would be interested in knowing what may have triggered the recheck. Is it enough to start up and all files are missing?

Iirc, one use case that was fixed a while back is when an entire mount point goes away. But these scenarios don't have great test coverage.

Do yo have logs from the time when the resume data was considered invalid? I assume by now the resume data has just been wiped and is just straight up missing.

[smiba]

What happened this time is that I rebooted the VM, qbittorrent autostarted but the file share didn't mount correctly (not mounted at all / missing)

This resulted in all files being reported as "Missing files", I shutdown qbittorrent again, fixed the mount, started qbittorrent again and this caused all the files to require rechecking.

The download directory / resume file is locally to the system (I move finished torrents manually through the "move" option), so the failure of this network share shouldn't cause issues with the resume file. The only thing I can imagine is that qbittorrent didn't shutdown safely the first time, this combined with the missing files might have pulled it over the edge? But fairly sure I did shutdown qbittorrent safely before the reboot.

The log looks like this (qbittorrent.log):

(N) 2022-05-24T16:22:15 - qBittorrent v4.2.5 started
(N) 2022-05-24T16:22:15 - Using config directory: /home/smiba/.config/qBittorrent/
(I) 2022-05-24T16:22:15 - Trying to listen on: 0.0.0.0:59232,[::]:59232
(N) 2022-05-24T16:22:16 - Peer ID: -qB4250-
(N) 2022-05-24T16:22:16 - HTTP User-Agent is 'qBittorrent/4.2.5'
(I) 2022-05-24T16:22:16 - DHT support [OFF]
(I) 2022-05-24T16:22:16 - Local Peer Discovery support [ON]
(I) 2022-05-24T16:22:16 - PeX support [ON]
(I) 2022-05-24T16:22:16 - Anonymous mode [OFF]
(I) 2022-05-24T16:22:16 - Encryption support [FORCED]
(I) 2022-05-24T16:22:16 - UPnP / NAT-PMP support [ON]
(I) 2022-05-24T16:22:16 - IP geolocation database loaded. Type: DBIP-Country-Lite. Build time: Fri Oct 1 00:06:22 2021.
(N) 2022-05-24T16:22:16 - Using built-in Web UI.
(W) 2022-05-24T16:22:16 - Couldn't load Web UI translation for selected locale (C).
(N) 2022-05-24T16:22:16 - Web UI: Now listening on IP: *, port: 8080
(I) 2022-05-24T16:22:16 - Successfully listening on IP: 127.0.0.1, port: TCP/59232
(I) 2022-05-24T16:22:16 - Successfully listening on IP: 127.0.0.1, port: UDP/59232
(I) 2022-05-24T16:22:16 - Successfully listening on IP: 192.168.2.153, port: TCP/59232
(I) 2022-05-24T16:22:16 - Successfully listening on IP: 192.168.2.153, port: UDP/59232
(I) 2022-05-24T16:22:16 - Successfully listening on IP: ::1, port: TCP/59232
(I) 2022-05-24T16:22:16 - Successfully listening on IP: ::1, port: UDP/59232
(I) 2022-05-24T16:22:16 - UPnP/NAT-PMP: Port mapping successful, message: successfully mapped port using UPnP. external port: TCP/59232
(I) 2022-05-24T16:22:16 - UPnP/NAT-PMP: Port mapping successful, message: successfully mapped port using UPnP. external port: UDP/59232
(N) 2022-05-24T16:22:16 - 'Torrent1' restored.
(N) 2022-05-24T16:22:16 - 'Torrent2' restored.
(N) 2022-05-24T16:22:16 - 'Torrent3' restored.
(N) 2022-05-24T16:22:16 - 'Torrent4' restored.
(N) 2022-05-24T16:22:16 - 'Torrent5' restored.
(N) 2022-05-24T16:22:16 - 'Torrent6' restored.
[...]

EDIT: I do realize I'm not running the latest version of qbittorrent, so maybe this has been fixed? I should probably update it at some point

[as-muncher]

@smiba I think that's what @arvidn was saying when he mentioned the latest version of qbittorrent, that qbittorrent used to be more prone to checking. You're using 4.2.5 when the latest now is 4.4.3. How about upgrading to 4.4.3 and then seeing if you keep getting this problem?

[smiba]

@smiba I think that's what @arvidn was saying when he mentioned the latest version of qbittorrent, that qbittorrent used to be more prone to checking. You're using 4.2.5 when the latest now is 4.4.3. How about upgrading to 4.4.3 and then seeing if you keep getting this problem?

Yeah only realized a moment ago it's been a good while since I've updated! I'll update soon and report back if it happens again, could take months though. Usually it just works without issues, as long as I don't mess around with my network or equipment too much :)

EDIT: 2024-04: I've not see this issue since updating, so this definitely seems to be resolved for me

[jbruchon]

For what it's worth, when my network drops with the 4.3.x series, I can close and re-open qBT after reconnecting the network shares and it picks up again without re-checking. This doesn't help with the original issue (removing and re-adding the torrent) but it helps with the "missing files" due to a missing device.

[arvidn]

The download directory / resume file is locally to the system (I move finished torrents manually through the "move" option), so the failure of this network share shouldn't cause issues with the resume file.

My concern would be that once you get into this state, it's recorded in the resume files when you shut down. i.e. it's possible that it's just that one event of the the files missing that's interesting.

Once the resume files say the files need rechecking, it's not surprising that they need checking.

[ElDavoo]

Network drives are much more error-prone, but an IO error does not necessarely mean data corruption. The moment a network hiccup happens, all read give an IO error and all torrents need to be rechecked. This is completely unfeasible. Just add the option already and let the user know this is very dangerous to do, chunks are already checked as they are seeded. (biglybt takes a lot of ram but i'm afraid i will switch to it)

[arvidn]

@ElDavoo to someone who hasn't experienced this issue, your message is a bit cryptic. I think "the option" you're talking about is called "seed mode", is it not? but I wouldn't consider it dangerous

[ElDavoo]

I was talking about "disable hash check" option. of course the client will check the hash "on-the-go", e.g. before sending it to the peer. Transmission does that, not sure about others

[as-muncher]

I'm actually really amazed by qbittorrent 4.4.3.1 on Windows 10 x64. I have my torrents on an external device, and my system has frozen, and so I initiated a manual blue screen. When I started up my computer again, started up qbittorrent, my torrents started downloading again right away without any rechecking, saving me a ton of time from all the rechecking. Great job to the developers!! Is this your doing @arvidn ? :)

[as-muncher]

@arvidn One thing I've noticed, though, and I haven't checked to see if someone has already made an issue about this, is that when I initiate a crash, and then reboot the computer and start up qbittorrent again, qbittorrent will start working right away without checking, but I'm sure glad that I have it set to add the .!qB extension to incomplete files and to recheck the torrent upon completion, because the torrent usually needs to be fixed. Maybe that data between the resume save state and the crash is not being checked, but at least that amount should be so that I don't have to recheck the torrent, and then download the missing pieces, and then check all over again. Maybe there could be some background checking. And with people not choosing to have the .!qB extension and not rechecking upon completion, I bet there are a lot of people who think they have completely downloaded the torrent, but it's actually faulty or incomplete.

[arvidn]

when I initiate a crash, and then reboot the computer and start up qbittorrent again, qbittorrent will start working right away without checking

I assume by "initiate a crash" you mean cutting power to the computer. Not a graceful shutdown.

The intention is to not re-check files willy-nilly. If there is a correct-looking resume data, it will be trusted. There are some cheap checks to ensure all files the resume data says should exist actually do.

In the scenario where you cut power though, it's possible that data sent to the operating system never got to the disk before losing power, but the most recent resume data may still say we have it. In this scenario your downloaded file will be corrupt and should be re-checked.

Many years ago, libtorrent would save file timestamps in the resume data, and if they didn't match the files on disk it would run a full check. That ended up with a lot of false positives for checking. It was primarily needed when using "compact" mode storage, where pieces are moved as new ones are downloaded, to ensure the file on disk is contiguous (not sparse). That mode was removed from libtorrent a long time ago, which also weakened the justification for the timestamp checks.

The current resume data, with data written to sparse files to the correct location, makes the assumption that more data may have been downloaded since the last resume data was written, but as long as resume data is saved frequently enough, it's fine to re-download that.

Does qbt save resume data regularly in the background? If so, how often?

[as-muncher]

@arvidn https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/forcing-a-system-crash-from-the-keyboard set to save every 5 minutes. I wish there was option like that for linux distributions, in case the system hangs, as it sometimes does in Windows. It would be nice if qbittorrent background checked generally the area of data that may be at fault, while at the same time downloading more, because at the end, when the torrent has downloaded, those seeders who have the needed data may not have it for quite some time, if ever. But I suppose it could also be my responsibility to check the torrent manually just in case, anyways on my own time.

[flamingodyssey]

With the "Skip hash check" option now available for "Watched Folder"s under "Options > Downloads", should this issue be considered fixed? At the very least, it seems to handily cover the remove-then-readd-torrent workflow mentioned in OP's comment.

[ghost]

@flamingodyssey No

[jbruchon]

Once again, I'm faced with TERABYTES of KNOWN GOOD TORRENTS that I have to wait to recheck. This needs to be implemented desperately. I'm getting sick of this.

[smiba]

Once again, I'm faced with TERABYTES of KNOWN GOOD TORRENTS that I have to wait to recheck. This needs to be implemented desperately. I'm getting sick of this.

Is this with the latest version of qbitorrent? What version are you running?

[jbruchon]

Oh, does the latest version of qBitTorrent somehow have the feature I'm the OP for added already? Because I don't see it.

[smiba]

Oh, does the latest version of qBitTorrent somehow have the feature I'm the OP for added already? Because I don't see it.

No, but the releases in the last 2 years or so have fixed this issue. I've not ran into it again since I've updated even when I've mistakenly started qbitorrent with the entire mount point disconnected. I think it has been fixed somewhere in 4.3.x or 4.4.x

I'm just wondering if you've tried a newer version since making this issue, no need to be rude

[jbruchon]

I made the original post. I'm a software developer so I wouldn't report issues without testing the latest versions first. I JUST had a problem today. Fast resume issues and fixes have absolutely NOTHING to do with a "trust me bro it's 100%" function either.

If you don't like perceiving rudeness then don't ask obviously unrelated and noob questions in a 3+ year old thread against the person who made the issue report. Wasting the time of others with unrelated nonsense is far more rude than pointing out the obvious to the oblivious.

[lukefor]

It's definitely not fixed :) Dug into it a bit when I last hit it on the latest version, needs fixed at the libtorrent level (no functions exposed that qBT could use for this) and is more complicated than you would expect.

[glassez]

@lukefor, @smiba It seems like you're talking about something other than @jbruchon.

IIRC, @jbruchon ask for a way of adding bunch of torrents in "seed mode" (i.e. without hash checking).

[jbruchon]

I'm about to put out a colossal YouTube archive torrent that will be several TB in size. If there is any way to ensure it won't suddenly be forced to recheck because of a power failure or network issue I'd appreciate it.

[smiba]

@lukefor, @smiba It seems like you're talking about something other than @jbruchon.

IIRC, @jbruchon ask for a way of adding bunch of torrents in "seed mode" (i.e. without hash checking).

I think you're right, but I'm still confused in what scenario @jbruchon runs into this situation that is not resolved by "skip hash check" during adding a torrent.

I'm probably missing something, but as far as I know there are three scenarios where qbittorent / libtorrent triggers a hash check:

1. Adding a torrent for which the data already exists, without selecting "skip hash check"
2. Starting qbittorrent without a successful fast resume
3. Manually asking for a recheck

It can't be 1., as @jbruchon mentions in one of the first posts in this issue that this is not what they're talking about. It apparently also isn't 2., as this got me a rude response, and I highly doubt it's 3.

Which keeps me confused as what he's trying to achieve, and how to reproduce this issue as I've not been able to reproduce this on my side even after pulling the power plug while in operation.

I'm about to put out a colossal YouTube archive torrent that will be several TB in size. If there is any way to ensure it won't suddenly be forced to recheck because of a power failure or network issue I'd appreciate it.

Could you give us any steps to consistently reproduce this issue? What do you do that causes this? I'm seeding roughly 3000 torrents and this has not happened to me in the last two years, where on 4.2.x this used to be something that happened nearly every reboot.

[lukefor]

For me the trigger was to restart qBT for a second time while some torrents were still in "checking resume data" state, then a random subset of those started fully rechecking. With 1000s of torrents it takes several minutes to check resume data on startup, so this is not hard to hit

Edit: Looking at #18341 this might be quite a bit harder to hit on qt6

[glassez]

Probably easier and better to take the Transmission approach of bailing out and erroring if files are missing, allowing to retry by restarting qBT or changing the file location, rather than binning the resume data and leaving no option other than a recheck

If I understand correctly, you mean the problem of "missing files" when starting qBittorrent. @smiba is right, the situation has been significantly improved in recent versions. If you find yourself in "missing files" state for some reason (for example, you forgot to connect the external drive where the torrent contents are stored), then qBittorrent will not lose existing progress (unless you explicitly tell it to do this by invoking Force Recheck), you do not even need to restart qBittorrent after you have connected the external drive - you can just start such torrents again. The same thing if you moved the torrent contents while qBittorrent was not running - just point the affected torrents to a new location (although this still does not handle well if the torrent has a separate "incomplete save path").

[jbruchon]

Could you give us any steps to consistently reproduce this issue? What do you do that causes this? I'm seeding roughly 3000 torrents and this has not happened to me in the last two years, where on 4.2.x this used to be something that happened nearly every reboot.

• Have a large RAID array in a Linux server connected to a Windows workstation via Samba (CIFS) and 10GbE
• Have qBitTorrent running on the Windows workstation seeding/downloading 2500+ torrents
• Crash the Linux server during 30 MiB/sec of downloads
• Reboot server and workstation
• Re-establish CIFS session
• Start qBitTorrent
• Most, if not all, torrents will recheck file data completely, even if they were seeding and couldn't possibly have had any activity.

The point is that this bad behavior wouldn't matter at all if I could just have seeds tagged with a category, select the entire category after failure, and right click and say "these are at 100%, don't check."

This request seems absolutely stupid if you have 10GB of torrents. Big deal. It's very different when you have 10TB+ of them. If 10TB of torrent data can be read at a sequential average rate of 300 MiB/sec (which is double that of a modern 10TB hard drive) it'd take 9.26 hours to finish recheck. We're talking small RAID array rebuild times every time something goes wrong. It's not trivial for my use case.

[glassez]

• Most, if not all, torrents will recheck file data completely, even if they were seeding and couldn't possibly have had any activity.

I wonder if they were initially added in "seed mode".

[glassez]

@jbruchon Also would you mind to provide your current qBittorrent version and other related info? (I don't want to break my head just because you're using an outdated version that behaves differently.)

[ElDavoo]

It's not trivial for my use case.

same (not using qb because of this, just monitoring this issue)

[p2424629]

Please implement a skip-hash-check in webgui also. Over 20TB of torrents changing from 100% seeding to recheck is difficult. Everytime it happens i just restore a backup of the QB files to have them as seeding again.

Using latest in docker.

[jbruchon]

@jbruchon Also would you mind to provide your current qBittorrent version and other related info? (I don't want to break my head just because you're using an outdated version that behaves differently.)

• qBittorrent v4.6.4
• Qt: 6.4.3
• Libtorrent: 1.2.19.0
• Boost: 1.84.0
• OpenSSL: 1.1.1w
• zlib: 1.3.1

[as-muncher]

I've had qbittorrent even just recently start rechecking torrents that were seeding for a long time, no problems, with no change in data. It doesn't make sense that they would need to be re-checked. Is qbittorrent seeing that something insignificant has changed? I can understand why the original poster wouldn't want this to happen. After all, as of now, while a torrent is checking, it has no movement on it. At least if something is in the process of being checked, perhaps at least the good part should be allowed to seed. I think a user can add a torrent and skip checking, though, but it seems that the real issue is the torrents having to be checked in full, seemingly at random. Thirdly, perhaps qbittorrent could check only that part of the torrent that was in use at the time of the network disconnect / power failure / other failure.

With qbittorrent checking a number of my torrents too, random ones (seemingly) that have been seeding for quite a while, when I quit qbittorrent, there were 17 torrents left to check. Upon starting it again, after seconds of quitting, now there are 37 torrents to check in full, when the numbers in Checking go down. That doesn't really make sense. It seems that qbittorrent is not saving its status as often as it should be. qB should save its progress every few seconds if not every second or continually.

Maybe is qbittorrent looking at "Last Accessed" and then thinking that the torrent data has changed and needs to be rechecked? Developers: try plugging in an external hard drive that contains the torrent data, then running Analyse/Analyze in the Defragmentation dialogue (right-click on drive letter in Windows Explorer, Properties, Tools, Optimise/Optimize, Analyse/Analyze) and maybe see if that messes things up. I use an external drive for my data.

EDIT: not just that, but when you "Set location" for a torrent, and move it onto another drive, sometimes qbittorrent will think that the data is no longer there and start downloading at the former location. Then, again, you have to "Set location", and the location will come up where it is supposed to download, and then force recheck. Why does this happen? I don't know, but it's probably related.

EDIT 2: qbittorrent may think something has changed, even when in airplane mode. When qbittorrent was checking my torrents, and made its way down to having 17 left to check fully, and I restarted qbittorrent, the numbers again started going down relatively fast, and then 37 were checking fully. This was all done in airplane mode. Then, after connecting to the web, 139 or more are checking fully. This is irrational.

[ElDavoo]

Everytime it happens i just restore a backup of the QB files to have them as seeding again.

This is actually a good idea, I might try this approach