search

qbittorrent / qBittorrent

qBittorrent BitTorrent client
https://www.qbittorrent.org
Other
28.35k stars 3.99k forks source link

"Missing files" (file sizes mismatch) error when qBittorrent restart and torrent progresses back to 0% #10408

Closed ccloli closed 9 months ago

ccloli commented 5 years ago

qBittorrent version and Operating System

If on linux, libtorrent and Qt version

N/A

What is the problem

When qBittorrent start up, a bunch of torrents show Missing files, and execution log lists as File sizes mismatch for torrent '${torrentName}', pausing it. or Fast resume data was rejected for torrent '${torrentName}'. Reason: ${torrentName} fast resume rejected. check resume(): missing or invalid 'file sizes' entry. Checking again...

image

After the file is checking completed, the status is still Missing files.

image

Once qBittorrent restarted, its progress still shows 0% with Missing files status (on screenshot it's rechecking again).

image

What is the expected behavior

The torrent should not be "Missing files", or should be marked as "Completed" after verification and won't restore after restarting.

Steps to reproduce

At this time I cannot reproduce it. I've tried making a new hard link on a same volume (mklink /d /h /j D:\source D:\target) or different volumes (mklink /d /h /j D:\source C:\target, which is my case) cannot reproduce it.

image

If I've got something new, I'll note them here.

Extra info(if any)

ccloli commented 5 years ago

Okay, I find a solution, right click the affected torrent and select Force resume, then it'll recheck again, but this time its status will not be Missing Files but Seeding or Completed.

I tried some torrents that less than 3GB, after restarted, they're not in Errored list, and their status are Seeding and progress are 100%. Finally my HDDs are rescued :-)

image

image

However I'm still confused why the torrent status is not changed to Completed after auto-recheck, so I think the issue still needs to be opened until someone finds the reason.

bogdansoima commented 5 years ago

I have the same problem as you have. Unfortunately qBittorrent is riddled with bugs (this one, the fact that it starts rechecking all torrents after a crash even if they were completed, the fact that its checking all files simultaneously and not in sequence, high CPU usage when having 4-500+ torrents in client)... I will try some other clients.

morzsa11 commented 5 years ago

I have the same problem with qBittorrent-nox v4.0.3 on Ubuntu 18.04 server. Almost every time I restart the server all torrents go in error and the log contains 'file size mismatch' for each torrent. Need to do restart/recheck to get hem out of error. Even if not in error torrents are almost always stalled, while the same at the same time seeds ok on my Win PC, qBitorrent.

ned-martin commented 4 years ago

I am experiencing this problem with v4.2.0

rubbo898 commented 4 years ago

Same problem here, It seems random...at least I didn't find a pattern. Any idea on how to prevent fix this kind of behaviour? With Docker it's really annoying...

Janaue commented 4 years ago

same situation with qbittorrent_4.2.0_x64_libtorrent_1_2_2_gitf49d194186

mlindner commented 4 years ago

This is a really frustrating behavior. If the resume data ever gets corrupted it just hangs and never tries to download the file again. The default behavior should be to recheck the file, if it can find it, and restart downloading from scratch.

Camios commented 4 years ago

seeing this in 4.2.1 64 bit windows.

Junker commented 4 years ago

same problem, version 4.2.5

palmarci commented 4 years ago

same on v4.2.5 on arch linux

FranciscoPombal commented 4 years ago

@Junker @palmarci

Can any of you provide detailed steps for others to reproduce a minimal case? Also, please post your logs and settings in plain text (make sure to redact any potentially sensitive information), so it's easier to setup a similar environment and more variables can be eliminated.

bluboyo commented 4 years ago

No fix on this? II have it to.. Windows 10

festoney8 commented 3 years ago

The same here. I try to restore 3k+ torrents to another server, and qbittorrent report the same error cause file missing(I remove some files when I organize my storage). Qb webui shows no torrent in the queue. All the torrent file in BT_backup folder is named in torrent_hash.torrent format so I cant find which torrent cause error. So now I have to parse the torrent and fastresume file manually to check which should be remove ...

update: I try remove some .torrent .fastresume file but qb still stuck somewhere after start and show nothing in webui. Finally I parse these torrent_hash.fastresume file in binary mode and extract torrent save path with regex "save_path\d+:(.+)/9:seed_mode" , then use flexget read BT_backup folder to add torrents, specify them correct savepath according to the torrent_hash - savepath relation. It worked.

C84186 commented 3 years ago

The same here. I try to restore 3k+ torrents to another server, and qbittorrent report the same error cause file missing(I remove some files when I organize my storage). Qb webui shows no torrent in the queue. All the torrent file in BT_backup folder is named in torrent_hash.torrent format so I cant find which torrent cause error.

I am in the exact same position as you.

update: I try remove some .torrent .fastresume file but qb still stuck somewhere after start and show nothing in webui. Finally I parse these torrent_hash.fastresume file in binary mode and extract torrent save path with regex "save_path\d+:(.+)/9:seed_mode" , then use flexget read BT_backup folder to add torrents, specify them correct savepath according to the torrent_hash - savepath relation. It worked.

Could you please share more information on how you accomplished this? Eg, your flexget config, script you used, etc.

C84186 commented 3 years ago

Personally, without having a real fix for this, my "solution" is as follows.

@festoney8 's comment suggests this problem is definitely with fastresume files, presumably multiple fastresume files.

I knew migrating came with risks, especially since the target host had already been downloading torrents, but, alas, I'm a dumbass, and forgot to backup my BT backup.

The webui is totally busted, and I don't have access to the desktop UI because of my setup (I run via a docker container which forces traffic through a VPN tunnel)

The things I DID have were copies of:

What I could use to recover

Also:

The qBittorrent data directory looks like this:

- qBittorrent
- Completed
     - category1
     - etc
     - ... uncategorized files at top level
  - Downloading
     - category1
     - etc
     - ... uncategorized files at top level
   - torrent_files
     - Completed
     - Downloading

Assumptions about how things work

Well, I know that .fastresume s do at least two things

I know that these copied .torrent files are NOT likely to encode the path, categories, or user-led changes to the torrent's structure (File renames, etc)

I strongly suspect that the hash.torrent files stored in BT_Backup are also not likely to encode local changes to the structure, but I don't know.

I also know that, (In my case) I am luck enough that all of my categories have very distinct trackers, because I download each from distinct sources.

I also know that all of my categories tend to have different filetypes & extensions.

Some path forward

Unbreaking qBittorrent

All I have to do to unfuck qBittorrent is the following:

# Your path will be different, this path is actually mounted by docker as the config path
cd /data/config/data/
ls
#  BT_backup  GeoDB  logs    rss 

mkdir /data/qbittorrent_backups

# Let's actually back this up. We'll use this backup in a later step, so I'm not just being careful.
# suffixing this one with .01. There's better ways to version this I'm sure but this will do
mkdir /data/qbittorrent_backups/BT_backup.fastresumes.01.busted
cp BT_backup /data/qbittorrent_backups/BT_backup.01.busted

# Get rid of all those nasty fastresumes
mv BT_backup/*.fastresume  /data/qbittorrent_backups/BT_backup.fastresumes.01.busted

Reintroducing your torrents

I can copy the backed up .torrent files (The ones I configured qBittorrent to put next to my downloads) to temporary directory

cp -r /data/qBittorrent/torrent_files /data/torrent_files_re_introduce

Collapsing/Flattening your category paths so that qBittorrent will recognize them.

Because these torrent files aren't smart enough to know I've changed categories, I need to move everything out of category subdirectories, into the root download dir.

cd /data/qBittorrent/Downloading

mv ./category1/* .
# If mv fails, because maybe somehow you've already restarted downloading into the "Downloading" dir, 
# use the "supermove" script I link to

mv ./category2/* .

# etc

cd /data/qBittorrent/Completed

mv ./category1/* .
mv ./category2/* .

# etc

Now I can start qBittorrent,

In qBittorrent, I nominate some "monitored" torrent directories, under Tools/Options/Downloads/Automatically Add Torrents From:

- path: /data/torrent_files_re_introduce/Completed
   save_location: default
- path: /data/torrent_files_re_introduce/Downloading
  save_location: default

I save the settings, and my list populates with torrents at 0%.

I press -A , all torrents are selected, I right click on a torrent and tell it to recheck.

Now I start seeing the slow recheck process (I bought large, suspiciously cheap, harddrives that I didn't realize were designed for CCTV storage so are slow AF).

I start seeing the "Seeding" status going up quickly, yay.

To ensure I've got everything, I also want to watch for my torrent files left over in BT_Backup

mkdir /data/torrent_files_re_introduce/from_BT_backup
cp /data/qbittorrent_backups/BT_backup.01.busted/*.torrent /data/torrent_files_re_introduce/from_BT_backup

In qbittorrent, I do the following:

Tag all existing torrents (Ctrl-A, then tag) with "already_checking". I dont want to have to re-recheck these, so it'll be nice to be able to tell them apart

I add the following:

- path: /data/torrent_files_re_introduce/from_BT_backup
   save_location: default

Save, click on the "untagged" tag category, select all & right click "recheck".

Next, I want to re-sort my torrent categories.

To automate this, I'm going to use the qbittorrent web API, but I'll document that once I finish writing the script.

festoney8 commented 3 years ago

@C84186

Mainly these steps below: Step 1: backup BT_backup folder to your pc or somewhere else just in case

Step 2: parse torrent_hash.fastresume file to get torrent save path and correspond the path to torrent_hash.

I wrote a python script to parse fastresume file, put the script in BT_backup folder then run with python3, and we get the torrent_hash - save_path relation.

import os, re

arr = os.listdir('.')
reg = re.compile(rb'save_path\d+:(.+).9:seed_mode')

for fpath in arr:
    if fpath.endswith(".fastresume"):
        with open(fpath, "rb") as f:
            match = re.search(reg, f.read())
            if match:
                save_path = match.groups()[0].decode('UTF-8')
                torrent_hash = fpath.split(".")[0]
                print("-", torrent_hash+":")
                print("    "+save_path)

the output may like below, I deliberately output them in that format cause step4 need it

- 29cf3f811ba102610c946ca6e048xxxxxxxxxxxx:
    path: /save/path/1
- f8d29f01d648b8dfefccd26e5e61xxxxxxxxxxxx:
    path: /save/path/2
- b9fef4a0576a8edee88e21c0f0a2xxxxxxxxxxxx:
    path: /save/path/3
- 76cd7e7da6f6d093e36ae5ff4da0xxxxxxxxxxxx:
    path: /save/path/4

Step 3: build up your new qb environment, keep the new BT_backup folder clean. Make sure all your data storage has not been moved to another path, otherwise, you need to batch edit these paths in step2

Step 4: add torrents to qb. There are two ways to add torrents to qb: use flexget watch torrents from the folder, add torrents to qb and specify the correct save path according to the relation we get in step 2 or write a script to add torrent to qb via qb-add-torrent API. Recheck or force jump hash check is according to your demand.

My choice is using flexget. Two plugins are needed, filesystem and qbittorrent. Filesystem plugin is used to watch the BT_backup folder, filter files end with ".torrent" and match torrent name to save path with regex. Templates part and tasks part of flexget config is like this:

templates:
  qb:
    qbittorrent:
      host: xxx
      port: xxx
      username: xxx
      password: xxx
      add_paused: True
      skip_check: False

tasks:
  mytask:
    filesystem:
      path:
        - /path/to/BT_backup
      mask: '*.torrent'
    accept_all: no
    regexp:
      accept:
        - 29cf3f811ba102610c946ca6e048xxxxxxxxxxxx:
            path: /save/path/1
        - f8d29f01d648b8dfefccd26e5e61xxxxxxxxxxxx:
            path: /save/path/2
        - b9fef4a0576a8edee88e21c0f0a2xxxxxxxxxxxx:
            path: /save/path/3
        - 76cd7e7da6f6d093e36ae5ff4da0xxxxxxxxxxxx:
            path: /save/path/4
      from:
        - title
    template:
      - qb

In qbittorrent part, enable add_paused to prevent qb start downloading if hash check failed. And you can enable skip_check if you have enough confidence :D The accept part all comes from step2, flexget regex match filename with these rules and specify save path to torrent. If you have too many torrents, that will be such a long config file. When flexget run it will automatically add torrent to qb

I advise you to use 5 or 10 torrents at first to test if flexget works

C84186 commented 3 years ago

Hey @festoney8 , this was super great so thank you - I had already gone ahead and done my "Fix" which involved collapsing all of my categories & rebuilding them.

Here's a script that uses webUI to automatically tag torrents based on content - You can use the tags to fix your library

https://github.com/C84186/qBittorrent_tag_by_pattern

Madhorse1 commented 3 years ago

Guys, I'm experiencing just the same problem as you! I am seeding around 200 video torrents that were created by myself. I'm running a private raspberry pi server with qbittorrent for seeding. After some time I ran into this issue missing files with several seeds where after a server restart, I'm missing smaller files (like 200 kb files and 0,8 Mb files from video for example) and cant figure out how to deal with this. Reading your posts seems like you have found some kind of solution but for me it's all very difficult to figure all this out and I don't really have much time to deal with it. If anyone would be so kind to maybe point out what to do to stop this behavior. (Just head up, I tried to reseed the same torrents from local computer and Utorrent also can't locate these tiny parts of torrent and I get the same error missing files.) It really seems like the tiny pieces of the torrent are deleted or missing for some strange reason.

Because of this issue, it's simply impossible to seed torrents and I need to sort this out once for all. Please help!! Thnx a lot in advance!! image

ccloli commented 3 years ago

After some time I ran into this issue missing files with several seeds where after a server restart, I'm missing smaller files (like 200 kb files and 0,8 Mb files from video for example) and cant figure out how to deal with this.

@Madhorse1 I don't think your issue are related to this one, our torrents are completed, and can be rechecked with no file missing, but the torrent status is still "Missing Files" and saying "file size mismatch" on log.

If you're downloading the torrents from somewhere (not created by yourself), your issue might be because you only dowloaded the video file, and other files like image and nfo files are not downloaded. qBittorrent will save the files not used but included in used file's block chunk in .unwanted folder, if you manually deleted the folder, you may have the issue.

kimfr commented 3 years ago

I've also been having issues with download torrents showing up as "Missing Files" after restart. To me it looks like an issue with network shares.

It's easy to see this with small files present:

My situation:

If I enable storing of incomplete torrents on a direct local drive, I can see that in the above example, the 2 files are downloaded correctly (ie. 1 30 bytes and the other 99 bytes) - however after being moved to the cifs share, they now appear incorrect (ie. both 99 bytes).

I'm running a openSUSE Leap 15.2 - was using qbittorrent 4.2.5 from the openSUSE repo's. Have in the meantime switched to the latest container from linuxserver.io (https://hub.docker.com/r/linuxserver/qbittorrent) - same result.

On the cifs mount line, adding "cache=strict" or "cache=none" on the linux mount command didn't resolve the issue.

dark-swordsman commented 2 years ago

Okay, I find a solution, right click the affected torrent and select Force resume, then it'll recheck again, but this time its status will not be Missing Files but Seeding or Completed.

@ccloli I just wanted to come here and say that you saved me. I have over 2 TB of stuff that it was suggesting I needed to re-download when I moved my collection to a TrueNAS pool. Forcing Resume fixes the problem and it finds the file. Note: I am using the temporary folder feature, which I think was part of my problem.

Edit: I guess recheck only worked for one file. Aw well.

kimfr commented 2 years ago

I've resolve the issue in my setup by switching from a samba/cifs share to a nfs share.

iliaskomp commented 2 years ago

I had the same issue with qbittorrent-nox on raspberry pi with external hard disk. It seems that the qbittorrent-nox service starts before hard disk is fully up and running and qbittorrent cannot find the files. My solution is to delay the qbittorrent-nox startup for 30 seconds, just to be on the safe side. Hope it helps.

porg commented 1 year ago

I checked my logs and got the same errors on some of the torrents.

In my case the affected torrents had at least some of these attributes:

Technically this happened

How to fix (the GUI only way)

1) Right click torrent > Pause. 2) For each unchecked file, check it and then uncheck it.

How to fix (with partial CLI work along the way)

Ad 2) Instead of toggling undesired files ON and back OFF again, you could also do this:

Retia-Adolf commented 1 year ago

I have this issue from time to time too. Now it still presents in 4.5.4.10 (on Ubuntu 22.04), and for me force rechecking stuck at 0%, only started rechecking after I pause and resume it. Didn't try 'force resume' though.

superbonaci commented 1 year ago

Could be a disk or sector error? Did you check with smartctl or CrystalDiskInfo?

Retia-Adolf commented 1 year ago

Could be a disk or sector error? Did you check with smartctl or CrystalDiskInfo?

If you mean me, I had done full-disk scan before attaching it to zfs, and gsmartcontrol doesn't show any error count or log either now.

The task could be resumed to normal downloading state, it's just got "missing files" error sometimes for unknown reason.

superbonaci commented 1 year ago

I also got this error when exporting the torrent:

7/11/2023 05:22 AM - Failed to restore torrent. Files were probably moved or storage isn't accessible. Torrent: "torrent_name". Reason: "torrent_name fast resume rejected. file_stat(fullpath.avi): mismatching file size"

However after force recheck the file fullpath.avi is 100% progress, so no idea how to reproduce this.

xavier2k6 commented 9 months ago
  • qBittorrent v4.1.4 (also in v4.1.5)
  • Windows 10 (1803, x64, lang: zh-CN) 10.0.17134

Version(s) are obsolete.

Thank you for your contribution, we are starting to close all old/stale/obsolete tickets.

Please update to latest qBittorrent 4.6.3 ATTOW.

If any issues are experienced, please open a new ticket.

Windows Users: (Note: Windows 10 1809+ / Windows Server 2019 required as minimum)