Merge external (in)complete file to already running torrent

[Snurre86]

The image show two different torrents with the same content where the availability is less the 100%. You can see in the image that if you combine the parts you have a 100% of the content. (http://imgur.com/Fw6VyEZ)

Here a feature of merging the contents comes in, as this is the only way to get all of the torrent content (complete the torrent).

There are two ways of merging the torrents manually:

1. You open a second BitTorrent client and become your own "Peer/Seed" (complicated).
2. You stop one of the torrents delete/rename the file. Copy the other torrent's file to the stopped torrents destination folder (rename the file if needed) and "force recheck". You can now download the missing parts/seed. This will delete the progress of one of the torrents if it is incomplete but you might get the missing parts.

Notes: If you have one complete file and you just want to seed it to the incomplete peers, option nr.2 is best. If you have two or more incomplete files of the same file, option nr.1 is best.

[Snurre86]

If you do decide to seed it to the incomplete peers the peers rarely become seeds as incomplete torrents tend to have a large number of downloaders that delete the torrent so the share ratio tends to get high and as the torrent complete "Seed torrents until there ratio is reached - then paused/remove" kicks in and the torrents automatically stay incomplete to others.

I would suggest making an exception in the case of a torrent staying incomplete (downloading only in status) for a long time as default. See #4353

[sledgehammer999]

This is such a corner case that it doesn't warrant a special function in qbt. And you have provided a viable alternative (number 2) anyway.

[Snurre86]

This happens to about 10% of my torrents.

[sledgehammer999]

I mean this way: It is a corner case for user to download the same torrent/files from 2 different bt clients and want to merge them together. If you absolutely want to use 2 clients then use them to download different torrents. Not the same.

[Snurre86]

if you look at the image above you can see two different torrents with the same file in it. How would you go about finishing them, note the availability.

[sledgehammer999]

This is still corner case. I am leaving this open for debate. Personally, I will not bother implementing this. I also think the coding solution will be complex

[Snurre86]

I don't know what corner case means. I just hope you understood what i wrote and see the issue I present. I know as much about coding as John Snow knows about everything (nothing). I just see the issue at hand.

[sledgehammer999]

I don't know what corner case means

This means that you are probably one of the very few of people that might encounter this particular problem. This group is so small that a solution to your problem isn't really top priority.

This is your issue:

1. You have unselected some files from a multi-file torrent 2. The rest of the files download 100% and the torrent starts seeding 3. After it reaches the ratio limit it is paused. 4. However, since the other files (the unselected ones) aren't downloaded you want to continue seeding indefinitely.

[Snurre86]

No what is going on here is that only 99% of the one file is availably and if I only leave it like that the file will never complete because of the lack of availability. So I found a second torrent with the same file and downloaded that but it also only have 99% of the one file availably but the piece that is not availably is different between the torrents so a merger is the only way to get the file completed. because the torrents are dead separately.

I think you are too busy to really read or do anything, so Ill leave you to what ever your doing. I will try and figure out a good way to make a solution but if there where a better way to talk to the developers it would be allot easier to cooperate.

[sledgehammer999]

Sorry, the description on my previous comment was wrong. I was thinking about your other open issue and mixed them. I understand what the problem is here. I can't tell you when a solution will be implemented though.

[Snurre86]

[Snurre86]

Cleanup and updated first comment, should be easier to understand.

[chrishirst]

Bittorrent payloads, or any part of, CANNOT be "merged" unless they are absolute binary duplicates of each other. Having the same file names and the same size is NOT sufficiently close, they have to be exact, bit for bit, byte for byte.

[Snurre86]

I don't know how all of this works, but can't qbt search for a file with the same hash or what ever in a different torrent, like "force recheck" does?

[Snurre86]

Also will "force recheck" delete parts that does not match if lets say there is a frame more in one of the video files etc.?

[GeographicCone]

I was looking for a solution to this as well. Too bad there isn't any elegant one. For uTorrent there's mergetorrent but in general it's either torram, which depends on Python, or running another qBitTorrent instance and manually adding 127.0.0.1:<port> as a peer.

[mimagind]

I've had this exact problem with a download. The main file was available in different torrents with different sets of tiny accompanying files. Eventually, I had three different torrents stalled at 90-95%. As searching only found results of people looking for a solution to this problem, I wrote a program to merge the three partial downloads. The resulting file was 99.8% complete after the re-check. Luckily, a painfully slow seed eventually appeared and pushed one of the torrents all the way to 100%.

I've attached the program for anyone who may want to try it. For the people wise enough not to run executables with no credentials here is the virustotal link.

If there is interest for it, I'd like to add it to qbittorrent.

[underlines]

@mimagind what i think is even more often the case and would benefit a lot of qbt users:

• you see a magnet link for a torrent with two files: a 40GB iso file, and a 1kb readme.txt, 2 peers seeding
• you see another magnet link for another torrent with the same 40GB iso file, but no readme.txt file 3 peers seeding

If we can start to download one of the torrents, then select the identical file (the 40GB iso file) out of one torrent, and add more torrents that contain the exact same file (binary exact). It could combine the peers to download that 40GB file, and use all 2+3=5 peers to download faster.

I even imagine a mechanism, that when I add multiple torrents that contain same files, that qbt would ask (after checking that it's binary identical) if you want to merge the binary identical files and combine their peers dynamically.

I thought that's such a common case, that there has to be solutions for this out there. Either I don't search correctly on google, or there isn't anyone doing this yet. All I found was this thread on github.

Does anyone knows if such a use-case has been implemented in any torrent client yet?

[mimagind]

@underlines I'm not sure it's actually such a common use case at all. In the case of well seeded torrents the vast majority of people will just wait a few minutes more for one torrent to complete the download instead of trying to combine multiple torrents for extra speed. Only when the peers do not have all the parts between themselves – which makes it impossible to complete the download from one torrent – does merging become worthwhile.

As far as I know, torrent is designed for tracking packages, not files. This means that there is no way to check anything about individual files and any feature centred on individual files as opposed to packages runs against the flow of everything else. This is probably why such a feature is nowhere to be found in any client.

[gungoksu]

This is an issue I struggle with often. Was hoping to find the answer here. use case: I am into books. usually, these are in archives (couple of GB in size, multiple folders, different writers). What happens is that different archives end up having the same books. I end up downloading them all, eating my very limited bandwidth. current solution: I manually find and disable duplicate directories if I can, or find and copy the already available file into the download folder so qbt finds and does not redownload. (same problem when I do distributed backup over WAN). Better solution: I'd like qbt to have the ability to recognize the CRC of the file in a directory I pre-point, and not redownload it. and/or to recognize the same file in two torrents and download that by merging the trackers/peers (as @underlines pointed out). a problem especially made worse with unattended (RSS) downloads. Vuze (and I thought read maybe deluge did but now cannot find the ref) does the second bit. Unfortunately, my programming skills are next to none. can only help debug. Is this even technically possible? I am aware this is not the same as the subject of this thread; Should I open a separate "feature request" or "wishlist" thread.

[trogmaniac]

sorry for reviving this old post, but i'd like to add my support for this feature request.

My use case is a little different: Sometimes i have a running torrent with no complete sources, and i have another incomplete file from another source, e.g. ftp, xdcc or a damaged CD. And i would like to merge these two files.

Ideally, in the "content" tab, i'd like to right click on any file, select "merge external file", get a file selector to select 1 file, and then this file is checked part-per-part if it matches the file in the torrent, and if yes, that part is added to the file.

I don't know how hard this would be to implement and if qBittorrent wasn't using Qt, i might even try to do it myself. But the amount of external dependencies and the effort to get a working build system is just insane. I gave up after a few hours.

[141]

I've been waiting for merging from two incomplete files forever. Some seeds are superseeding were there aren't enough peers to do so, and then they stop seeding because a peer couldn't share. I use my phone to do most torrent downloading, but when things fail I move them to a computer. The issue is, lets say, I managed to get 76% on the phone, I copied that over, and both managed to download some more in the coming days. Concluding with having 94% on the computer, and 88% on the phone. What I want, is to see if the phone was able to complete pieces the computer did not, and merge all the completed pieces together easily in qbittorrent.

[JiiPee74]

I have faced this low seeding issue now couple times and I have solved it also couple times by doing this:

• Find another torrent with same file size
• Add it to qbittorrent
• Pause original torrent
• Copy original to new source (If you run btrfs, this copy happens instantly, because btrfs do not really copy whole file(s)
• Force recheck new source
• If all goes well you can start downloading new source where you was left with original source

But this does not solve issue where you have same file already downloading as couple of different torrents. Also I have faced issue few times when I have been adding new sources that they have different torrent id, but content is exactly same. qbittorrent does NOT check if folder is already in use and you can end up 2 different torrent trying to download exactly same directory. And I think we all know how that will end up right?

About merge. There is 2 situations: 1) You find many sources of exactly same file but torrent id differs but chunk size is same. 2) You find many sources of exactly same file but torrent id differs and also chunk size is different.

Case 1 would be quite easy to merge. You could just compare filenames/chunk hashes and if they match, merge thouse torrents. Case 2 I think won't have any easy solution for merge because chunk size are different, so there ain't nothing to compare, until you have download something. I have used my copy method successfully for this case also, when I have got lets say 60% downloaded from one source and it runs out of seeds. I have then copy it to other source what use different chunk size, done force recheck and it will pick up parts what is already downloaded.

Anyway most annoying issue for me is that qbittorrent can happily use exactly same folder/file name for multiple torrents. I think there should be a check for that.

[tbertels]

BiglyBT has implemented this: https://github.com/BiglySoftware/BiglyBT/wiki/Swarm-Merging I just tested it with two similar torrents, one of them with three files instead of one: Torrent1: FILE1_name1 Torrent2: FILE1_name2, FILE2, FILE3

BiglyBT merged FILE1_name1 and FILE1_name2 to allow both torrents to complete. FILE2 and FILE3 were small text files.