Auto backup and restore system

[ghost]

Please provide the following information

qBittorrent version and Operating System

4.2.5 win10

What is the problem

Since sometimes torrents disappear out of nowhere, there should be a way to automatically backup the fastresume and configuration files. In the event of fastresume corruption user can easily restore from the backup. In BiglyBt client there's a easy backup and restore process. Maybe something like that could be added in qBt as well.

[FranciscoPombal]

Since sometimes torrents disappear out of nowhere,

Do you mean that the .fastresumes disappear (as in, get deleted)? Or the torrent just disappears from the UI, but the .fastresume is still there? And can you give examples of such a thing happening in recent versions (link issue, or describe situation, please).

[ghost]

Sometimes fastresume gets corrupted. And there are countless issues if you search. Also not everything makes its way into the issue tracker. I've seen many users face this torrent disappearing issue which is most likely due to fastresume corruption. And the incidents increased since recent updates. https://github.com/qbittorrent/qBittorrent/issues/12642

[FranciscoPombal]

@an0n666 In general, are you saying 1 (or few) corrupt .fastresumes cause everything else to get corrupted/disappear, or is everything getting corrupted/disappearing, or are only those few disappearing, but leaving everything else ok? Because these 3 are quite different situations. Has it happened to you, and if so, have you attempted to debug it?

Also, it would be helpful if you could link more examples. The issue you linked is not much to go on (could have happened due to external factors). The other recent examples I know of "torrents disappearing from the UI" are reports from users with corrupt tags that only seem to affect the WebUI. I performed a cursory search and did not find anything else recent that's related.

As for:

not everything makes its way into the issue tracker.

Can you link other sources where people are reporting this? If they are private trackers, just drop the acronym, I'll check the respective forum.

[DarkVoyage]

As long as you, @FranciscoPombal , closed this #4850 incorrectly calling it "not an issue with data loss" — this is exactly issue with data loss, because "data" is not only what is downloaded, this is all data regarding torrent exchanging process. As you know torrent community lives by the help of torrent keepers, who seed for many years. Torrents should never disappear without trace from the list. So I will just repeat my message below from the closed thread here, because it is absolutely actual.

I had 13000+ torrents. Lost about 2000 due to "no disk space". And then after each crash or restart lost more and more. And finally there were only about 6300 left. Still didn't finish the restoration process, because had no spare days...

I have some temporary recommendations to seeders based on bad experience and lost days of work. This will help you get more adequate behavior of qB, if you have thousands of torrents.

1. Stop all torrents before shutdown/restart. Wait as much as needed, 5 minutes. Then QB will shutdown in several seconds instead of several minutes, because if you shutdown the actively working client it is left in background saving fastresume files for changed jobs. It is still fine in case you also follow "2". Stopping the job saves it and it will not be lost or damaged in most of the cases. You may backup manually at this stage, then start torrents again and stop again. They will be saved once more (if changed).
2. Disable periodic saving of resume data - first of all this is pointless, if you are seeding. If job changes it will save anyway, periodic saving just updates active fastresume files cyclically with same data. If you have thousands of torrents you will end up in endless loop of saving, QB saves them by about 10 files each second, so when the queue ends - it already starts from the beginning. If you shutdown client with active torrents - it will never shutdown and will continue to work in background forever. My copy was working for a day after shutdown, until I killed it.

For qB developers:

1. (more interesting) There should be backup system. Fastresume files are small and you can save several backups of them (compressed). Fastresume files doesn't disappear physically and this is great, but some of them are silently ignored during restart if client, if they are corrupted. This is not correct. There's should be a possibility to always "know" the integrity of them. I mean you can make a check of integrity at start and then show dialog with information of health. Then in case of damaged fastresume files let user decide, what to do - restore from backup, remove, save the list to txt or anything else.
2. (same as uTorrent) Working fastresume files with damaged torrent files may still present in torrent list as a "broken job". Then you know, which torrents should be fixed. Otherwise it is very difficult to restore them as all files in backup folder are named with hash.
3. (also) There might be a feature to clean the data folder from broken/lost/forgotten/useless files. It is quite tricky to clean that folder manually. This will also resolve known problem with reappearing folders and subfolders in categories because of the leftover fastresume files for removed jobs.

[FranciscoPombal]

@DarkVoyage For me data loss means payload data loss. .fastresumes are not payload data, they are metadata. Perhaps there should be a label for that. I'm not denying that losing many .fastresumes can be a real hassle, but it's very different than qBittorrent eating your files. I think the distinction is important.

In your post you do raise some valid points. The ones that stand out to me, if I understand correctly, are the following:

• regarding the issue of shutdown/restart, you don't need to stop all jobs before exiting qBittorret, you just need to make sure qBittorrent has exited cleanly before you shutdown your system. Remember that on most systems, including Windows, shutdown has a timer after which it will just kill the application, which in the case of qBittorrent this most likely means interrupting the .fastresume saving.
• qBittorrent is apparently very slow/inefficient in saving .fastresume data, to the point of making the periodic save of .fastresume feature a liability when there are many torrents.
• I agree that broken fastresumes should not be silently ignored or discarded, and the user should be prompted to take action in the case that some kind of corruption was detected and qBittorrent was not able to fix it on its own
• If the previous point is satisfied, then the "cleaning" feature is not actually needed, since qBittorrent would just show the leftover jobs in the UI as "broken jobs".

[DarkVoyage]

I can't say it is a big problem that qB saves .fastresumes slowly, but that option offers saving those files in cycles no matter if they are changed or not. At least, when I disabled it, qB exits quite fast even with thousands of torrents. If you pause them and let him save the data for some minutes then you can exit right now and it will not be hanging as a background process at all.

Sure, I agree that there is enough to have at least some information system of broken jobs.

A little pity, when you compare to stability of uTorrent, which you can kill at any point of time and it will just continue where it left in most cases. But I find qBittorrent having a good response, when you have 10000 torrents. In fact I don't see any lags at all, it could have 100k without a problem. uTorrent over some thousand torrents is very hard to work with.

My current idea is to backup whole data folder between sessions and do it regularly. At least I can always return to some point in time and continue from there.

[FranciscoPombal]

The thing about this idea of a "backup and restore system" is that it is just a crutch. I'd much rather fix the underlying issues.

If the fastresume saving system is made more robust, it would probably be able to withstand anything short of a process kill (as in SIGKILL) or sudden power loss in the middle of writing a file. Even then, those situations can be dealt with some further improvements to robustness.

[DarkVoyage]

Don't forget "no empty space". It leads to writing 0-size fastresumes and complete loss of resume data. Should be simple to just stop writing them, if space is less than 100 MB.

[Snake883]

I'd like the ability to backup the QBT metadata.

Torrent name Folders names Files names Hash Tags Location/Save Path Trackers Category

Justifications: I've lost torrent metadata information. I may wish to transfer QBT to another machine. Backups are always important.

[DarkVoyage]

I see that there are some reports #11520 #13547 #13555 regarding loading of broken jobs. Should I understand that this is the result of realization of some features, that we requested here? Some people doesn't like to see broken jobs, they are ok, if they just disappear without a trace... :(