search

StuffAnThings / qbit_manage

This tool will help manage tedious tasks in qBittorrent and automate them. Tag, categorize, remove Orphaned data, remove unregistered torrents and much much more.
MIT License
721 stars 45 forks source link

[FR]: Ignore torrents with less than x time active #619

Open washedszn opened 1 month ago

washedszn commented 1 month ago

Is your feature request related to a problem? Please elaborate.

Not necessarily a qbitmanage problem, it runs as it should. However, in scenarios where external programs like cross-seed are adding torrents to qbittorrent, there's a small window where if both are running at the same time, the rem_orphaned job will remove torrents where the data hasn't finished linking - this happened to a user here, which resulted in those cross seeds starting to redownload.

Describe the solution you'd like

Implementing an ignore_age (or some other naming, unsure what's clearest) - which could be global or specificaly for orphaned data config. Trying to think if this could help protect against other edgecases, maybe in cases where force_auto_tmm is used?

Does your solution involve any of the following?

Describe alternatives you've considered

An alternative would be to ensure external programs do not run at the same time as qbitmanage, which would reduce the chances of this case happening - however this wouldn't fully prevent it as cross-seed can use announces from rss feeds.

Who will this benefit?

Can see this benefitting all cross-seed users and potentially others that use external programs to inject torrents into qbittorrent.

Additional Information

No response

NLZ commented 1 month ago

Here are the logs for easier access when qbitman and cross-seed collided: https://logs.notifiarr.com/?33765dd632341dfb#CqofPuN6hLd66FQ5Aw6cxdrmwb6DdweLBmYFwujSiaUq

Sadly hardlinks have a single, unified Access/Modify/Change timestamps for stats, so that cannot be used for pre-checks. Maybe the directory timestamp could be used.

Or some other improvement to the code to reduce the window of collision. Does qbitman checks the file links first, or the torrents' files?

zakkarry commented 1 month ago

Just to be clear that I understand this properly...

AFTER qbm did its indexing of the torrent list, cross-seed linked a torrent's structure that was found in the orphaned data since qbm had no awareness of this torrent in its list of active torrents?

NLZ commented 1 month ago

cross-seed linked and added a torrent, but then qbm decided that the linked files doesn't belong to a torrent and moved it to orphaned.

I also see this in the qbit logs:

(N) 2024-07-26T12:58:04 - Added new torrent. Torrent: "Zootopia+.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264-NTb"
(W) 2024-07-26T13:18:02 - File error alert. Torrent: "Zootopia+.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264-NTb". File: "/data/torrents/cross-seed-link/LST/Zootopia+.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264-NTb/Zootopia+.S01E02.The.Real.Rodents.of.a.Little.Rodentia.1080p.DSNP.WEB-DL.DDP5.1.H.264-NTb.mkv". Reason: "Zootopia+.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264-NTb file_open (/data/torrents/cross-seed-link/LST/Zootopia+.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264-NTb/Zootopia+.S01E02.The.Real.Rodents.of.a.Little.Rodentia.1080p.DSNP.WEB-DL.DDP5.1.H.264-NTb.mkv) error: No such file or directory"
(N) 2024-07-26T13:21:42 - Torrent download finished. Torrent: "Zootopia+.S01.1080p.DSNP.WEB-DL.DDP5.1.H.264-NTb"

So the timeline was something like this:

zakkarry commented 1 month ago

I think whether the 3rd and 4th steps happened in that order is kind of irrelevant...

One solution I would just float for consideration would be that, at least in the orphaned data - since it has rather big significance deleting files - perhaps waits to perform the orphaned actions until the end of processing in a batch, and then reindexes the torrent list to make sure nothing has changed DURING the orphaned job run (or before it AFTER the indexing, whenever that is) - and double checking the torrent list against the "updated" list before acting on any data deemed orphaned.

This avoids the need for an additional (and mis-configurable) option, while more thoroughly addressing potential pitfalls.

I'm not sure if other jobs qbitm runs could benefit from this approach because of race conditions, but it seems like something to consider. Especially given that pulling a torrent list is negligible for automated background processes.

NLZ commented 1 month ago

Looking at the code qbm seems to be checking the disk first, then it gets a fresh list of torrents for their files:

https://github.com/StuffAnThings/qbit_manage/blob/9f08404b4c745dac0f1f53afc87b938342c7e1f9/modules/core/remove_orphaned.py#L34-L48

So all I can think of is that it went something like:

  1. cross-seed links files
  2. qbm gets files
  3. qbitman gets torrents
  4. cross-seed inserts torrent
zakkarry commented 1 month ago

That's sort of crazy. Thanks for digging into the code, it does appear @bobokun is already doing this. The thing is, that window has to be absolutely microscopic given that it's only AT THE MOST as large as the time between when we start linking and when we inject the torrent. In a normal scenario, I'd say it would be less than a second.

I guess you could always recheck it twice (seems like the wrong approach), after the exclude patterns or something before acting on it, but I'm not even sure this is likely to occur at all regularly. Bobo will have more of an idea than me on this.

You have to be matching and injecting a torrent in a ~1-second window WHILE qbitmanage is running and indexing its orphaned files.

Let me know when we're going to Vegas.

NLZ commented 1 month ago

Yeah, it seems like one in a million chance to happen.

I'm not sure if it would worth the effort and performance hit to check the os.stat(folderpath).st_mtime or something to prevent it. I wouldn't be bothered if nothing would get chanced, unless it keeps happens to others as well.