Feramance / qBitrr

A simple Python script to talk to qBittorrent and Arr's
MIT License
61 stars 2 forks source link

[REQUEST] Option to keep stalled torrents. #84

Open devanteweary opened 1 week ago

devanteweary commented 1 week ago

Hey, touching on my questions from earlier, thought I'd make some suggestions here. I don't know if you like 1 post per topic. Sorry if it seems like I'm spamming!

So this suggestion is:

Sort of like there's an option to never delete seeding torrents, have the same option for torrents when searching for new ones to replace.

It might look like this:

[Radarr-1080.Torrent]
AutoDelete = false
IgnoreTorrentsYoungerThan = 86400
MaximumETA = -1
MaximumDeletablePercentage = 0.75
DoNotRemoveSlow = true

KeepStalled = true

And the behavior would be:

This way, the old one has a chance to complete one day, but you still have qBitrr looking for new ones.

Some uses cases... sometimes there will be a torrent that has stalled for a week or two, but by the grace of the torrent gods, someone will connect with the rest.

I see there's the qbitrr-ignored tag, but that's not really the same thing I believe.
That just causes qBitrr to ignore that one being stalled if I'm not mistaken.

This would be super helpful for torrent from private trackers that are stalled. If one is stalled, I don't want it delete because it'll probably give me a hit and run. But still want qBitrr to look for a different release.

No delete. Just add! :P'

OK thank you!

Feramance commented 1 week ago

This is doable, shouldn't be terrible to implement. At first I'll implement this using a tag, proceeding from there I'll make the changes for the no tagging version.

This change will also need to adjust the search behaviour as currently qBitrr only searches for items that aren't in the queue for Radarr/Sonarr

devanteweary commented 1 week ago

Dang dude you're awesome. Check your Patreon messages, by the way.

Feramance commented 1 week ago

This specific issue should be resolved currently in docker tag pr-86. Give it a try and let me know how it goes

devanteweary commented 1 week ago

Great and thank you.

I tested it out with each of the following:

and

Looks like it's just ignoring the new settings outright. It's still going in and removing the stalled torrent and re-searching for a new one.

According to the logs, it's qBitrr v4.7.5-7c72ac4 Although I think that version number is the same as it was yesterday.

When I have it create the config.rename_me, it does in fact have the new AllowedStalled and StalledDelay fields.

Hope this helps!

Feramance commented 1 week ago

The version number for now will remain the same, only the hash will change, until the change is completed. Can you set logs to trace and send over a log?

Feramance commented 1 week ago

If not tonight, tomorrow, I'll be doing some testing myself too

devanteweary commented 1 week ago

Whoops shoulda had that for you already.

Here are the logs: qBitrr Logs Also included my config.toml in there.

The one I'm looking at is Lionheart.
I know there are no good seeds out there so it's always gonna get to around 90% and then stall, so it's a good test.


By the way, the reason I can keep reusing Lionheart is because even though qBitrr keeps adding it to the blocklist and multiple entries at that, it seems like Radarr is ignoring the blocklist and re-adding it to qbittorrent everytime. Funny thing is in an interactive search, it DOES show the blocklisted icon. Was gonna bring it up later as an issue but didn't want to complicate things. Just mentioning right now in case you were wondering how I'm using the same torrent test over and over.

image

Feramance commented 1 week ago

The reason for grabbing the same torrent is due to use of multiple trackers which would have the same torrent. For Radarr/Sonarr each torrent on each tracker is different, so this happens a lot

Feramance commented 1 week ago

Also, let the trace logs run for 15-30 mins before sending them over. There's no info to work with in the logs you sent me

devanteweary commented 1 week ago

Also, let the trace logs run for 15-30 mins before sending them over. There's no info to work with in the logs you sent me

OK, I let it run for about an hour: qBitrr Logs

SHould show it replacing Lionheart every 30 minutes.

Feramance commented 1 week ago

I went over everything and it just dawned on me. Can you test again tagging the torrent with the new tag qBitrr-allowed_stalled? Once it's working with the tag I can get to completing work on the tagless update

Feramance commented 1 week ago

Made some additional changes, kindly update your container and test again please

Feramance commented 1 week ago

If you can send me the magnet link to the Lionheart torrent so I can test please

devanteweary commented 1 week ago

If you can send me the magnet link to the Lionheart torrent so I can test please

OK all done. Same link: qBitrr Logs Magnet is in there too.

Hope it helps!

Oh and if you want a quicker response, I'm DevanteWeary on Discord.

Feramance commented 1 week ago

I noticed an issue in the logs you sent, I'll fix it and hopefully the logic will be functional

Feramance commented 1 week ago

Update to the latest version, I think it is now resolved and functional. Once confirmed I'll push the official update and continue work on the tagless update

Feramance commented 1 week ago

Though remind me, you wanted a search to still occur if the torrent is stalled right?

devanteweary commented 6 days ago

Though remind me, you wanted a search to still occur if the torrent is stalled right?

OK. I did the same update. Forced updated the pr-86 tag.
Unfortunately it still is having issues.

This time I tried three scenarios so there are three sets of logs: qBitrr Logs I also threw another movie in the mix - Anchorman. What I did was add a torrent I knew would be stalled for it. (no seeders).

Here's what happened:

logs_minus_one_no_tag: StalledDelay set to -1. Did not assign tag.

logs_zero_no_tag: StalledDelay set to 0. Did not assign tag.

logs_zero_using_tags: StalledDelay set to 0. Manually assigned qBitrr-allowed_stalled tag.

Also, with Anchorman, I noticed it will remove the stalled torrent, add it to blocklist, and re-add a new torrent that has a bunch of seeders and actively downloads (desired behavior). Then if I pause it, it'll get removed on the next loops (which I set to 5 minutes).

One more thing, each time I tested, I removed all the blocklist entries as well. Oh and I let them run for a while. At least 30 minutes each time I think.

OK that's it for now!

devanteweary commented 6 days ago

I just had another movie downloading, had plenty of seeds and was actively downloading.

Just on a hunch, I ran qBitrr and sure enough, it removed that torrent, blocklisted it, and re-searched it.

And I forgot to answer you...

Yeah what I was looking for originally is for it to go ahead and blocklist and re-search stalled torrents.
Just not remove them from qbittorrent. Allowing the stalled torrent and newly searched replacement torrent to run side by side.

Feramance commented 6 days ago

Noted, have you tried with a price value for a delay? Say 60 (minutes)?

Feramance commented 6 days ago

Otherwise I think I can have this ready by Sunday at the most (maybe Monday)

devanteweary commented 6 days ago

Noted, have you tried with a price value for a delay? Say 60 (minutes)?

I just tried with 3600 (1 hour) and it just did the same thing. Removed torrent (ignoring the `StalledDelay = 3600'), blocklist, re-search (adding the same exact torrent, ignoring Radarr's blocklist).

The odd thing about the blocklist situation is it seems like this is the only movie it's ignoring the blocklist for. I can tell it's re-adding the same one because it's the same hash that it's blocking.

I'll do some more testing on that front, but seems like a Radarr issue.

If you ever want to remote into my server and see what it's doing in realtime, just let me know here or Discord (DevanteWeary).

Feramance commented 6 days ago

Noted, I've made some changes and added more logs. I'm also testing this functionality with the Lionheart torrent you provided and things seem to be working correctly now. Please update and let me know, trace logs are now clearer on what the new functionality is doing

Feramance commented 6 days ago

Also the delay is in minutes, so 3600 would be equal to 60 hours. I've updated the config description

devanteweary commented 5 days ago

Noted, I've made some changes and added more logs. I'm also testing this functionality with the Lionheart torrent you provided and things seem to be working correctly now. Please update and let me know, trace logs are now clearer on what the new functionality is doing

OK updated the pr_86 again and still didn't work.

Also note how the last time Lionheart we removed and re-searched, it was at 54% so was actively downloading. It stalls at about 94ish%.

Oh and I changed the StalledDelay to 60.

devanteweary commented 5 days ago

I also tested out some other movies: Bad Moms and Wayne's World.

Then re-searched those as well. And finally, I tagged them in this test.

More qBitrr Logs

image

So to sum up, the current issues are:

That's it!

Feramance commented 5 days ago

Ok so going through your logs I can deduce the following:

The stalled delay is working, I wasn't clear on how it works. It checks for the time delay from the time the torrent was added and when looking at your logs, the ones removed had been added more than 60 minutes prior to the run, so that logic is functional.

What's definitely wrong is the processing of paused torrents, that I'll need to adjust for sure

devanteweary commented 5 days ago

That's what I understood. In other words, if a torrent is only 57 minutes old, it will not be removed/replaced IF stalledDelay is set to 60.

I haven't looked at those logs in particular but I can tell you it's ignoring the 60 minutes rule for sure. For example, it's just replacing the Lionheart every loop (5 minutes - ultimately I'll set it to 60 minutes).

And remember, it was ignoring the setting even when I had it set to 60 hours too.

Sorry man I know you've been working hard on it. And I'm always open to you remoting in and taking a look for yourself firsthand.

I did one quick test just now. I had Lionheart in there since 9pm last night. I just now turned on qBitrr and it immediately removed Lionheart then re-searched and added it back. Then after the 5 minute loop, it removed and re-searched Lionheart again. And I noticed Lionheart wasn't even stalled yet. It was still actively downloading.

Here's a screenshot of a second or two before it removed Lionheart, and then a screenshot of the log right after.

Right before: image

And right after: image

And of course the actual logs.
This is a brand new set of logs cleared before the test: qBitrr Logs - Quick Test

Feramance commented 5 days ago

Ok, think I know what's going on, as soon as I get the chance I'll make some adjustments

Feramance commented 5 days ago

I just had a look, and I think you need to check your localisation (time and date). Ensure that both qbittorrent and qBitrr have the same time/timezone applied as per your logs the delay is functional it's just that your qBitrr insurance thinks it's 7 hours ahead of your qbittorrent instance. That or set the delay to 480 to account for the difference:

Stalled check: Lionheart.1990.1080p.BluRay.x265.HEVC.10bit.2ch.(xxxpav69) [Current:2024-06-30 04:12:36.255702][Added:2024-06-29 21:44:57][Limit:2024-06-29 22:44:57]

I also pushed an update which should fix the processing of paused, uploading torrents too

devanteweary commented 4 days ago

OK got some results... Completed uninstalled / reinstalled qBitrr. Reused my config.

Ensured the time was correct by 1) adding the TZ variable to the container settings and 2) running the date command in the CLI of Unraid, qbittorrent, and qBitrr to make sure they match.

1st run - 60 delay setting - qBitrr Logs

2nd run - 480 delay setting just to test - qBitrr Logs - 480 delay

By the way, are we sure there's nothing else in my config that could be causing it?

Feramance commented 4 days ago

Took a look at your config file and logs and noticed a pattern with the File extension allow list. Try it this way and let me know if anything is different: config.zip

Feramance commented 3 days ago

I've made adjustments which should allow an empty File extension allow list and upon further testing, on my end the logic is working flawlessly. I'll wait till I get feedback from your end before releasing though

Feramance commented 3 days ago

So a quick update, just added logic for re-searching, untested yet and some config changes as seen below: image

devanteweary commented 3 days ago

OK done testing! Seems to be doing a lot better on my side.

After testing it out, it's almost perfect. The only thing I see now is that it is NOT adding to blocklist unless it removes the stalled torrent. It still searches, just doesn't block.

Check out the Delay settings part below to see what I mean.

Also, since you made some adjustments after posting your config, I didn't use it so let me know if you still want me to.


Logs (in order of testing)


Notes

Delay settings:


I also thought of something not related to the above that might complicate things... sorry!

Let's consider stalled torrents left in qBittorrent.

1st pass: It detects stalled torrent #1 and blocklists and re-searches. This search adds torrent #2. Torrent #1 is kept in qBittorrent like we want. 2nd pass: It now detects stalled torrent #1 and completed torrent #2.

Will it see torrent #1 as stalled again? And perform another blocklist and re-search? And just forever block and re-search on every looped as long as the stalled torrent is in there? I'm thinking some kind of list/db that says "I already processed this stalled one, ignore it from now on."

Or did I just type that for nothing and that's what the "Already been cleaned up" entry in the log means? ha

Feramance commented 3 days ago

Excellent news overall, glad to hear. To address your concerns:

Hope this clears things up a bit

devanteweary commented 3 days ago
  • To be clear, the functionality you're looking for is for the lionheart to be removed from the Radarr queue and block listed but not removed from qbittorrent as soon as it stalls, then performs a search as usual?

Yep you got it!

  • Could you specify what reason you have for pausing torrents? As overall all the way qBitrr works, it only pauses torrents if there isn't enough free space or if the torrent has reached it's seeding/ratio limit and it's not set to remove, otherwise it tries to keep torrents active. Maybe this could be another feature?

Oh man, there are various reasons. I'll give you an example of what I have currently. I have 5 torrents for Looney Tunes that vary in sizes from 150GB to 500GB. It was easy for me to add them as I was searching and then pause them and try one at a time so I didn't download all that space unnecessarily. Kind of like having them as a backup in case the others don't work.

I have another one where I added the torrent, then found another torrent of better quality but it's very slow (taking a week so far) with low seeders. So I'm keeping the first torrent paused in case the second one loses all its seeders and never comes back.

Hope this clears things up a bit

Let me just say again that you're awesome man.
Thank you for working on everything so hard.

Feramance commented 3 days ago

Yep you got it!\n\nDo NOT removed stalled torrent from qBittorrent. Keep it forever.\nAdd stalled torrent to blocklist in Radarr.\nRe-search for new torrent.

Ok I can fix this, I actually added logic to not cause a blocklist until the delay is finished thinking it would be preferred.

Oh man, there are various reasons.\nI'll give you an example of what I have currently.\nI have 5 torrents for Looney Tunes that vary in sizes from 150GB to 500GB.\nIt was easy for me to add them as I was searching and then pause them and try one at a time so I didn't download all that space unnecessarily. Kind of like having them as a backup in case the others don't work.\n\nI have another one where I added the torrent, then found another torrent of better quality but it's very slow (taking a week so far) with low seeders.\nSo I'm keeping the first torrent paused in case the second one loses all its seeders and never comes back.

This seems to me to be very manual and there's no real way to handle these outside of tagging them to be ignored using the tag qBitrr-ignored

Let me just say again that you're awesome man.\nThank you for working on everything so hard.

You're very welcome, thank you for the support and it's a pleasure seeing this tool and any changes I add to it being used

devanteweary commented 3 days ago

Oh man, there are various reasons.\nI'll give you an example of what I have currently.\nI have 5 torrents for Looney Tunes that vary in sizes from 150GB to 500GB.\nIt was easy for me to add them as I was searching and then pause them and try one at a time so I didn't download all that space unnecessarily. Kind of like having them as a backup in case the others don't work.\n\nI have another one where I added the torrent, then found another torrent of better quality but it's very slow (taking a week so far) with low seeders.\nSo I'm keeping the first torrent paused in case the second one loses all its seeders and never comes back.

This seems to me to be very manual and there's no real way to handle these outside of tagging them to be ignored using the tag qBitrr-ignored

Do you think something like a general PauseWhenLow = true and/or PauseResume = false would work?
If being honest, I don't think qBitrr needs to be starting, stopping, resuming, or even checking for free space.
There are so many other spots that check for free space.

For instance, Radarr/Sonarr checks for free space and most server OSes (like Unraid in my case) checks for free space. CheckFreeSpace

Just throwing that out there!

Feramance commented 3 days ago

PauseWhenLow how exactly? What parameters, that's the real issue. And the whole point of qBitrr is automating torrent management for Radarr/Sonarr.

Also although the Arrs do have logic in place for free space, they only account for importing not downloading and in my case for example, I have my downloads going to an SSD for faster downloads and general processing before the files are moved to HDDs for plex to read off of, which means that the Arr apps have no idea what storage capacity exists for the downloads folder.

I could add an option to disallow pausing and resuming, rather simple to do actually, but you will lose a large part (in my opinion) of the overall functionality that qBitrr provides.

If PauseResume = false is something you'd be interested in, let me know and I'll get it sorted, got no problem with that

devanteweary commented 3 days ago

Also although the Arrs do have logic in place for free space, they only account for importing not downloading and in my case for example, I have my downloads going to an SSD for faster downloads and general processing before the files are moved to HDDs for plex to read off of, which means that the Arr apps have no idea what storage capacity exists for the downloads folder.

Actually yeah this makes sense to me.

I could add an option to disallow pausing and resuming, rather simple to do actually, but you will lose a large part (in my opinion) of the overall functionality that qBitrr provides.

When you say large part, what are we talking here?

If PauseResume = false is something you'd be interested in, let me know and I'll get it sorted, got no problem with that

Yeah man that'd be great. I personally am not worried about any kind of space issues but can see how others would be for sure.

Feramance commented 3 days ago
When you say large part, what are we talking here?

Mainly to do with ensuring seeding as configured, and overall automation, I guess it's larger for me and some others than it may be for you to be honest.

Yeah man that'd be great. I personally am not worried about any kind of space issues but can see how others would be for sure.

No worries, I'll get these changes sorted tomorrow I think

Feramance commented 3 days ago

I have updated with the agreed changes. The new config is under the top most Settings section as seen below: image

Let me know if things are working as intended

devanteweary commented 1 day ago

I have updated with the agreed changes. The new config is under the top most Settings section as seen below: image

Let me know if things are working as intended

Hey sorry was busy all day yesterday and today. I got to check it out just a little bit today and long story short, looks like it's back to removing stalled torrents. But let me get home tonight and give it a full test and get back to you!

Feramance commented 1 day ago

That's ok, might I suggest enabling and disabling the re-search functionality to see if the removing of the torrents is due to that? On my end I think that's what's causing it

Feramance commented 6 hours ago

I'm going to go ahead and release the changes as is, but I'm not closing this issue quite yet till I see the re-searching is working as intended

devanteweary commented 6 hours ago

I'm going to go ahead and release the changes as is, but I'm not closing this issue quite yet till I see the re-searching is working as intended

Yeah sorry about that. Was super busy yesterday and then today my server was giving me all kinds of problems and been working on that most of the day. Tomorrow morning I'll 100% be able to test out those changes and report back!

Feramance commented 6 hours ago

No worries, update to the latest release (4.8.0) first