Closed HomebrewDotNET closed 2 months ago
Could you please 1) provide a screenshot of qbit 2) switch go to sonarr or radarr and navigate to the queue menu. Now activate your browsers network messages tab (f12 usually does that for you), refresh the page, and look for ‚queue‘. Paste the response to a pastebin. (Preferrably you paste the queue response for both radarr/sonarr separately) 3) can you clarify if this is only needed for so arr/radarr or for lidarr/readarr too?
Thx
yes agree on (1). Will build smth for sonarr/radarr; would appreciate if you could then help me with testing / potential debug.
thx for the logs, those are helpful
Update - just pushed a new version to "dev" - can you please pull the package and test if it does the trick for you?
Thanks for the quick changes. Pulled the last version and I now see this: False | Removing downloads that fail on import (no format upgrade) Which env variable do I need to enable it?
See dev-Readme: REMOVE_NO_FORMAT_UPGRADE=True
Sorry missed that readme. Managed to enable it:
[INFO]: *** Current Settings ***
[INFO]: Version: dev
[INFO]: Commit: 637b901
[INFO]:
[INFO]: True | Removing failed downloads
[INFO]: True | Removing downloads missing metadata
[INFO]: True | Removing downloads missing files
[INFO]: True | Removing downloads that fail on import (no format upgrade)
[INFO]: False | Removing orphan downloads
[INFO]: True | Removing slow downloads
[INFO]: True | Removing stalled downloads
[INFO]: True | Removing downloads belonging to unmonitored items
[INFO]:
[INFO]: Running every: 0 days 0 hours 1.0 minutes
[INFO]: Minimum speed enforced: 1 KB/s
[INFO]: Permitted number of times before stalled/missing metadata/slow downloads are removed: 60
[INFO]: Downloads with this tag will be skipped: "~type_PrivateTracker"
[INFO]: Private Trackers will be skipped: True
[INFO]:
[INFO]: *** Configured Instances ***
[INFO]: Radarr: http://127.0.0.1:7878/api/v3
[INFO]: Sonarr: http://127.0.0.1:8989/api/v3
[INFO]: Lidarr: http://127.0.0.1:8686/api/v1
[INFO]: qBittorrent: http://127.0.0.1:6882/api/v2
[INFO]:
[INFO]: *** Check Instances ***
[INFO]: OK | Radarr
[INFO]: OK | Sonarr
[INFO]: OK | Lidarr
[INFO]: OK | qBittorrent
[INFO]:
[INFO]: ##################################################
[INFO]: LOG_LEVEL = INFO: Only logging changes (switch to VERBOSE for more info)
[INFO]: >>> Detected no format upgrade download (1 out of 60 permitted times): Tropa.de.Elite.2007.1080p.BluRay.DTS.x264-ESiR
[INFO]: >>> Detected stalled download (1 out of 60 permitted times): Black.Swan.2010.Bluray.1080p.DTS-HD.x264-Grym
[INFO]: >>> Detected stalled download (1 out of 60 permitted times): The.Tomorrow.War.2021.2160p.AMZN.WEB-DL.DDP5.1.HDR.HEVC-CMRG[TGx
[INFO]: >>> Detected no format upgrade download (1 out of 60 permitted times): The.Mandalorian.S03E01.2160p.DSNP.WEB-DL.DDP5.1.HDR.H.265-NTb[eztv.re].mkv[eztvx.to]
[INFO]: >>> Detected no format upgrade download (1 out of 60 permitted times): The.Mandalorian.S03E02.Chapter.18.The.Mines.of.Mandalore.2160p.DSNP.WEB-DL.DDP5.1.HDR.H.265-NTb[eztv.re].mkv[eztvx.to]
I notice though that it skips entries for private trackers with IGNORE_PRIVATE_TRACKERS=True set. Would it be possible to remove these items from the queue but not delete the torrent itself? Either using Removal Method 'Ignore download' or 'Change category' like defined in Radarr. Thanks for the changes!
Not sure i know what you mean by this, could you pls explain/potentially with screenshots?
Either using Removal Method 'Ignore download' or 'Change category' like defined in Radarr.
If you manually try and delete items in Radarr and Sonnarr you get these options: https://imgur.com/a/YiLcG6Y The bottom 2 shouldn't give issues with private trackers
If you chose ‚ignore download‘, will it then again start fetching better quality downloads if available (ie queue gets unblocked)?
can you chose ‚ignore‘ on one of them and paste the network delete request URL (queue api request)?
I don't think it will as the download descision maker should block it.
The url for the request is
http://192.168.1.11:7878/api/v3/queue/bulk?removeFromClient=false&blocklist=false&skipRedownload=false&changeCategory=false
The body is:
{
"ids": [
701387199
]
}
So did some testing and I noticed that sometimes they get undetected:
2024-04-08T12:07:03.825423883Z [INFO]: >>> Detected no format upgrade download (49 out of 60 permitted times): Tenet.2020.UHD.BluRay.2160p.DTS-HD.MA.5.1.DV.HEVC.HYBRID.REMUX-FraMeSToR
2024-04-08T12:07:05.408512839Z [INFO]: >>> Download no longer marked as no format upgrade: The.Mandalorian.S03E01.2160p.DSNP.WEB-DL.DDP5.1.HDR.H.265-NTb[eztv.re].mkv[eztvx.to]
2024-04-08T12:08:09.388751011Z [INFO]: >>> Detected no format upgrade download (50 out of 60 permitted times): Tenet.2020.UHD.BluRay.2160p.DTS-HD.MA.5.1.DV.HEVC.HYBRID.REMUX-FraMeSToR
2024-04-08T12:08:11.551032665Z [INFO]: >>> Detected no format upgrade download (1 out of 60 permitted times): The.Mandalorian.S03E01.2160p.DSNP.WEB-DL.DDP5.1.HDR.H.265-NTb[eztv.re].mkv[eztvx.to]
I get "Download no longer marked as no format upgrade" but not really sure as to why
I just added the functionality that for private trackers, the queue items get removed, but not the associated torrent files. Can you please test?
In the URL you posted, (and as per your comment) the parameter blocklist=false was set; that in my opinion should be true, else the *arr app will potentially grab that download just once again. Should not be a big deal if that lesser quality torrent is added to blocklist though since you have a better version already anyways?
So did some testing and I noticed that sometimes they get undetected:
I turned of the check for x/n times (immediate removal now)
So finally managed to test again.
Firstly I agree on the block thing. Doesn't really matter if we block it because we already have a better version.
I did notice however it seems to have stopped reporting on the stuck torrents:
##################################################
[INFO]: Decluttarr - Application Started!
[INFO]:
[INFO]: *** Current Settings ***
[INFO]: Version: dev
[INFO]: Commit: 3d33e88
[INFO]:
[INFO]: True | Removing failed downloads
[INFO]: True | Removing downloads missing metadata
[INFO]: True | Removing downloads missing files
[INFO]: True | Removing downloads that fail on import (no format upgrade)
[INFO]: False | Removing orphan downloads
[INFO]: True | Removing slow downloads
[INFO]: True | Removing stalled downloads
[INFO]: True | Removing downloads belonging to unmonitored items
[INFO]:
[INFO]: Running every: 0 days 0 hours 1.0 minutes
[INFO]: Minimum speed enforced: 1 KB/s
[INFO]: Permitted number of times before stalled/missing metadata/slow downloads are removed: 60
[INFO]: Downloads with this tag will be skipped: "~type_PrivateTracker"
[INFO]: Private Trackers will be skipped: True
[INFO]:
[INFO]: *** Configured Instances ***
[INFO]: Radarr: http://127.0.0.1:7878/api/v3
[INFO]: Sonarr: http://127.0.0.1:8989/api/v3
[INFO]: Lidarr: http://127.0.0.1:8686/api/v1
[INFO]: qBittorrent: http://127.0.0.1:6882/api/v2
[INFO]:
[INFO]: *** Check Instances ***
[INFO]: OK | Radarr
[INFO]: OK | Sonarr
[INFO]: OK | Lidarr
[INFO]: OK | qBittorrent
[INFO]:
[INFO]: ##################################################
[INFO]: LOG_LEVEL = INFO: Only logging changes (switch to VERBOSE for more info)
[INFO]: >>> Detected stalled download (1 out of 60 permitted times): Oceans.Eight.2018.2160p.BluRay.x265.10bit.SDR.DTS-HD.MA.TrueHD.7
[INFO]: >>> Detected stalled download (2 out of 60 permitted times): Oceans.Eight.2018.2160p.BluRay.x265.10bit.SDR.DTS-HD.MA.TrueHD.7
[INFO]: >>> Detected stalled download (3 out of 60 permitted times): Oceans.Eight.2018.2160p.BluRay.x265.10bit.SDR.DTS-HD.MA.TrueHD.7
[INFO]: >>> Detected stalled download (4 out of 60 permitted times): Oceans.Eight.2018.2160p.BluRay.x265.10bit.SDR.DTS-HD.MA.TrueHD.7
[INFO]: >>> Detected stalled download (5 out of 60 permitted times): Oceans.Eight.2018.2160p.BluRay.x265.10bit.SDR.DTS-HD.MA.TrueHD.7
[INFO]: >>> Detected stalled download (6 out of 60 permitted times): Oceans.Eight.2018.2160p.BluRay.x265.10bit.SDR.DTS-HD.MA.TrueHD.7
[INFO]: >>> Detected stalled download (7 out of 60 permitted times): Oceans.Eight.2018.2160p.BluRay.x265.10bit.SDR.DTS-HD.MA.TrueHD.7
[INFO]: >>> Detected stalled download (8 out of 60 permitted times): Oceans.Eight.2018.2160p.BluRay.x265.10bit.SDR.DTS-HD.MA.TrueHD.7
[INFO]: >>> Detected stalled download (9 out of 60 permitted times): Oceans.Eight.2018.2160p.BluRay.x265.10bit.SDR.DTS-HD.MA.TrueHD.7
I have several torrents stuck with the 'not a custom format upgrade' but declutarr doesn't seem to see them
Could you switch to debug mode please, turn off everything but REMOVE_NO_FORMAT_UPGRADE=True and pastebin the logs pls?
Sure.
Had to use WeTransfer though since file was larger than 512KB: https://we.tl/t-tEPBCh5H0c
No wonder it's half a MB: 2024-04-10T07:17:35.190566869Z [INFO]: True | Removing failed downloads 2024-04-10T07:17:35.190578430Z [INFO]: True | Removing downloads missing metadata 2024-04-10T07:17:35.190588589Z [INFO]: True | Removing downloads missing files 2024-04-10T07:17:35.190607455Z [INFO]: True | Removing downloads that fail on import (no format upgrade) 2024-04-10T07:17:35.190631259Z [INFO]: False | Removing orphan downloads 2024-04-10T07:17:35.190641739Z [INFO]: True | Removing slow downloads 2024-04-10T07:17:35.190778296Z [INFO]: True | Removing stalled downloads 2024-04-10T07:17:35.190885228Z [INFO]: True | Removing downloads belonging to unmonitored items
this should be all "False" except for 2024-04-10T07:17:35.190607455Z [INFO]: True | Removing downloads that fail on import (no format upgrade)
did you do this? :)
turn off everything but REMOVE_NO_FORMAT_UPGRADE=True
Looking at your logs, what I observed is that for instance "DD7C74205632DE4E881ADD20F58003D83642AF5D" is on the protectedDownloadIDs list, which means supposedly in qbit it's tagged to keep it and not be removed. The tag accoring to your logs that you specified is "~type_PrivateTracker"
can you share a screenshot of that movie from qbit please? Is it really tagged? If yes, then no wonder it doesn't get removed ;)
I though the removing meant removed from the queue? They are indeed tagged with '~type_PrivateTracker' sinse I want these to remain in qbit. Also sorry was on my phone so didn't see the last bit
let's try to break this down.
NO_STALLED_REMOVAL_QBIT_TAG -> If a torrent is tagged with this tag, the items in the queue that use this torrent are not removed by any rule
IGNORE_PRIVATE_TRACKERS -> If this setting is on, and the torrent in question is a private tracker, the queue item is not removed. Exception: if it's a case of post-download problem (REMOVE_NO_FORMAT_UPGRADE), then the queue item is deleted, but not the torrent.
Therefore: if you apply the tag NO_STALLED_REMOVAL_QBIT_TAG to your private trackers, and you already have turned on IGNORE_PRIVATE_TRACKERS, you double-protect them, which is why REMOVE_NO_FORMAT_UPGRADE does not take effect. REMOVE_NO_FORMAT_UPGRADE would in this case (since private tracker) only delete the queue item, but not the torrent. for public trackers, the torrent would be removed too.
makes sense?
Thus to test this, please
So removed the NO_STALLED_REMOVAL_QBIT_TAG and left IGNORE_PRIVATE_TRACKERS enabled but it seems they got removed from qbit: https://pastebin.com/t7cqiPgU
Ok. Meaning you‘re happy with the implementation or any open items?
No I expected the torrents to not be removed since they were from private trackers. If you check the logs it trigger delete from queue with the RemoveFromClient set to true for private trackers. Don't think that's supposed to happen?
For example these lines:
024-04-10T11:06:12.533346381Z [INFO]: >>> Removing no format upgrade download: The.Lord.of.the.Rings.The.Fellowship.of.the.Ring.2001.EXTENDED.4K.HDR.DV.2160p.BDRemux Ita Eng x265-NAHOM
2024-04-10T11:06:12.534694623Z [DEBUG]: Starting new HTTP connection (1): 127.0.0.1:7878
2024-04-10T11:06:12.706006586Z [DEBUG]: http://127.0.0.1:7878 "DELETE /api/v3/queue/1832671985?removeFromClient=True&blocklist=True HTTP/1.1" 200 0
humm. sorry for that. just pushed another version that gives more logs to see where it's going amiss. suggest you put TEST_RUN = True so you don't lose any more torrents.
can you run it again pls and paste the logs?
No worries. Can always download the torrent again if I get flagged. Enabled the test run in the meantime. Will have to wait for new errors to pop up to first. Will keep you posted once I get an item again.
I guess you could simulate it by re-requesting the same torrent that was rejected previously by searching for it manually? Or would that overwrite the better quality version you already have?
I removed the 'autosearched' tag from upgradinatorr so it will go through each move/series again. Should pull the previously deleted torrent if no new ones popped up. Will keep you posted
I have a hunch what it could be. can you pull the dev version again and paste the logs (even if there is currently no item that would be flagged under REMOVE_NO_FORMAT_UPGRADE). just need to see the logs when there are private tracker torrents added logs at the beginning, want to see if the private trackers get recognized correctly.
you will now see a log entry called "main/privateDowloadIDs: [...]". If that list is empty, it means that for whatever reason your private torrents are not recognized as such. if that's the case, can you please 1) write what qbittorrent image you are using 2) open your qbit and go to network tab, and paste what you see for the "/torrents/info" response?
what i am looking for in 2) is that private trackers should have a key "is_private: true"
I see this in my logs:
2024-04-11T06:38:34.551357991Z [DEBUG]: main/privateDowloadIDs: []
So I guess there's something wrong with the flag?
it means that for whatever reason your private torrents are not recognized as such.
if that's the case, can you please
write what qbittorrent image you are using
open your qbit and go to network tab, and paste what you see for the "/torrents/info" response?
what i am looking for in 2) is that private trackers should have a key "is_private: true"
Image I'm using is: lscr.io/linuxserver/qbittorrent:latest
I checked some torrents from each private tracker and they do all have the "is_private": true Here's an example:
{
"addition_date": 1712622878,
"comment": "https://hawke.uno/torrents/45921",
"completion_date": 1712622878,
"created_by": "Edited by HUNO",
"creation_date": 1709850961,
"dl_limit": -1,
"dl_speed": 0,
"dl_speed_avg": 0,
"download_path": "/downloads",
"eta": 8640000,
"hash": "d8df0a5b064080e05e469173a96ee04de4ad6b6e",
"infohash_v1": "d8df0a5b064080e05e469173a96ee04de4ad6b6e",
"infohash_v2": "",
"is_private": true,
"last_seen": -1,
"name": "Requiem.2006.1080p.MUBI.WEB-DL.AAC2.0.H.264-Roi",
"nb_connections": 0,
"nb_connections_limit": 1000,
"peers": 0,
"peers_total": 0,
"piece_size": 2097152,
"pieces_have": 1724,
"pieces_num": 1724,
"reannounce": 856,
"save_path": "/finished/radarr",
"seeding_time": 185438,
"seeds": 0,
"seeds_total": 10,
"share_ratio": 0,
"time_elapsed": 185438,
"total_downloaded": 0,
"total_downloaded_session": 0,
"total_size": 3614397049,
"total_uploaded": 0,
"total_uploaded_session": 0,
"total_wasted": 0,
"up_limit": -1,
"up_speed": 0,
"up_speed_avg": 0
}
Ok that‘s interesting. can you pls paste the full response to a pastebin? Will help me to understand what the problem is
Image I'm using is: lscr.io/linuxserver/qbittorrent:latest
perfect.
Btw I couldnn't find what you meant with '/torrents/info'. I had to select a torrent and go the general tab for the above to pop up "properties" request. Any idea where in qbit I can get the full list?
I was referring to the network tab (F12 in your browser). but never mind. I added another dev version that will print out what we need. can you please run it and paste the logs?
Oh I know. I just couldn't find any request except the general tab that had the is_private property. Here a zip with the logs: https://we.tl/t-NighLf5PXk
What i meant was in the network traffic tab (which normally opens with F12 on your keyboard of your browser; not within qbit.
thx for the logs, will check it out
Which web ui are you using? Mine looks completely different https://imgur.com/a/cZpYy3A
Is it possible that the logs you provided are truncated?
Which web ui are you using? Mine looks completely different https://imgur.com/a/cZpYy3A
just a different skin.
qbittorrent:
image: linuxserver/qbittorrent:latest
environment:
? DOCKER_MODS=arafatamim/linuxserver-io-mod-vuetorrent
I rewrote a part of the script which detects the private trackers. can you please pull again and test?
Ah cool didn't know you had skins.
Managed to update in the meantime: https://we.tl/t-WacCUJhGra
I think I've found out what's going on. the /info API of qBit does not return the isPrivate flag, only the /properties returns it..
Inconvenient, because the /properties needs to be queried for each torrent separately.
Created a PR on qBit, maybe the flag can be added to the response of the /info API, so that we don't have to fetch it individually for each torrent from the /properties API: https://github.com/qbittorrent/qBittorrent/pull/20686
Let's see what they say. If they don't approve the PR in the next days, I'll update the code on my end to fetch it via the /properties api
Ok sounds good.
Thanks btw for the quick reponse/coding!
I've changed the API call on my side. Can you pls check out dev if it works for you now?
Sorry for the delay. Was busy with work. Pulled the latest branch: https://we.tl/t-0VTy9uVO6e
Lines 4136/4137: You now see that the hashes are listed as private Download ids; before that list was empty.
[DEBUG]: main/getProtectedAndPrivateFromQbit/protectedDownloadIDs: []
[DEBUG]: main/getProtectedAndPrivateFromQbit/privateDowloadIDs: ['F4B35151518AE3156AA4E904FA4144CBCCB0532E', '64E8605321A39D8B0138D08A0C7FB6DF8B5E3C67', 'E8C7865FC4632EE34862219A5FCE07D70C6C0BAD', 'D6F4576080903CB21781D0217A5C915C27C09343', '295A6C25C469F8A712ABE3F4D3FB4C41DFF97B46', '77719C0AD9E34A757BCA728327DA75F5D88395D4', 'DD7C74205632DE4E881ADD20F58003D83642AF5D', '1C9EB47C9987EDA27995C1BFED44C9D142FE4B6F',
Further up you see the zillion calls to qbittorrent; that is because I needed to fire a separate call to the "properties" api for each hash individually since the "info" api of qbit doesn't return the isPrivate flag (PR still open).
[DEBUG]: http://127.0.0.1:6882 "GET /api/v2/torrents/properties?hash=4cd5f3bc3d2f74ba4f96060829424da306ac6410 HTTP/1.1" 200 596
[DEBUG]: Starting new HTTP connection (1): 127.0.0.1:6882
Now that the private ones get correctly identified, does what you wanted to get work (for private ones that failed import since no format upgrade that they don't get deleted from qbit, but removed from *arr app)?
See entries like on line 1386:
[INFO]: >>> Removing no format upgrade download (without removing torrent): ....
Did some double checks and it seems they are correctly identified now. Nice job :D.
Btw are they also blocklisted if they get removed? Just curious because as you said otherwise they would be pulled again
They do get blocklisted, although I think it would not matter. My interpretation on what is happening here is this:
now we kill the 720p and blocklist it. so the arr apps would not fetch this specific torrent again. however, since a better version is already available, I doubt they would fetch it anyways (even if we had not blocklisted it). because at the time, when it pulled the 720p the first time, a better version was not (yet) available.
just my 2 cents, not sure it's correct. as long as the result is what you wanted it to be, I'm happy. Ready to close the ticket?
Yea that's probably why. Removed the rest run flag and now everything gets properly handled.
Will save me a lot of time going forwards.
Thanks man :D
My setup is fully automated using the trash guides but I've noticed quite a lot of downloads get stuck in queue with message. (Not a Custom Format upgrade for existing movie file(s))
I was wondering if could get something to remove these items as well?