Only check torrents during slow check

[phyzical]

Hey,

just stumbled across this cool idea to solve the problem.

From my initial testing i ran into two issues

• wasn't obvious that the stalled job only works with qbit - so updated the readme
• The slow check doesn't account for the fact there may be post processing occurring, so ive added logic to avoid running it for downloads that have 0 bytes left

Ive just gotta build and test this. ill let you know when ive done this

Edit: instead its been adjusted to just only run slow checks for torrents only

[phyzical]

hmm bummer..

looks like this might not work as i expected for example This is the payload from sonarr for a torrent that hasn't had a single byte go through, it seems to report hat theres nothing left to download..

{'seriesId': 383, 'episodeId': 25437, 'seasonNumber': 3, 'languages': [{'id': 8, 'name': 'Japanese'}], 'quality': {'quality': {'id': 3, 'name': 'WEBDL-1080p', 'source': 'web', 'resolution': 1080}, 'revision': {'version': 1, 'real': 0, 'isRepack': False}}, 'customFormats': [], 'customFormatScore': 0, 'size': 0, 'title': 'One Piece S03 1080p CR WEB-DL AAC2.0 H.264-KAN3D2M', 'sizeleft': 0, 'timeleft': '00:00:00', 'estimatedCompletionTime': '2024-05-10T16:08:11Z', 'added': '2024-05-09T11:09:25Z', 'status': 'downloading', 'trackedDownloadStatus': 'ok', 'trackedDownloadState': 'downloading', 'statusMessages': [], 'downloadId': '65D789A2F62F706F93F7DE71A33F35BDF3E5FBA5', 'protocol': 'torrent', 'downloadClient': 'Deluge', 'downloadClientHasPostImportCategory': False, 'indexer': 'kickasstorrents.to (Prowlarr)', 'outputPath': '/downloads/incoming/One Piece S03 1080p CR WEB-DL AAC2.0 H.264-KAN3D2M', 'episodeHasFile': True, 'id': 1040458739}

for an item thats stuck in sab doing post processing it would work as expected though

{'seriesId': 338, 'episodeId': 19820, 'seasonNumber': 1, 'languages': [{'id': 2, 'name': 'French'}], 'quality': {'quality': {'id': 6, 'name': 'Bluray-720p', 'source': 'bluray', 'resolution': 720}, 'revision': {'version': 1, 'real': 0, 'isRepack': False}}, 'customFormats': [], 'customFormatScore': 0, 'size': 808611467, 'title': 'Dragon.Ball.Z.Kai.E77.FRENCH.720p.BluRay.x264-KAZETV', 'sizeleft': 0, 'timeleft': '00:00:00', 'estimatedCompletionTime': '2024-05-10T16:12:37Z', 'added': '2024-05-10T11:43:39Z', 'status': 'downloading', 'trackedDownloadStatus': 'ok', 'trackedDownloadState': 'downloading', 'statusMessages': [], 'errorMessage': '', 'downloadId': 'SABnzbd_nzo_ywz0eq9b', 'protocol': 'usenet', 'downloadClient': 'sab', 'downloadClientHasPostImportCategory': False, 'indexer': 'NZBFinder (yearly) (Prowlarr)', 'episodeHasFile': True, 'id': 1230492900}

im thinking maybe the play is dont check for slow if size > 0 && sizeleft == 0

[phyzical]

Alright can confirm this logic no longer reports finished post processing sab items as "slow" .

Though im not 100% confident it wont fight logic around qbit.

If you feel it not testable just lmk and i can chuck this logic behind queueItem['downloadClient'] != 'qBittorrent'

[phyzical]

hmm actually, now that ive thought about this more, how does one use this in combination with sab?

Ive just noticed that downloads stuck in sabs queue, as sab is synchronous will be marked as "slow" i guess maybe the correct way to solve this would be to have some flag/tag detection

idealy i would want the slow check to only occur for torrents

so maybe the play would be

'protocol': 'torrent'

Would the slow check ever be useful for usenet usecases?

[phyzical]

Alright this iteration seems to work for me as id expect

[ManiMatter]

Hi @phyzical ,

thanks for the contribution.

The "Slowness" check does not just work for qbit, it works for any torrent downloader. However, for qBit, real-time connection is used to check the current download progress, whereas for all other torrent downloaders the information in the arr apps is used, which may be cached and thus slightly outdated. If intervals are used that are more narrow than the intervals in which the arr apps query the torrent downloader to refresh the download progress of the queue items, then decluttarr receveis outdated progress infomration and may flag torrents as slow (even if they might not be). Simple fix: Either use qbit, or increase the periodicity by which decluttarr runs (eg. every 15min instead of every 10min or smth like this).

For SAB, the check should also work, since this code checks if the download client of the respective queue item is qbit (which it won't be for SAB), and then defaults to using the information provided by the *arr apps

https://github.com/ManiMatter/decluttarr/blob/main/src/jobs/remove_slow.py

async def getDownloadedSize(settingsDict, queueItem, download_sizes_tracker, NAME):
    try:
        # Determines the speed of download
        # Since Sonarr/Radarr do not update the downlodedSize on realtime, if possible, fetch it directly from qBit
        if settingsDict['QBITTORRENT_URL'] and queueItem['downloadClient'] == 'qBittorrent':
            qbitInfo = await rest_get(settingsDict['QBITTORRENT_URL']+'/torrents/info',params={'hashes': queueItem['downloadId']}, cookies=settingsDict['QBIT_COOKIE']  )
            downloadedSize = qbitInfo[0]['completed']

        else:  # THIS SHOULD KICK IN FOR SAB SINCE DOWNLOAD CLIENT WON'T BE QBIT
            logger.debug('getDownloadedSize/WARN: Using imprecise method to determine download increments because no direct qBIT query is possible')

            downloadedSize = queueItem['size'] - queueItem['sizeleft']

Therefore, rather than deactivating this for SAB, my question would be what needs to be refined so that it works properly?

[phyzical]

the issue i found was for example, if i had a queue with say 1000 items, all would be regarded as "slow" as they were in the queue waiting to be downloaded. when in reality they were just in the queue, as sab will just download 1 by one vs how torrents work

I did try to adjust the logic to be more magical, but it really didn't feel very reliable

https://github.com/ManiMatter/decluttarr/pull/97/commits/d284b9c43be2ba09a2403957e0f83b7d7a20d421

for example this iteration, worked for torrents, not in qbit but failed for sab

i think (from memory) the issue become that "size" and "sizeleft" are both 0 for downloads that haven't started (for sab). <- we would not want to revmove these

But these are also 0 for torrents that have not received any bytes <- we would want these removed

So i guess (if im remembering correctly) we could use the type of indexer as a check to use size/sizeleft as an indicator. Buuut i kinda felt like the increased complexity wasnt really worth it as ive never had to action these sort of items for usenet

[ManiMatter]

the current code has this condition: if queueItem['status'] == 'downloading' and ...

therefore, items that are "queued" should not be flagged by slowness check.

If for SAB this is not working, then my question would be: why is the status "downloading" and not queued?

Can you paste a screenshot of your SAB downloader, and the "queue" API response from your network tab that you get when you are on the *arr app under the "queue" tab?

[phyzical]

this is the season 11 that is downloading

{
    "5": {
        "seriesId": 1632,
        "episodeId": 152398,
        "seasonNumber": 11,
        "languages": [
            {
                "id": 1,
                "name": "English"
            }
        ],
        "quality": {
            "quality": {
                "id": 15,
                "name": "WEBRip-1080p",
                "source": "webRip",
                "resolution": 1080
            },
            "revision": {
                "version": 1,
                "real": 0,
                "isRepack": false
            }
        },
        "customFormats": [],
        "customFormatScore": 0,
        "size": 18246176604,
        "title": "Heartland.(2007).S11.WEBRip.AAC.5.1.1080p.x265-SiQ-xpost",
        "sizeleft": 6776799887,
        "timeleft": "00:09:29",
        "estimatedCompletionTime": "2024-05-13T10:27:47Z",
        "added": "2024-05-13T08:09:27Z",
        "status": "downloading",
        "trackedDownloadStatus": "ok",
        "trackedDownloadState": "downloading",
        "statusMessages": [],
        "downloadId": "SABnzbd_nzo_nv2w7d80",
        "protocol": "usenet",
        "downloadClient": "sab",
        "downloadClientHasPostImportCategory": false,
        "indexer": "Nzb.su (yearly) (Prowlarr)",
        "episodeHasFile": false,
        "id": 260192043
    }
}

this is the season 12 that is qeued

{
    "17": {
        "seriesId": 1632,
        "episodeId": 152410,
        "seasonNumber": 12,
        "languages": [
            {
                "id": 1,
                "name": "English"
            }
        ],
        "quality": {
            "quality": {
                "id": 15,
                "name": "WEBRip-1080p",
                "source": "webRip",
                "resolution": 1080
            },
            "revision": {
                "version": 1,
                "real": 0,
                "isRepack": false
            }
        },
        "customFormats": [],
        "customFormatScore": 0,
        "size": 10116891934,
        "title": "Heartland.(2007).S12.WEBRip.AAC.2.0.1080p.x265-SiQ-xpost",
        "sizeleft": 10116891934,
        "timeleft": "00:23:40",
        "estimatedCompletionTime": "2024-05-13T10:41:58Z",
        "added": "2024-05-13T08:09:53Z",
        "status": "downloading",
        "trackedDownloadStatus": "ok",
        "trackedDownloadState": "downloading",
        "statusMessages": [],
        "downloadId": "SABnzbd_nzo_ayfkef7a",
        "protocol": "usenet",
        "downloadClient": "sab",
        "downloadClientHasPostImportCategory": false,
        "indexer": "Nzb.su (yearly) (Prowlarr)",
        "episodeHasFile": false,
        "id": 821849519
    }
}

As you can see they are both marked as "downloading" to sonars point of view

We could do the check i orignally suggested which is if sizeleft == size but we would still want to only do this for usenet

so it would still end up being sizeleft != size && protocol == 'usenet'

But even if we did this there is still the other issue around items in post proccessing so then we also need something along the lines of sizeleft != 0 && protocol == 'usenet'

So i guess it becomes (sizeleft != size || sizeleft != 0) && protocol == 'usenet'

Which then made me feel well idk if usenet really needs the slow check at all tbh

thoughts @ManiMatter ?

[ManiMatter]

Which then made me feel well idk if usenet really needs the slow check at all tbh well, if things are slow on usenet, why would we not want to kill it?

when you go to your usenet app, and you look at the network responses, what is the status that you see for the download that is currently being queued? Is it "downloading" as well?

If it is "queued" or anything simililar, then the fix is easy: PR to sonarr/radarr etc to not map "queued" to "downloading" but to keep it in a separate value as they do for torrents..

[phyzical]

sab side it looks like this, this is one that is "queued"

also returns that it is downloading, so i dont think this can really be solved sab side

{
    "1": {
        "index": 1,
        "nzo_id": "SABnzbd_nzo_17cl7wky",
        "unpackopts": "3",
        "priority": "Normal",
        "script": "None",
        "filename": "Heartland.(2007).S14.WEBRip.AAC.2.0.1080p.x265-SiQ",
        "labels": [],
        "password": "",
        "cat": "tv",
        "mbleft": "7411.35",
        "mb": "7411.35",
        "size": "7.2 GB",
        "sizeleft": "7.2 GB",
        "percentage": "0",
        "mbmissing": "0.00",
        "direct_unpack": null,
        "status": "Downloading",
        "timeleft": "0:06:09",
        "avg_age": "99d"
    }
}

[phyzical]

well, if things are slow on usenet, why would we not want to kill it?

its direct download unlike torrents, usenet you get the speed you pay for. thats it, if its slow downloads you need to stop paying/using that provider.

where as torrent its all peer based

[ManiMatter]

Fair, I think it makes sense to skip usenet for slowness check. Just merged #104 to dev -> does this do the trick? Could you pls pull dev image and verify?

[phyzical]

that looks like it should do the job. i think the important stuff i need should be in dev by now, ill spin it up now and run test mode

[phyzical]

Queue cleaning failed on Lidarr. (File: remove_slow.py / Line: 27 / Error Message: 'usenet' / Error Type: <class 'KeyError'>)

[phyzical]

you want

if queueItem['protocol'] == 'usenet':

[ManiMatter]

yeah. not my brightest moment..

fixed already - pls can you try again?

[phyzical]

Its no longer erroring i think this is closable 👍