search

ManiMatter / decluttarr

Watches radarr, sonarr, lidarr, readarr and whisparr download queues and removes downloads if they become stalled or no longer needed.
GNU General Public License v3.0
99 stars 15 forks source link

[Feature request] Remove downloads that fail to import #112

Closed SixFive7 closed 1 month ago

SixFive7 commented 1 month ago

It would be nice if Decluttarr could automatically remove and blacklist downloads that fail to import. Right now these downloads get stuck requiring manual intervention. Related to ticket https://github.com/ManiMatter/decluttarr/issues/78 though that ticket seems to focus more on Lidarr auto import issues.

Downloads that are stuck have a qbit download state of completed and show the following state in Radarr:

{
  "movieId": "REDACTED",
  "languages": [],
  "quality": {
    "quality": {
      "id": 12,
      "name": "WEBRip-480p",
      "source": "webrip",
      "resolution": 480,
      "modifier": "none"
    },
    "revision": {
      "version": 1,
      "real": 0,
      "isRepack": false
    }
  },
  "customFormats": [],
  "customFormatScore": 0,
  "size": "REDACTED",
  "title": "REDACTED.mp4",
  "sizeleft": 0,
  "timeleft": "00:00:00",
  "estimatedCompletionTime": "20XX-XX-XXTXX:XX:XXZ",
  "added": "20XX-XX-XXTXX:XX:XXZ",
  "status": "completed",
  "trackedDownloadStatus": "warning", <- Maybe we can track this for the remove timer?
  "trackedDownloadState": "importPending", <- Maybe we can track this for the remove timer?
  "statusMessages": [
    {
      "title": "One or more movies expected in this release were not imported or missing", <- Seems this is
mostly consistent. At least for Radarr. Sonarr, Lidarr, Readarr and Whisparr seem to have different titles.
      "messages": [] <- Seems to always be empty?
    },
    {
      "title": "REDACTED",
      "messages": [
        "Unable to parse file" <- This differs a lot.
      ]
    }
  ],
  "downloadId": "REDACTED",
  "protocol": "torrent",
  "downloadClient": "qBittorrent",
  "downloadClientHasPostImportCategory": false,
  "indexer": "REDACTED",
  "outputPath": "REDACTED",
  "id": "REDACTED"
},

I you think this might be a fun feature I'd be happy to test around.

ManiMatter commented 1 month ago

Your suggestion sounds to me a broader, more generic approach to failed imports to the already existing feature below (REMOVE_NO_FORMAT_UPGRADE):

https://github.com/ManiMatter/decluttarr/blob/342f4eac43b843adaaa547f6a1c3b60f3f5cb0b4/src/jobs/remove_no_format_upgrade.py#L23-L26

Can you explain if you are proposing to remove any downloads that have finished and fail to import, or only specific ones (based on error type)?

I do not know what error types exist, and I am a bit afraid of generating a "catch-all" removal that may remove too many things. wdyt?

SixFive7 commented 1 month ago

I'd for one would very much like an REMOVE_FAILED_IMPORT_ANY_REASON flag with which I can opt in to remove any failed download making it completely hands free and trusting the arr apps themselves to sort it out. However, I can image some user wanting to only specify certain conditions and do the rest manually.

Maybe we could leave REMOVE_NO_FORMAT_UPGRADE. Allow for future REMOVE_SOME_IMPORT_ERROR_REASON whenever there is a specific reason happening a lot. But for now also add an opt-in to REMOVE_FAILED_IMPORT_ANY. Documentation could then state this to remove all failed imports whatever the arr app's own reason for declining the import. Superseding more specific reason like REMOVE_NO_FORMAT_UPGRADE or any other future specific reasons, Thus leaving it fully up to the arr apps if the user want to.

This way we could maybe have it both? The code should be a simple short circuit on line 24?

I'd be happy to create a pull request except I have zero knowledge setting up a python dev environment (yet).

ManiMatter commented 1 month ago

What I don't like this this approach is that the app gets more and more settings which are super difficult to read. How about this? We change the REMOVE_NO_FORMAT_UPGRADE to REMOVE_FAILED_IMPORTS and we add a "FAILED_IMPORT_MESSAGE_PATTERNS" that can be configured. Default is ["Not a Custom Format upgrade for existing"]

And then we a) check whether the feature is on (REMOVE_FAILED_IMPORTS) b) if it is on, whether the status is "warning" and import is pending c) if it is, we check if it has a status message that contains any of the entries on the FAILED_IMPORT_MESSAGE_PATTERNS list. (contains, not equals)

This way, you can specify exactly which warnings you want to kill, and which ones you want to keep. If you leave FAILED_IMPORT_MESSAGE_PATTERNS on [] (or don't specify it at all), any item with a warning gets deleted (since all messages will contain "")

What do you think

ManiMatter commented 1 month ago

Already coded it. Can you pls pull dev and test?

SixFive7 commented 1 month ago

Wow you move fast. Yeah that sounds like a far better and flexible approach.

I just tested against commit 197152c with the following config:

REMOVE_FAILED_IMPORTS: True
FAILED_IMPORT_MESSAGE_PATTERNS: '["Not a Custom Format upgrade for existing","Not an upgrade for existing",".arj"]'

Because I have a download with the message Invalid video file, unsupported extension: '.arj'. But nothing seems to be happening. Also I get the startup message False | Removing downloads that fail on import (no format upgrade) on startup. I might be doing something silly though. Any clue?

Output:

[INFO]: ##################################################
[INFO]: Decluttarr - Application Started!
[INFO]: 
[INFO]: Like this app? Thanks for giving it a ⭐️ on GitHub!
[INFO]: https://github.com/ManiMatter/decluttarr/
[INFO]: 
[INFO]: *** Current Settings ***
[INFO]: Version: dev
[INFO]: Commit: 197152c
[INFO]: 
[INFO]: True | Removing failed downloads
[INFO]: True | Removing downloads missing metadata
[INFO]: True | Removing downloads missing files
[INFO]: False | Removing downloads that fail on import (no format upgrade)
[INFO]: True | Removing orphan downloads
[INFO]: True | Removing slow downloads
[INFO]: True | Removing stalled downloads
[INFO]: True | Removing downloads belonging to unmonitored items
[INFO]: 
[INFO]: Running every: 0 days 0 hours 1.0 minutes
[INFO]: Minimum speed enforced: 100 KB/s
[INFO]: Permitted number of times before stalled/missing metadata/slow downloads are removed: 10
[INFO]: Downloads with this tag will be skipped: "Don't Kill"
[INFO]: Private Trackers will be skipped: True
[INFO]: 
[INFO]: *** Configured Instances ***
[INFO]: Radarr: http://media-radarr:7878/api/v3
[INFO]: Sonarr: http://media-sonarr:8989/api/v3
[INFO]: Lidarr: http://media-lidarr:8686/api/v1
[INFO]: Readarr: http://media-readarr:8787/api/v1
[INFO]: qBittorrent: http://media-qbittorrent:8080/api/v2
[INFO]: 
[INFO]: *** Check Instances ***
[INFO]: OK | Radarr
[INFO]: OK | Sonarr
[INFO]: OK | Lidarr
[INFO]: OK | Readarr
[INFO]: OK | qBittorrent
[INFO]: 
[INFO]: ##################################################
[INFO]: LOG_LEVEL = INFO: Only logging changes (switch to VERBOSE for more info)
ManiMatter commented 1 month ago

Why would you test against 197152c?? That's 2 weeks old. https://github.com/ManiMatter/decluttarr/commits/dev/

You need to remove your existing dev image whenever you test a new version, so that it pulls the latest one.

SixFive7 commented 1 month ago

Whoops my bad. Had a cache that needed clearing.

Testing against 93a3ca1 is giving me:

[INFO]: ##################################################
[INFO]: Decluttarr - Application Started!
[INFO]: 
[INFO]: Like this app? Thanks for giving it a ⭐️ on GitHub!
[INFO]: https://github.com/ManiMatter/decluttarr/
[INFO]: 
[INFO]: *** Current Settings ***
[INFO]: Version: dev
[INFO]: Commit: 93a3ca1
[INFO]: 
[INFO]: True | Removing failed downloads (REMOVE_FAILED)
[INFO]: True | Removing failed imports (REMOVE_FAILED_IMPORTS)
[INFO]: True | Removing downloads missing metadata (REMOVE_METADATA_MISSING)
[INFO]: True | Removing downloads missing files (REMOVE_MISSING_FILES)
[INFO]: True | Removing orphan downloads (REMOVE_ORPHANS)
[INFO]: True | Removing slow downloads (REMOVE_SLOW)
[INFO]: True | Removing stalled downloads (REMOVE_STALLED)
[INFO]: True | Removing downloads belonging to unmonitored items (REMOVE_UNMONITORED)
[INFO]: 
[INFO]: Running every: 0 days 0 hours 1.0 minutes
[INFO]: Minimum speed enforced: 100 KB/s
[INFO]: Permitted number of times before stalled/missing metadata/slow downloads are removed: 10
[INFO]: Downloads with this tag will be skipped: "Don't Kill"
[INFO]: Private Trackers will be skipped: True
[INFO]: 
[INFO]: *** Configured Instances ***
[INFO]: Radarr: http://media-radarr:7878/api/v3
[INFO]: Sonarr: http://media-sonarr:8989/api/v3
[INFO]: Lidarr: http://media-lidarr:8686/api/v1
[INFO]: Readarr: http://media-readarr:8787/api/v1
[INFO]: qBittorrent: http://media-qbittorrent:8080/api/v2
[INFO]: 
[INFO]: *** Check Instances ***
[INFO]: OK | Radarr
[INFO]: OK | Sonarr
[INFO]: OK | Lidarr
[INFO]: OK | Readarr
[INFO]: OK | qBittorrent
[INFO]: 
[INFO]: ##################################################
[INFO]: LOG_LEVEL = INFO: Only logging changes (switch to VERBOSE for more info)
[WARNING]: >>> Queue cleaning failed on Radarr. (File: remove_failed_imports.py / Line: 26 / Error Message: name 'messages' is not defined / Error Type: <class 'NameError'>)
[WARNING]: >>> Queue cleaning failed on Radarr. (File: remove_failed_imports.py / Line: 26 / Error Message: name 'messages' is not defined / Error Type: <class 'NameError'>)
[WARNING]: >>> Queue cleaning failed on Radarr. (File: remove_failed_imports.py / Line: 26 / Error Message: name 'messages' is not defined / Error Type: <class 'NameError'>)
[WARNING]: >>> Queue cleaning failed on Radarr. (File: remove_failed_imports.py / Line: 26 / Error Message: name 'messages' is not defined / Error Type: <class 'NameError'>)
[WARNING]: >>> Queue cleaning failed on Radarr. (File: remove_failed_imports.py / Line: 26 / Error Message: name 'messages' is not defined / Error Type: <class 'NameError'>)
[WARNING]: >>> Queue cleaning failed on Radarr. (File: remove_failed_imports.py / Line: 26 / Error Message: name 'messages' is not defined / Error Type: <class 'NameError'>)
ManiMatter commented 1 month ago

Sorry, glitch from my side. Can you pls try again?

SixFive7 commented 1 month ago

I just checked against commit a825f41 and everything looks perfect. I first tested a few patterns to remove some collected stuck imports with specific messages. They were removed. When left with three different messages I removed the FAILED_IMPORT_MESSAGE_PATTERNS variable. It then successfully removed everything. So, all good. Thank you!

Output of last run:

[INFO]: ##################################################
[INFO]: Decluttarr - Application Started!
[INFO]: 
[INFO]: Like this app? Thanks for giving it a ⭐️ on GitHub!
[INFO]: https://github.com/ManiMatter/decluttarr/
[INFO]: 
[INFO]: *** Current Settings ***
[INFO]: Version: dev
[INFO]: Commit: a825f41
[INFO]: 
[INFO]: True | Removing failed downloads (REMOVE_FAILED)
[INFO]: True | Removing failed imports (REMOVE_FAILED_IMPORTS)
[INFO]: True | Removing downloads missing metadata (REMOVE_METADATA_MISSING)
[INFO]: True | Removing downloads missing files (REMOVE_MISSING_FILES)
[INFO]: True | Removing orphan downloads (REMOVE_ORPHANS)
[INFO]: True | Removing slow downloads (REMOVE_SLOW)
[INFO]: True | Removing stalled downloads (REMOVE_STALLED)
[INFO]: True | Removing downloads belonging to unmonitored items (REMOVE_UNMONITORED)
[INFO]: 
[INFO]: Running every: 0 days 0 hours 1.0 minutes
[INFO]: Minimum speed enforced: 100 KB/s
[INFO]: Permitted number of times before stalled/missing metadata/slow downloads are removed: 10
[INFO]: Downloads with this tag will be skipped: "Don't Kill"
[INFO]: Private Trackers will be skipped: True
[INFO]: 
[INFO]: *** Configured Instances ***
[INFO]: Radarr: http://media-radarr:7878/api/v3
[INFO]: Sonarr: http://media-sonarr:8989/api/v3
[INFO]: Lidarr: http://media-lidarr:8686/api/v1
[INFO]: Readarr: http://media-readarr:8787/api/v1
[INFO]: qBittorrent: http://media-qbittorrent:8080/api/v2
[INFO]: 
[INFO]: *** Check Instances ***
[INFO]: OK | Radarr
[INFO]: OK | Sonarr
[INFO]: OK | Lidarr
[INFO]: OK | Readarr
[INFO]: OK | qBittorrent
[INFO]: 
[INFO]: ##################################################
[INFO]: LOG_LEVEL = INFO: Only logging changes (switch to VERBOSE for more info)
[INFO]: >>> Removing failed import download: REDACTED
[INFO]: >>> Removing failed import download: REDACTED
[INFO]: >>> Removing failed import download: REDACTED
[INFO]: >>> Removing failed import download: REDACTED
[INFO]: >>> Removing failed import download: REDACTED
[INFO]: >>> Removing failed import download: REDACTED
[INFO]: >>> Removing failed import download: REDACTED
[INFO]: >>> Detected slow download that has completed downloading - skipping check (torrent files likely in process of being moved): REDACTED
[INFO]: >>> Detected slow download that has completed downloading - skipping check (torrent files likely in process of being moved): REDACTED
[INFO]: >>> Detected slow download (1 out of 10 permitted times): REDACTED
[INFO]: >>> Detected slow download (2 out of 10 permitted times): REDACTED
[INFO]: >>> Download no longer marked as slow: REDACTED
[INFO]: >>> Detected slow download (1 out of 10 permitted times): REDACTED
[INFO]: >>> Download no longer marked as slow: REDACTED
[INFO]: >>> Detected slow download (1 out of 10 permitted times): REDACTED
[INFO]: >>> Download no longer marked as slow: REDACTED