search

ManiMatter / decluttarr

Watches radarr, sonarr, lidarr, readarr and whisparr download queues and removes downloads if they become stalled or no longer needed.
GNU General Public License v3.0
148 stars 20 forks source link

Feature Request: Don't remove torrents while "Connection status != Connected" #124

Closed kmaid closed 4 weeks ago

kmaid commented 1 month ago

I use qBittorrent with a VPN that allows ports to be "leased". Most of the time everything works as expected however sometimes I lose the lease on the port. This results in an orange globe and all torrents being unable to download metadata.

image

Could you make an option that disables decluttarr while the connection status isn't connected? This solution isn't ideal as the connection status is not determined until after a torrent successfully downloads metadata.

ManiMatter commented 1 month ago

Hi Is the connection status in the queue response of sonarr/etc? If yes, can be done. If no, 2 options: A) request sonarr to add it B) switch to qbit

kmaid commented 1 month ago

Sorry I meant qbittorrent. I have amended my original message.

This status is not available in the Sonarr queue response.

ManiMatter commented 1 month ago

Should be possible. Will take a look when I get some time, might take a few weeks. Would appreciate if you could then support with testing

kmaid commented 1 month ago

Happy to help test. Could try my hand at python and create a PR if you would like

ManiMatter commented 1 month ago

Thx no need in this one, I have smth in mind how to it across all jobs with limited effort

appreciate the offer 🫡

kmaid commented 1 month ago

I think the best way for you to be able to test is to manually set a port in qBittorent that isn't opened on your router (disabling UNPnP). This should simulate the edge case. Ping me here and happy to test on my setup 👍

ManiMatter commented 1 month ago

I thought of checking connection status when the jobs run. If disconnected, simply skip.. wdyt?

kmaid commented 1 month ago

Yeah, I think that will work.

in the main data response it shows the connection status as 

    "connection_status": "firewalled",

and the status bar looks like

image
ManiMatter commented 1 month ago

Jup I already use that in certain places of the code; will expand on that

ManiMatter commented 1 month ago

I coded it in a way that it checks qbit connection status for those jobs where qbit connection matters. Could you pls pull dev-image and test?

kmaid commented 1 month ago

I have setup declutarr as shown below. I am not 100% sure if I am missing an env var to enable the new behaviour. It wasn't clear from the commits. 

 decluttarr:
    image: ghcr.io/manimatter/decluttarr:dev
    container_name: decluttarr
    restart: always
    environment:
      - TZ=${TIMEZONE}
      ## General
      - LOG_LEVEL=VERBOSE
      ## Features
      - REMOVE_TIMER=1
      - REMOVE_FAILED=True
      - REMOVE_FAILED_IMPORTS=True
      - REMOVE_METADATA_MISSING=True
      - REMOVE_MISSING_FILES=True
      - REMOVE_ORPHANS=True
      - REMOVE_SLOW=True
      - REMOVE_STALLED=True
      - REMOVE_UNMONITORED=True
      - MIN_DOWNLOAD_SPEED=100
      - PERMITTED_ATTEMPTS=3
      - IGNORE_PRIVATE_TRACKERS=True
      - FAILED_IMPORT_MESSAGE_PATTERNS=["Not an upgrade for existing", "Not a Custom Format upgrade for existing"]
      ## Radarr
      - RADARR_URL=http://radarr:7878
      - RADARR_KEY=${RADARR_API_KEY}
      ## Sonarr
      - SONARR_URL=http://sonarr:8989
      - SONARR_KEY=${SONARR_API_KEY}
      ## Readarr
      - READARR_URL=http://readarr:8787
      - READARR_KEY=${READARR_API_KEY}
      ## qBittorrent
      - QBITTORRENT_URL=http://vpn:8080
      - QBITTORRENT_USERNAME=${BITTORRENT_USER}
      - QBITTORRENT_PASSWORD=${BITTORRENT_PASSWORD}
    depends_on:
      vpn:
        condition: service_healthy
      sonarr:
        condition: service_healthy
      radarr:
        condition: service_healthy
      qbittorrent:
        condition: service_healthy
      readarr:
        condition: service_started
    networks:
      - web
      - default

I have attempted to change the port to see if I can get downloads to stall because of network problems rather than torrent health problems however even with the wrong port the torrent continues to work. 

[INFO   ]: ##################################################
[INFO   ]: Decluttarr - Application Started!
[INFO   ]: 
[INFO   ]: Like this app? Thanks for giving it a ⭐️ on GitHub!
[INFO   ]: https://github.com/ManiMatter/decluttarr/
[INFO   ]: 
[INFO   ]: *** Current Settings ***
[INFO   ]: Version: dev
[INFO   ]: Commit: d9c4969
[INFO   ]: 
[INFO   ]: True | Removing failed downloads (REMOVE_FAILED)
[INFO   ]: True | Removing failed imports (REMOVE_FAILED_IMPORTS)
[VERBOSE]: > Imports with a warning flag are considered failed if the status message contains any of the following patterns:
[VERBOSE]:   - "Not an upgrade for existing"
[VERBOSE]:   - "Not a Custom Format upgrade for existing"
[INFO   ]: True | Removing downloads missing metadata (REMOVE_METADATA_MISSING)
[INFO   ]: True | Removing downloads missing files (REMOVE_MISSING_FILES)
[INFO   ]: True | Removing orphan downloads (REMOVE_ORPHANS)
[INFO   ]: True | Removing slow downloads (REMOVE_SLOW)
[INFO   ]: True | Removing stalled downloads (REMOVE_STALLED)
[INFO   ]: True | Removing downloads belonging to unmonitored items (REMOVE_UNMONITORED)
[INFO   ]: 
[INFO   ]: Running every: 0 days 0 hours 1.0 minutes
[INFO   ]: Minimum speed enforced: 100 KB/s
[INFO   ]: Permitted number of times before stalled/missing metadata/slow downloads are removed: 3
[INFO   ]: Downloads with this tag will be skipped: "Don't Kill"
[INFO   ]: Private Trackers will be skipped: True
[INFO   ]: 
[INFO   ]: *** Configured Instances ***
[INFO   ]: Radarr: http://radarr:7878/api/v3
[INFO   ]: Sonarr: http://sonarr:8989/api/v3
[INFO   ]: Readarr: http://readarr:8787/api/v1
[INFO   ]: qBittorrent: http://vpn:8080/api/v2
[INFO   ]: 
[INFO   ]: *** Check Instances ***
[INFO   ]: OK | Radarr
[INFO   ]: OK | Sonarr
[INFO   ]: OK | Readarr
[INFO   ]: OK | qBittorrent
[INFO   ]: 
[INFO   ]: ##################################################
[INFO   ]: 
[VERBOSE]: --------------------------------------------------
[VERBOSE]: Cleaning queue on Radarr:
[VERBOSE]: >>> Queue is empty.
[VERBOSE]: Cleaning queue on Sonarr:
[VERBOSE]: >>> Queue is empty.
[VERBOSE]: Cleaning queue on Readarr:
[VERBOSE]: >>> Queue is empty.
[VERBOSE]: 
[VERBOSE]: Queue clean-up complete!
[VERBOSE]: --------------------------------------------------
[VERBOSE]: Cleaning queue on Radarr:
[VERBOSE]: >>> Queue is empty.
[VERBOSE]: Cleaning queue on Sonarr:
[INFO   ]: >>> Detected stalled download (1 out of 3 permitted times): [Redacted]
[VERBOSE]: >>> Queue is clean.
[VERBOSE]: Cleaning queue on Readarr:
[VERBOSE]: >>> Queue is empty.
[VERBOSE]: 
[VERBOSE]: Queue clean-up complete!
[VERBOSE]: --------------------------------------------------
[VERBOSE]: Cleaning queue on Radarr:
[VERBOSE]: >>> Queue is empty.
[VERBOSE]: Cleaning queue on Sonarr:
[INFO   ]: >>> Download no longer marked as stalled: [Redacted]
[VERBOSE]: >>> Queue is clean.
[VERBOSE]: Cleaning queue on Readarr:
[VERBOSE]: >>> Queue is empty.

I am thinking how to proceed. I am tempted to set it to dry run and wait for the problem to occur and see what happens.

Any suggestions?

ManiMatter commented 1 month ago

Settings: You do not need to set an addition al var. If the user configures qbit (so that we can establish a direct connection as opposed to rely on queue-info from the *arr apps), then the qbit connection status is checked automatically.

Testing: You face the same problem as I had; I couldn't figure out how to disconnect qbit. If you have a possibilty to for instance, pull the network plug physically, I guess that should do the trick to put it offline. When that is the case and you have a slow or stalled download, then you should see the warning that you probably have seen in the commit

To generate a "stalled" torrent manually: a bit of a challenge. What I normally do is add a torrent that has 0 seeds, and then remove all trackers from the torrent in qbit before it can download anything.. it's a bit of a "try and error"..

Once you have the stalled download, make sure to run with dry_run = true, so you don't kill your "test torrent" accidentially (in case you then want to still rely on it for further tests)

kmaid commented 1 month ago

I will give it a go on Sunday as I am away until then

ManiMatter commented 1 month ago

I will give it a go on Sunday as I am away until then

Did it work?

kmaid commented 1 month ago

Sorry, I started a new job on Monday, decided I needed a redundant DNS server while this work was carried out, found that my server was also connected by wifi and a bunch of other things slowed me down in testing this.

I tested the project to see a torrent was removed normally and it was. I am surprised declutarr doesn't mark the torrent as failed and or search for another source.

To test I added the dead torrent into qbittorrent manually and gave it a category sonarr would pickup. I waited for it to be detected once. I then physically disconnected the cable from my server to see what would happen and set a timer for 4 mins. (Checks every min removes at three so theoretically 1 min margin of error) 

2024-07-17T23:35:11.000544699Z [VERBOSE]: Queue clean-up complete!
2024-07-17T23:36:11.060554963Z [VERBOSE]: --------------------------------------------------
2024-07-17T23:36:11.067161356Z [VERBOSE]: Cleaning queue on Radarr:
2024-07-17T23:36:11.099794898Z [VERBOSE]: >>> Queue is empty.
2024-07-17T23:36:11.099827797Z [VERBOSE]: Cleaning queue on Sonarr:
2024-07-17T23:36:11.224167754Z [INFO   ]: >>> Detected missing metadata download (1 out of 3 permitted times): [REDACTED]
2024-07-17T23:36:11.611283733Z [VERBOSE]: >>> Queue is clean.
2024-07-17T23:36:11.611342476Z [VERBOSE]: Cleaning queue on Readarr:
2024-07-17T23:36:11.628716703Z [VERBOSE]: >>> Queue is empty.
2024-07-17T23:36:11.628772234Z [VERBOSE]: 
2024-07-17T23:36:11.628819103Z [VERBOSE]: Queue clean-up complete!
2024-07-17T23:37:11.655630316Z [VERBOSE]: --------------------------------------------------
2024-07-17T23:39:26.813468226Z TimeoutError: [Errno 110] Connection timed out
2024-07-17T23:39:26.813501611Z 
2024-07-17T23:39:26.813506780Z The above exception was the direct cause of the following exception:
2024-07-17T23:39:26.813512507Z 
2024-07-17T23:39:26.813518304Z urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f221765e140>, 'Connection to vpn timed out. (connect timeout=None)')
2024-07-17T23:39:26.813524520Z 
2024-07-17T23:39:26.813530736Z The above exception was the direct cause of the following exception:
2024-07-17T23:39:26.813536394Z 
2024-07-17T23:39:26.813542051Z urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='vpn', port=8080): Max retries exceeded with url: /api/v2/torrents/info (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f221765e140>, 'Connection to vpn timed out. (connect timeout=None)'))
2024-07-17T23:39:26.813550781Z 
2024-07-17T23:39:26.813556439Z During handling of the above exception, another exception occurred:
2024-07-17T23:39:26.813562166Z 
2024-07-17T23:39:26.813567893Z requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='vpn', port=8080): Max retries exceeded with url: /api/v2/torrents/info (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f221765e140>, 'Connection to vpn timed out. (connect timeout=None)'))
2024-07-17T23:39:26.813574040Z 
2024-07-17T23:39:26.813579837Z During handling of the above exception, another exception occurred:
2024-07-17T23:39:26.813586053Z 
2024-07-17T23:39:26.813591710Z UnboundLocalError: local variable 'response' referenced before assignment
2024-07-17T23:39:27.977316505Z [INFO   ]: ##################################################
2024-07-17T23:39:27.977359389Z [INFO   ]: Decluttarr - Application Started!
2024-07-17T23:39:27.977376710Z [INFO   ]: 
2024-07-17T23:39:27.977388444Z [INFO   ]: Like this app? Thanks for giving it a ⭐️ on GitHub!
2024-07-17T23:39:27.977456612Z [INFO   ]: https://github.com/ManiMatter/decluttarr/
2024-07-17T23:39:27.977495864Z [INFO   ]: 
2024-07-17T23:39:27.977549365Z [INFO   ]: *** Current Settings ***
2024-07-17T23:39:27.977680951Z [INFO   ]: Version: dev
2024-07-17T23:39:27.977726140Z [INFO   ]: Commit: d9c4969
2024-07-17T23:39:27.977795566Z [INFO   ]: 
2024-07-17T23:39:27.977832723Z [INFO   ]: True | Removing failed downloads (REMOVE_FAILED)
2024-07-17T23:39:27.977901310Z [INFO   ]: True | Removing failed imports (REMOVE_FAILED_IMPORTS)
2024-07-17T23:39:27.977935673Z [VERBOSE]: > Imports with a warning flag are considered failed if the status message contains any of the following patterns:
2024-07-17T23:39:27.977983516Z [VERBOSE]:   - "Not an upgrade for existing"
2024-07-17T23:39:27.978051544Z [VERBOSE]:   - "Not a Custom Format upgrade for existing"
2024-07-17T23:39:27.978116849Z [INFO   ]: True | Removing downloads missing metadata (REMOVE_METADATA_MISSING)
2024-07-17T23:39:27.978148837Z [INFO   ]: True | Removing downloads missing files (REMOVE_MISSING_FILES)
2024-07-17T23:39:27.978195493Z [INFO   ]: True | Removing orphan downloads (REMOVE_ORPHANS)
2024-07-17T23:39:27.978210859Z [INFO   ]: True | Removing slow downloads (REMOVE_SLOW)
2024-07-17T23:39:27.978223082Z [INFO   ]: True | Removing stalled downloads (REMOVE_STALLED)
2024-07-17T23:39:27.978237120Z [INFO   ]: True | Removing downloads belonging to unmonitored items (REMOVE_UNMONITORED)
2024-07-17T23:39:27.978249832Z [INFO   ]: 
2024-07-17T23:39:27.978312482Z [INFO   ]: Running every: 0 days 0 hours 1.0 minutes
2024-07-17T23:39:27.978335880Z [INFO   ]: Minimum speed enforced: 100 KB/s
2024-07-17T23:39:27.978350128Z [INFO   ]: Permitted number of times before stalled/missing metadata/slow downloads are removed: 3
2024-07-17T23:39:27.978360605Z [INFO   ]: Downloads with this tag will be skipped: "Don't Kill"
2024-07-17T23:39:27.978382536Z [INFO   ]: Private Trackers will be skipped: True
2024-07-17T23:39:27.978394479Z [INFO   ]: 
2024-07-17T23:39:27.978421160Z [INFO   ]: *** Configured Instances ***
2024-07-17T23:39:27.978434151Z [INFO   ]: Radarr: http://radarr:7878/api/v3
2024-07-17T23:39:27.978462298Z [INFO   ]: Sonarr: http://sonarr:8989/api/v3
2024-07-17T23:39:27.978472635Z [INFO   ]: Readarr: http://readarr:8787/api/v1
2024-07-17T23:39:27.978483600Z [INFO   ]: qBittorrent: http://vpn:8080/api/v2
2024-07-17T23:39:27.978494217Z [INFO   ]: 
2024-07-17T23:39:27.978537799Z [INFO   ]: *** Check Instances ***
2024-07-17T23:39:27.994037544Z [INFO   ]: OK | Radarr
2024-07-17T23:39:28.002778400Z [INFO   ]: OK | Sonarr
2024-07-17T23:39:28.010076276Z [INFO   ]: OK | Readarr
2024-07-17T23:41:37.884939904Z [ERROR  ]: !! qBittorrent Error: !!
2024-07-17T23:41:37.884983347Z [ERROR  ]: HTTPConnectionPool(host='vpn', port=8080): Max retries exceeded with url: /api/v2/auth/login (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f3a639fcd00>, 'Connection to vpn timed out. (connect timeout=None)'))
2024-07-17T23:41:37.884997595Z [ERROR  ]: Details:
2024-07-17T23:41:37.885578068Z TimeoutError: [Errno 110] Connection timed out
2024-07-17T23:41:37.885592316Z 
2024-07-17T23:41:37.885599300Z The above exception was the direct cause of the following exception:
2024-07-17T23:41:37.885605027Z 
2024-07-17T23:41:37.885610685Z urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f3a639fcd00>, 'Connection to vpn timed out. (connect timeout=None)')
2024-07-17T23:41:37.885629403Z 
2024-07-17T23:41:37.885635968Z The above exception was the direct cause of the following exception:
2024-07-17T23:41:37.885642673Z 
2024-07-17T23:41:37.885649448Z urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='vpn', port=8080): Max retries exceeded with url: /api/v2/auth/login (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f3a639fcd00>, 'Connection to vpn timed out. (connect timeout=None)'))
2024-07-17T23:41:37.885655175Z 
2024-07-17T23:41:37.885660414Z During handling of the above exception, another exception occurred:
2024-07-17T23:41:37.885666490Z 
2024-07-17T23:41:37.885671728Z requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='vpn', port=8080): Max retries exceeded with url: /api/v2/auth/login (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f3a639fcd00>, 'Connection to vpn timed out. (connect timeout=None)'))
2024-07-17T23:41:37.885678014Z 
2024-07-17T23:41:37.885683811Z During handling of the above exception, another exception occurred:
2024-07-17T23:41:37.885689469Z 
2024-07-17T23:41:37.885694777Z UnboundLocalError: local variable 'response' referenced before assignment

As you can see the docker container crashed when it couldn't connect to the VPN container. The VPN container (https://github.com/thrnz/docker-wireguard-pia) just kept trying to reconnect.

This isn't the behaviour I expected. Everything recovered when the network cable was restored. I will have to leave this running longer for whatever edge case to smite me again 👎

ManiMatter commented 1 month ago

I am surprised declutarr doesn't mark the torrent as failed and or search for another source.

Please check the readme - some jobs get marked as failed, others don't (by design)

2024-07-17T23:39:26.813468226Z TimeoutError: [Errno 110] Connection timed out
2024-07-17T23:39:26.813501611Z 
2024-07-17T23:39:26.813506780Z The above exception was the direct cause of the following exception:
2024-07-17T23:39:26.813512507Z 
2024-07-17T23:39:26.813518304Z urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f221765e140>, 'Connection to vpn timed out. (connect timeout=None)')
2024-07-17T23:39:26.813524520Z 
2024-07-17T23:39:26.813530736Z The above exception was the direct cause of the following exception:
2024-07-17T23:39:26.813536394Z 
2024-07-17T23:39:26.813542051Z urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='vpn', port=8080): Max retries exceeded with url: /api/v2/torrents/info (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f221765e140>, 'Connection to vpn timed out. (connect timeout=None)'))
2024-07-17T23:39:26.813550781Z 
2024-07-17T23:39:26.813556439Z During handling of the above exception, another exception occurred:
2024-07-17T23:39:26.813562166Z 
2024-07-17T23:39:26.813567893Z requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='vpn', port=8080): Max retries exceeded with url: /api/v2/torrents/info (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f221765e140>, 'Connection to vpn timed out. (connect timeout=None)'))
2024-07-17T23:39:26.813574040Z 
2024-07-17T23:39:26.813579837Z During handling of the above exception, another exception occurred:
2024-07-17T23:39:26.813586053Z 
2024-07-17T23:39:26.813591710Z UnboundLocalError: local variable 'response' referenced before assignment

This looks to me like decluttar can't reach qbit anymore. If qbit can't be reached, decluttarr will crash, exit, and restart until qbit comes available again. This looks OK.

However, this is not the scenario we had been talking about. My understanding was: Your qbit keeps working and being available to decluttarr to connect. What changes is that for some reason qbit gets disconnected from the internet. When that happens, qbit will still reach it, but it will not perform the cleanup jobs that require an active connection to be meaningful (like slowness check).

Can you try that?

Btw - if you switch to "debug" logs, and turn Testrun = true (see readme), and focus maybe on one job (like the slowness one, and turn the other ones off), it's easier to test.

ManiMatter commented 1 month ago

hi - just wanted to check if you were able to follow-up on this? would be great moving towards closure

ManiMatter commented 4 weeks ago

Considered closed.