search

cross-seed / cross-seed

Fully-automatic cross-seeding with Torznab
https://cross-seed.org
Apache License 2.0
761 stars 69 forks source link

Snatch timeout on specific Prowlarr indexer #560

Closed plz12345 closed 6 months ago

plz12345 commented 7 months ago

I'm troubleshooting a consistent timeout when cross-seed is trying to snatch a torrent from Powlarr for a specific indexer. If I review the logs and rebuild the URL (and un-redact the API key), the request works properly and I get the .torrent file.

I think this is a false positive as far as it actually being a timeout since I set snatchTimeout: "30sec",

And my manual test of the supposedly timed out URL takes less than 5 seconds to get me a .torrent.

I'm wondering what else I can do to debug the actual cause.

Even the verbose log isn't much help, as it only has these few lines. I have no issues with other trackers for snatch requests in cross-seed. I also doubt it's a config issue with the Torznab config entry as the searches would also fail, and they succeed. I manually confirmed on the tracker that two results coming back when searching by he file name, and they are the two that cross--seed is claiming are timing out on snatch.

I'm running both cross-seed and QBittorrent in Docker and all volume mounts use absolute paths.

2023-12-01 00:44:45 info: [torznab] Searching 3 indexers for The.Last.Of.Us.S01E02.2160p.HMAX.WEB-DL.DDP5.1.HDR.x265-NTb
2023-12-01 00:44:45 verbose: [torznab] http://prowlarr.lan:9696/6/api?apikey=[REDACTED]&t=tvsearch&q=The+Last+Of+Us&season=1&ep=2
2023-12-01 00:44:45 verbose: [torznab] http://prowlarr.lan:9696/5/api?apikey=[REDACTED]&t=tvsearch&q=The+Last+Of+Us&season=1&ep=2
2023-12-01 00:44:45 verbose: [torznab] http://prowlarr.lan:9696/9/api?apikey=[REDACTED]&t=tvsearch&q=The+Last+Of+Us&season=1&ep=2
2023-12-01 00:45:01 verbose: [decide] The.Last.Of.Us.S01E02.2160p.HMAX.WEB-DL.DDP5.1.HDR.x265-NTb - no match for TorrentLeech torrent The Last Of Us S01E02 2160p HMAX WEB-DL DDPA5 1 HDR10 DV HEVC-SMURF - it has a different file tree
2023-12-01 00:45:08 verbose: [decide] The.Last.Of.Us.S01E02.2160p.HMAX.WEB-DL.DDP5.1.HDR.x265-NTb - no match for TorrentLeech torrent The Last of Us S01E02 Infected 2160p HMAX WEB-DL DDP5 1 Atmos DV HDR H 265-playWEB - it has a different file tree
2023-12-01 00:45:10 verbose: [decide] The.Last.Of.Us.S01E02.2160p.HMAX.WEB-DL.DDP5.1.HDR.x265-NTb - no match for TorrentLeech torrent The Last Of Us S01E02 2160p HMAX WEB-DL DDPA5 1 HDR HEVC-NTb - it has a different file tree
2023-12-01 00:45:14 verbose: [decide] The.Last.Of.Us.S01E02.2160p.HMAX.WEB-DL.DDP5.1.HDR.x265-NTb - no match for TorrentLeech torrent The Last Of Us S01E02 2160p HMAX WEB-DL DDPA5 1 DV HEVC-SMURF - it has a different file tree
2023-12-01 00:45:16 error: snatching http://prowlarr.lan:9696/5/download?apikey=[REDACTED]&link=QWYyRGY4aXFFY2NHNHJzbTZUSnhhdjZ4Rlp5b1h0S0lyVjNMTW1Fc3pQZHAvMUYwR1E4S1JsTXpyQU04QWxWWTZNejB6eWNmNXU1cmdZVk9SSmZTYW9PLzQ1cnN4dVRPNHY2bUNoWjZhYW9XTFU3MWk0emVXYVhINzhCa2EzTkZ4SEcxRlUrakFKVUtXbWpqNWxaZHhVQUFrWkdnWFNkbjNzNmhWd2JjbjVLL0VHc3lrM2FNWmtVaE1BNmpOV0tp&file=The+Last+Of+Us+S01E02+2160p+HMAX+WEB-DL+DDPA5+1+DV+HEVC-NTb timed out
2023-12-01 00:45:16 verbose: [decide] The.Last.Of.Us.S01E02.2160p.HMAX.WEB-DL.DDP5.1.HDR.x265-NTb - no match for TorrentLeech torrent The Last Of Us S01E02 2160p HMAX WEB-DL DDPA5 1 DV HEVC-NTb - the torrent file failed to download
2023-12-01 00:45:16 error: snatching http://prowlarr.lan:9696/5/download?apikey=[REDACTED]&link=cThPS2wwOTlmZzQ5bUJYRjFHZEtPV1k4Sk5FcTg3UzNET0JRWXBKM3MzTUZtalNPeXpqRngweXBkWmkzZTVNVTdmWkNLelUrNko1OWxMTU9kNzJLUlhvRnEzRG4wb3lqUEJ1WXZYdWtwVEU3aHY3U3NHd2gwQVM5dDRGQi9hT1RjdXRNc1BPbTBaZU9pUG1lVDUveU13Z1ZibzBXNWY0ZG90MHlhZHBNaEVBPQ&file=The+Last+of+Us+S01E02+DV+2160p+WEB+H265-GLHF timed out
2023-12-01 00:45:16 verbose: [decide] The.Last.Of.Us.S01E02.2160p.HMAX.WEB-DL.DDP5.1.HDR.x265-NTb - no match for TorrentLeech torrent The Last of Us S01E02 DV 2160p WEB H265-GLHF - the torrent file failed to download
2023-12-01 05:36:56 verbose: [configdump] {
  torznab: [
    'http://prowlarr.lan:9696/6/api?apikey=[REDACTED]',
    'http://prowlarr.lan:9696/5/api?apikey=[REDACTED]',
    'http://prowlarr.lan:9696/9/api?apikey=[REDACTED]'
  ],
  matchMode: 'safe',
  dataCategory: 'cross-seed-data',
  linkType: 'symlink',
  skipRecheck: false,
  maxDataDepth: 2,
  torrentDir: '/torrents',
  outputDir: '/cross-seeds',
  includeNonVideos: false,
  includeEpisodes: false,
  includeSingleEpisodes: true,
  fuzzySizeThreshold: 0.02,
  verbose: false,
  action: 'inject',
  qbittorrentUrl: 'http://redacted:[REDACTED]@qbt.lan:8080',
  duplicateCategories: true,
  delay: 6,
  snatchTimeout: 30000,
  searchTimeout: 30000,
  searchLimit: 0
}
zakkarry commented 6 months ago

Is this all prowlarr indexers or just one? im on mobile so cant really read logs well.

I had a similar issue in my dev environment, i redid the whole config from the template and it fixed it. still dont know what the issue was.

if you want to try this, dont forget to clear-cache if it doesn't work the first time.

Still not sure what changed, but itnwas doing the timeouts on all indexers across two vms with that config.

will follow up when im home from work and can look at logs.

mmgoodnow commented 6 months ago

where does it say it timed out? all I see is "the torrent file failed to download"

mmgoodnow commented 6 months ago

ah nvm found it

mmgoodnow commented 6 months ago

My hunch is that snatches might be getting rate limited by Prowlarr - I think Prowlarr has a queueing behavior where if you make 10 concurrent API calls through Prowlarr, it will not send them through to the upstream all at once, but rather it'll send the first one and then make the other 9 wait. In this case, I see that several are coming from the same indexer, which might be the problem. You could try to reproduce this by running a bunch of curl requests in parallel with curl url1 & curl url2 & curl url3 etc.

zakkarry commented 6 months ago

@mynameisbogdan would know

mynameisbogdan commented 6 months ago

Prowlarr has a queueing behavior where if you make 10 concurrent API calls through Prowlarr, it will not send them through to the upstream all at once, but rather it'll send the first one and then make the other 9 wait.

This is correct, kind sir.

Any details on the indexer's name and logs from prowlarr?

mmgoodnow commented 6 months ago

I wonder how much slower it would make things to do snatches serially instead of all at once - it'd likely not be as bad as it would seem due to this Prowlarr queueing behavior effectively doing it for us.

mynameisbogdan commented 6 months ago

WDYM all at once? Current behavior one by one is working fine with the delay. I didn't checked, but I think sending all the requests at once won't play that well with Prowlarr's rate limit.

mmgoodnow commented 6 months ago

Sorry, I'm saying on cross-seed's end. We issue snatch requests for everything all at the same time

mmgoodnow commented 6 months ago

Cross-seed's current behavior is to issue search requests in parallel to all trackers for a single release name, then once all of the search results come back for that release name, we issue snatch requests in parallel for all search results that look promising (the size is close).

Regarding Prowlarr's queueing, could you help clarify these points?

Relevant code: https://github.com/cross-seed/cross-seed/blob/7ba06fa4d994e07f81b317c54417ea55a9b876eb/src/pipeline.ts#L97

mynameisbogdan commented 6 months ago
  • Are both searches and snatches queued?

Yes, both types are rate limited to a minimum of 2s/request. Coupled with Query and Grabs limits as well.

  • Is the search queue global or per-indexer?

Per host, since some have the same site added multiple times.

Is it useful to issue searches to N different indexers in parallel, or will N-1 of them get paused?

It's okay to send them at once, since they'll be rate limited.

Sending 6 requests in the same time:

2023-12-17 05:07:56.0|Trace|RateLimitService|Rate Limit triggered, delaying 'host.io' for 1.999 sec
2023-12-17 05:07:56.0|Trace|RateLimitService|Rate Limit triggered, delaying 'host.io' for 3.998 sec
2023-12-17 05:07:56.0|Trace|RateLimitService|Rate Limit triggered, delaying 'host.io' for 5.998 sec
2023-12-17 05:07:56.0|Trace|RateLimitService|Rate Limit triggered, delaying 'host.io' for 7.997 sec
2023-12-17 05:07:56.0|Trace|RateLimitService|Rate Limit triggered, delaying 'host.io' for 9.997 sec

I think issues might appear on cross-seed side when a rate limit delay is too big in Prowlarr, when triggering a timeout.

  • Is the snatch queue global or per-indexer?

Per host, same as searches they use the same Rate Limiter.

mmgoodnow commented 6 months ago

Thanks for the clarifications! I'm building a good mental model of how this works.

Per host, same as searches they use the same Rate Limiter.

Does this mean that issuing a snatch and a search at the same time would delay one of them? (I'm assuming yes)

mynameisbogdan commented 6 months ago

Yes, you're assuming this correctly.

plz12345 commented 6 months ago

So I'd need to reproduce, but confirm Prowlarr's logs? IIRC, the log for that indexer was successfully grabbing the torrent file from the indexer, and I was assuming something was happening in cross-seed triggering a "timeout" (I thought this might be a red herring of an error). In this case, the indexer was alpharatio, but I do see successful cross-seeds on it in my download client, so I'm assuming this issue was specific to a couple torrents, or transient.

Checking cross-seed's verbose log for today, I don't see any timeouts, so it looks like it did eventually snatch these and set them up for seeding. Otherwise it'd still be retrying.

mmgoodnow commented 6 months ago

I think we figured it out, no need for additional reproduction

Greatdane commented 5 months ago

Can I ask how you figured this out? I am seeing the same, 

error: snatching http://192.168.0.200:9696/4/download?apikey=[REDACTED]&link=dGR1NDdwTUp5MUUxc21wQmFWOFRPTHpwcnc3bG9Yby9yL0VmSEFxeXVDZVdZZXJNUHJ5SFY4bzRjOVJSQXhJb2p0NThmZmpLS3crdU1DUXN6NnFWNVNIUzhLQys2a0ZFa2R6b1JYSXhoN3BVTjM5SHhORUtra1ZSeUdSMHY2Z0U&file=Migration+2023+1080p+MA+WEB-DL+DD%2B+5.1+Atmos+H.264-FLUX timed out

If I go into Prowlarr, I can search and get the torrent fine without issue. If I add the URL to a browser directly, it will download the torrent without issue, so not sure why its timing out with cross-seed

This is my config;

module.exports = {
    delay: 90,
    torznab: [
        http://192.168.0.200:9696/1/api?apikey=REDACTED, 
        http://192.168.0.200:9696/2/api?apikey=REDACTED,
        http://192.168.0.200:9696/3/api?apikey=REDACTED,
        http://192.168.0.200:9696/4/api?apikey=REDACTED,
        http://192.168.0.200:9696/5/api?apikey=REDACTED,
        http://192.168.0.200:9696/6/api?apikey=REDACTED,
        http://192.168.0.200:9696/7/api?apikey=REDACTED,
        http://192.168.0.200:9696/8/api?apikey=REDACTED,
        http://192.168.0.200:9696/9/api?apikey=REDACTED,
        http://192.168.0.200:9696/10/api?apikey=REDACTED,
        http://192.168.0.200:9696/11/api?apikey=REDACTED,
        http://192.168.0.200:9696/12/api?apikey=REDACTED,
        http://192.168.0.200:9696/13/api?apikey=REDACTED
    ],
    dataDirs: ["/media/Anime.Movies", "/media/Anime.TV", "/media/Kids.Movies", "/media/Kids.TV", "/media/Kumi.Movies",  "/media/Kumi.TV", "/media/Movies.4K", "/media/Movies.HD", "/media/TV", "/media/TV.4K"],
    matchMode: "safe",
    dataCategory: undefined,
    linkDir: "/downloads/seedlinks",
    linkType: "symlink",
    skipRecheck: true,
    maxDataDepth: 3,
    torrentDir: "/torrents",
    outputDir: "/cross-seeds",
    includeEpisodes: true,
    includeSingleEpisodes: true,
    includeNonVideos: true,
    fuzzySizeThreshold: 0.02,
    excludeOlder: undefined,
    excludeRecentSearch: "52w",
    action: "inject",
    rtorrentRpcUrl: undefined,
    qbittorrentUrl: "http://user:pass@192.168.0.200:8080",
    transmissionRpcUrl: undefined,
    duplicateCategories: false,
    notificationWebhookUrl: undefined,
    port: 2468,
    host: undefined,
    rssCadence: "4w",
    searchCadence: "26w",
    snatchTimeout: "90sec",
    searchTimeout: "90sec",
    searchLimit: undefined,
    apiAuth: true,
};
zakkarry commented 5 months ago

We addressed a possible cause of this a few versions back, are you sure you are on 5.9.2?

Greatdane commented 5 months ago

Ah, thanks. I see I am on v5.8.5, which Unraid was saying was the latest. But after some refreshing, I can see an update and applied it. Looks to have fixed the issue!

zakkarry commented 5 months ago

Ah, thanks. I see I am on v5.8.5, which Unraid was saying was the latest. But after some refreshing, I can see an update and applied it. Looks to have fixed the issue!

Make sure you're using the official tagged container I have on the community apps page. its on ambipro's repository.

The old unofficial one is linked to an old mirror.

Greatdane commented 5 months ago

Make sure you're using the official tagged container I have on the community apps page. its on ambipro's repository. The old unofficial one is linked to an old mirror.

Ah, I am using the one in LTM's repository - is that one no longer getting official updates? I assume changing to the other app won't require any config changes, just remap the paths when installing?

zakkarry commented 5 months ago

That's an unofficial one, and there are a few container settings (such as running in privileged mode by default) that I disabled on the official one, as it wasn't necessary and it uses the official container repository.

It shouldn't make a huge difference with the repo, but it could at some point. Better to address it now than run into issues if it ever stops being updated downstream.

Edit: No, your config will remain the same. Just carry over your path mappings (and maybe port if non-standard) to the appropriate spot.

2fst4u commented 3 months ago

I'm on 5.9.2 and I've found 11 indexers seems to be the limit for what cross-seed and prowlarr can handle. More than that and I get a cascading negative feedback loop of timeouts causing more and more issues as they go on.

Are those prowlarr efficiencies still a possibility to come in future?

zakkarry commented 3 months ago

I'm on 5.9.2 and I've found 11 indexers seems to be the limit for what cross-seed and prowlarr can handle. More than that and I get a cascading negative feedback loop of timeouts causing more and more issues as they go on.

Are those prowlarr efficiencies still a possibility to come in future?

I've seen vastly more than 11 indexers used in cross-seed and prowlarr both, so these sound like inefficiencies in your specific configuration/setup rather than anything attributed to the programs themselves.

The prowlarr logs would tell you what is really going on.