search

rogerfar / rdt-client

Real-Debrid Client Proxy
MIT License
907 stars 113 forks source link

Rdtclient will download all files despite of size limite or exclude filter #429

Closed aburchio closed 7 months ago

aburchio commented 8 months ago

What version are you using? 2.0.6.3

Wat OS are you running? Debian 12

Are you using Docker or as a service? Docker

Which debrid provider are you using? Real-Debrid

Which downloader are you using? Radarr

I have been struggling with this issue since many versions, rdtclient downloading all files in a torrent (small file like srt. nfo .txt), I understood how to use the "minimum size" by trial and error and have set it up to 25 and usually works but, let's say 5 torrent out of 10 will still download any 1k .nfo or 20k .srt files. 

2024-03-06 22:10:04.612 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Determining which files are over 26214400 bytes for torrent Sleepless.2001.1080p.BluRay.x264.DTS-FGT (KENTF3JATPD5O - waiting_files_selection 0%) (4ca3af79-9097-4900-8119-bd49976d3d43)
2024-03-06 22:10:04.612 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Found 0 files that match the minimum file size criterea for torrent Sleepless.2001.1080p.BluRay.x264.DTS-FGT (KENTF3JATPD5O - waiting_files_selection 0%) (4ca3af79-9097-4900-8119-bd49976d3d43)
2024-03-06 22:10:04.612 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Filtered all files out! Downloading ALL files instead! for torrent Sleepless.2001.1080p.BluRay.x264.DTS-FGT (KENTF3JATPD5O - waiting_files_selection 0%) (4ca3af79-9097-4900-8119-bd49976d3d43)
2024-03-06 22:10:04.613 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Selecting files:
2024-03-06 22:10:04.613 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: 1: /RARBG.txt (31b)
2024-03-06 22:10:04.613 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: 2: /Sleepless.2001.1080p.BluRay.x264.DTS-FGT.mkv (9241652798b)

I have seens the new "include" and "exclude" options but again, I can not find any example of a working setup. I should be a regular expression so I am expecting that this:

.*\.(jpg|txt|nfo|bmp|png|srt|exe)

would stop any unwanted file, but I am still experiencing the same issue. Seems that size check take precedence and include/exclude rules are not processed. Also I am using only exclude rule and not both, as stated. I am surely missing something but I need a direction to figure it out.

rogerfar commented 8 months ago

Can you post a log of when this happens.

aburchio commented 8 months ago

I posted a log above, the relevant part being

Determining which files are over 26214400 bytes for torrent
Found 0 files that match the minimum file size criterea for torrent
Filtered all files out! Downloading ALL files instead! for torrent 
Selecting files:
RARBG.txt (31b)

unfortunately I set up debug to trace down this issue, and rdtclient produces a very large amount of logs which are rotated very fast. I increased the size and will monitor for future log entries, for now this is the only relevant one but I have seen it happen many times. Another strange thing: to resolve the issue, I do the following: ask rdtclient to "Retry torrent" and immediately refresh the real debrid torrent web page, so I can intercept the real debrid windows asking for file selection. The thing is that in this windows the small files are already not selected, which is good. If I let rdtclient "Retry torrent" and do it on its own, it will still download the 31b file.

About the exclude rule: I set i up and saved but rules were not applied until I restarted the docker container, now I can see entries in the logs. Maybe I could switch from size to file type filter if it solves my problem.

aburchio commented 8 months ago

Another case, in the first log we have a "Found 0 files that match the minimum file size" (which is not true) and then a "Filtered all files out! Downloading ALL files instead!". No exclude or include rule is processed. A 31b .txt file is being downloaded.


2024-03-07 12:35:34.337 +01:00 [DBG] RdtClient.Service.Services.TorrentRunner: Processing for torrent The.Devil.and.Holy.Water.1983.ITALIAN.1080p.NF.WEBRip.DDP2.0.x264-PAAI (REXTWGCETTLWC - waiting_files_selection 0%) (2eeba59f-e2b6-46b5-b667-ea2d7a484ef4)
2024-03-07 12:35:34.337 +01:00 [DBG] RdtClient.Service.Services.TorrentRunner: Selecting files for torrent The.Devil.and.Holy.Water.1983.ITALIAN.1080p.NF.WEBRip.DDP2.0.x264-PAAI (REXTWGCETTLWC - waiting_files_selection 0%) (2eeba59f-e2b6-46b5-b667-ea2d7a484ef4)
2024-03-07 12:35:34.423 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Seleting files for torrent The.Devil.and.Holy.Water.1983.ITALIAN.1080p.NF.WEBRip.DDP2.0.x264-PAAI (REXTWGCETTLWC - waiting_files_selection 0%) (2eeba59f-e2b6-46b5-b667-ea2d7a484ef4)
2024-03-07 12:35:34.423 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Determining which files are already available on RealDebrid for torrent The.Devil.and.Holy.Water.1983.ITALIAN.1080p.NF.WEBRip.DDP2.0.x264-PAAI (REXTWGCETTLWC - waiting_files_selection 0%) (2eeba59f-e2b6-46b5-b667-ea2d7a484ef4)
2024-03-07 12:35:34.521 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Found 2/2 available files on RealDebrid for torrent The.Devil.and.Holy.Water.1983.ITALIAN.1080p.NF.WEBRip.DDP2.0.x264-PAAI (REXTWGCETTLWC - waiting_files_selection 0%) (2eeba59f-e2b6-46b5-b667-ea2d7a484ef4)
2024-03-07 12:35:34.522 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Selecting 0/2 files for torrent The.Devil.and.Holy.Water.1983.ITALIAN.1080p.NF.WEBRip.DDP2.0.x264-PAAI (REXTWGCETTLWC - waiting_files_selection 0%) (2eeba59f-e2b6-46b5-b667-ea2d7a484ef4)
2024-03-07 12:35:34.522 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Determining which files are over 26214400 bytes for torrent The.Devil.and.Holy.Water.1983.ITALIAN.1080p.NF.WEBRip.DDP2.0.x264-PAAI (REXTWGCETTLWC - waiting_files_selection 0%) (2eeba59f-e2b6-46b5-b667-ea2d7a484ef4)
2024-03-07 12:35:34.522 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Found 0 files that match the minimum file size criterea for torrent The.Devil.and.Holy.Water.1983.ITALIAN.1080p.NF.WEBRip.DDP2.0.x264-PAAI (REXTWGCETTLWC - waiting_files_selection 0%) (2eeba59f-e2b6-46b5-b667-ea2d7a484ef4)
2024-03-07 12:35:34.522 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Filtered all files out! Downloading ALL files instead! for torrent The.Devil.and.Holy.Water.1983.ITALIAN.1080p.NF.WEBRip.DDP2.0.x264-PAAI (REXTWGCETTLWC - waiting_files_selection 0%) (2eeba59f-e2b6-46b5-b667-ea2d7a484ef4)
2024-03-07 12:35:34.522 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: 2: /The.Devil.and.Holy.Water.1983.ITALIAN.1080p.NF.WEB-DL.DDP2.0.x264-PAAI.mkv (5465482618b)
2024-03-07 12:35:34.522 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient:  for torrent The.Devil.and.Holy.Water.1983.ITALIAN.1080p.NF.WEBRip.DDP2.0.x264-PAAI (REXTWGCETTLWC - waiting_files_selection 0%) (2eeba59f-e2b6-46b5-b667-ea2d7a484ef4)
2024-03-07 12:35:36.526 +01:00 [DBG] RdtClient.Service.Services.TorrentRunner: Currently 0 queued downloads and 0 total active downloads for torrent The.Devil.and.Holy.Water.1983.ITALIAN.1080p.NF.WEBRip.DDP2.0.x264-PAAI (REXTWGCETTLWC - downloading 0%) (2eeba59f-e2b6-46b5-b667-ea2d7a484ef4)

Here we have a "size rule" correctly matching, and then a regular expression exclude rule applied. 

2024-03-07 12:28:53.084 +01:00 [DBG] RdtClient.Service.Services.TorrentRunner: Processing for torrent X-Men - Apocalisse (2016) - 1080p EXTRA iCV-MIRCrew (L7U2F3IVTJPTS - waiting_files_selection 0%) (24c890e8-1de0-44f5-80d4-702da3114a78)
2024-03-07 12:28:53.084 +01:00 [DBG] RdtClient.Service.Services.TorrentRunner: Selecting files for torrent X-Men - Apocalisse (2016) - 1080p EXTRA iCV-MIRCrew (L7U2F3IVTJPTS - waiting_files_selection 0%) (24c890e8-1de0-44f5-80d4-702da3114a78)
2024-03-07 12:28:53.283 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Seleting files for torrent X-Men - Apocalisse (2016) - 1080p EXTRA iCV-MIRCrew (L7U2F3IVTJPTS - waiting_files_selection 0%) (24c890e8-1de0-44f5-80d4-702da3114a78)
2024-03-07 12:28:53.283 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Determining which files are already available on RealDebrid for torrent X-Men - Apocalisse (2016) - 1080p EXTRA iCV-MIRCrew (L7U2F3IVTJPTS - waiting_files_selection 0%) (24c890e8-1de0-44f5-80d4-702da3114a78)
2024-03-07 12:28:53.420 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Found 8/8 available files on RealDebrid for torrent X-Men - Apocalisse (2016) - 1080p EXTRA iCV-MIRCrew (L7U2F3IVTJPTS - waiting_files_selection 0%) (24c890e8-1de0-44f5-80d4-702da3114a78)
2024-03-07 12:28:53.422 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Selecting 8/8 files for torrent X-Men - Apocalisse (2016) - 1080p EXTRA iCV-MIRCrew (L7U2F3IVTJPTS - waiting_files_selection 0%) (24c890e8-1de0-44f5-80d4-702da3114a78)
2024-03-07 12:28:53.422 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Determining which files are over 26214400 bytes for torrent X-Men - Apocalisse (2016) - 1080p EXTRA iCV-MIRCrew (L7U2F3IVTJPTS - waiting_files_selection 0%) (24c890e8-1de0-44f5-80d4-702da3114a78)
2024-03-07 12:28:53.422 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Found 8 files that match the minimum file size criterea for torrent X-Men - Apocalisse (2016) - 1080p EXTRA iCV-MIRCrew (L7U2F3IVTJPTS - waiting_files_selection 0%) (24c890e8-1de0-44f5-80d4-702da3114a78)
2024-03-07 12:28:53.423 +01:00 [DBG] RdtClient.Service.Services.TorrentClients.RealDebridTorrentClient: Using regular expression  to ignore files matching this regex for torrent X-Men - Apocalisse (2016) - 1080p EXTRA iCV-MIRCrew (L7U2F3IVTJPTS - waiting_files_selection 0%) (24c890e8-1de0-44f5-80d4-702da3114a78)
rogerfar commented 8 months ago

It works as intended, if it filters out all files because of a size or regex rule, it will simply download all files.

aburchio commented 8 months ago

Ok. It seemed to me that a size rule of 26214400 bytes not matching a 31 bytes file and another 9241652798 bytes causing both to download, was not working as intended, this is why I opened the issue. What surprises me is that most of the times just works. I suspect it could be a symptom of a problem in the real debrid cache but I would not know how to investigate it.

Wolokin commented 7 months ago

I have the same issue, I've set the minimum file size to 1MB, but sometimes that matches no files and, as a result, rdt-client tries to download all files (which then fails since the .txts are not seeded):

[10:50:15 DBG] Processing 1 torrents
[10:50:15 DBG] Currently 0 queued downloads and 0 total active downloads for torrent <TORRENT_NAME> (<HASH> - waiting_files_selection 0%) (<ID>)
[10:50:15 DBG] Processing for torrent <TORRENT_NAME> (<HASH> - waiting_files_selection 0%) (<ID>)
[10:50:15 DBG] Selecting files for torrent <TORRENT_NAME> (<HASH> - waiting_files_selection 0%) (<ID>)
[10:50:15 DBG] Seleting files for torrent <TORRENT_NAME> (<HASH> - waiting_files_selection 0%) (<ID>)
[10:50:15 DBG] Determining which files are already available on RealDebrid for torrent <TORRENT_NAME> (<HASH> - waiting_files_selection 0%) (<ID>)
[10:50:15 DBG] Updating torrent info from debrid provider
[10:50:15 DBG] Found 22/22 available files on RealDebrid for torrent <TORRENT_NAME> (<HASH> - waiting_files_selection 0%) (<ID>)
[10:50:15 DBG] Selecting 0/22 files for torrent <TORRENT_NAME> (<HASH> - waiting_files_selection 0%) (<ID>)
[10:50:15 DBG] Determining which files are over 1048576 bytes for torrent <TORRENT_NAME> (<HASH> - waiting_files_selection 0%) (<ID>)
[10:50:15 DBG] Found 0 files that match the minimum file size criterea for torrent <TORRENT_NAME> (<HASH> - waiting_files_selection 0%) (<ID>)
[10:50:15 DBG] Filtered all files out! Downloading ALL files instead! for torrent <TORRENT_NAME> (<HASH> - waiting_files_selection 0%) (<ID>)
[10:50:15 DBG] Selecting files:
[10:50:15 DBG] 1: /<FILE_NAME> (2301259732b)
[10:50:15 DBG] 2: /<FILE_NAME> (2418521289b)
[10:50:15 DBG] 3: /<FILE_NAME> (2443343076b)
[10:50:15 DBG] 4: /<FILE_NAME> (2415281163b)
[10:50:15 DBG] 5: /<FILE_NAME> (2422632632b)
[10:50:15 DBG] 6: /<FILE_NAME> (2415202572b)
[10:50:15 DBG] 7: /<FILE_NAME> (2403269273b)
[10:50:15 DBG] 8: /<FILE_NAME> (2353468273b)
[10:50:15 DBG] 9: /<FILE_NAME> (2387615939b)
[10:50:15 DBG] 10: /<FILE_NAME> (2395343125b)
[10:50:15 DBG] 11: /<FILE_NAME> (2385598239b)
[10:50:15 DBG] 12: /<FILE_NAME> (2394812081b)
[10:50:15 DBG] 13: /<FILE_NAME> (2448105611b)
[10:50:15 DBG] 14: /<FILE_NAME> (2434930891b)
[10:50:15 DBG] 15: /<FILE_NAME> (2396717459b)
[10:50:15 DBG] 16: /<FILE_NAME> (2436730080b)
[10:50:15 DBG] 17: /<FILE_NAME> (2380377070b)
[10:50:15 DBG] 18: /<FILE_NAME> (2399578690b)
[10:50:15 DBG] 19: /<FILE_NAME> (1988399957b)
[10:50:15 DBG] 20: /<FILE_NAME> (2489581910b)
[10:50:15 DBG] 21: /<FILE_NAME> (4899646942b)
[10:50:15 DBG] 22: /RARBG.txt (31b)
[10:50:15 DBG]  for torrent <TORRENT_NAME> (<HASH> - waiting_files_selection 0%) (<ID>)
[10:50:16 DBG] Finished updating torrent info from debrid provider, next update in 10 seconds

EDIT: I dug around in the options a bit, and after disabling Only download available files on debrid provider in the *arr section, this particular torrent got added correctly through sonarr (the RARBG.txt file was ignored and everything else got downloaded)

rogerfar commented 7 months ago

I'm going to close this it seems to work as intended.

As your example shows, if you are filtering out all files, it will simply revert to downloading everything.

kstanleyadams commented 3 weeks ago

@rogerfar if the exclude filter matches all files it would be good to have the option to reject the download entirely and mark it as failed. This is a useful technique to filter out malware based on unwanted file extensions. What do you think?