search

rembo10 / headphones

Automatic music downloader for SABnzbd
GNU General Public License v3.0
3.41k stars 600 forks source link

Pirate Bay Not Finding Results #3305

Open wymangr opened 2 years ago

wymangr commented 2 years ago

I have been using headphones for a while now and it has worked in the past and I've always used pirate bay as my search provider. However, it hasn't found any new music in quite a long time.

Headphones Version: 83398cb10231087399e284daf9245dbfedbfd005 Pirate Bay Proxy URL: Tried blank (Default) and https://thepiratebay.org

Logs (Using RHCP as example): 26-May-2022 09:29:43 - INFO :: ('CP Server Thread-11',) : Marking album: 0580e402-e6e8-4b5b-894a-399a10c63adb as wanted... 26-May-2022 09:29:43 - INFO :: ('CP Server Thread-11',) : Searching for wanted albums 26-May-2022 09:29:43 - INFO :: ('CP Server Thread-11',) : Searching for "Red Hot Chili Peppers - Unlimited Love" since it was marked as wanted 26-May-2022 09:29:43 - DEBUG :: ('CP Server Thread-11',) : Using search term: Red Hot Chili Peppers Unlimited Love 26-May-2022 09:29:43 - INFO :: ('CP Server Thread-11',) : Searching The Pirate Bay using term: Red Hot Chili Peppers Unlimited Love 26-May-2022 09:29:43 - DEBUG :: ('CP Server Thread-11',) : Requesting URL via GET method: https://thepiratebay.org/search/Red Hot Chili Peppers Unlimited Love/0/7/101 26-May-2022 09:29:45 - INFO :: ('CP Server Thread-11',) : No results found from The Pirate Bay using term: Red Hot Chili Peppers Unlimited Love 26-May-2022 09:29:45 - INFO :: ('CP Server Thread-11',) : No more results found for: Red Hot Chili Peppers - Unlimited Love

Using the requesting URL, I see multiple results in the browser:

image

wymangr commented 2 years ago

From what I can tell, this is the HTML that is stored in the data variable when is searches: data = request.request_soup(url=providerurl + category, headers=headers)

And looking at that, it appears that maybe Pirate Bay has switched to rendering it's result table to javascript and the requests library is unable to get the HTML of the table? 

<!DOCTYPE html>
<html lang="en">
   <head>
      <meta charset="utf-8"/>
      <meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport"/>
      <title>The Pirate Bay - The galaxy's most resilient bittorrent site</title>
      <link href="https://torrindex.net/static/normalize.css" rel="stylesheet" type="text/css"/>
      <link href="https://torrindex.net/static/tpb.css" rel="stylesheet" type="text/css"/>
      <link href="//apibay.org/" rel="dns-prefetch"/>
      <link href="//torrindex.net/" rel="dns-prefetch"/>
      <link href="//ilyf4amifh.com/" rel="dns-prefetch"/>
      <link href="//abservinean.com/" rel="dns-prefetch"/>
      <link href="//a.exdynsrv.com/" rel="dns-prefetch"/>
      <link href="//a.exosrv.com/" rel="dns-prefetch"/>
      <link href="//cdn.engine.spotscenered.info/" rel="dns-prefetch"/>
      <link href="//syndication.exdynsrv.com/" rel="dns-prefetch"/>
      <link href="//d1f05vr3sjsuy7.cloudfront.net/" rel="dns-prefetch"/>
      <script>var country='CH';</script>
      <script src="https://thepiratebay.org/static/main.js"></script>
      <script src="https://torrindex.net/static/tinysort.min.js"></script>
      <script async="" data-cfasync="false" src="//ilyf4amifh.com/pn21ywqw/z/sc/scssx/1846387/lib.js" type="text/javascript"></script>
      <script data-cfasync="false" src="//d1f05vr3sjsuy7.cloudfront.net/?srvfd=908284"></script>
   </head>
   <body id="browse" onload="jswarnclear()">
      <main>
         <b><font color="RED"><label id="jscrwarn">Enable JS in your browser!</label></font></b>
         <script>document.getElementById("jscrwarn").innerHTML='';</script>
         <b><font color="RED"><label id="jscrwarn2">You may be blocking important javascript components, check that main.js is loaded or the webpage won't work.</label></font></b>
         <h1><label id="tlt"></label></h1>
         <div class="adblock" id="ad-bottom">
            <div class="ad728 align-center">
               <a href="https://ttf.trmobc.com/aff_ad?campaign_id=6&amp;aff_id=460" style="text-decoration:none; border-bottom:none; color:none;"><img src="https://torrindex.net/images/epv/728t.png"/></a> 
            </div>
            <div class="ad468 align-center">
               <a href="https://ttf.trmobc.com/aff_ad?campaign_id=6&amp;aff_id=460" style="text-decoration:none; border-bottom:none; color:none;"><img src="https://torrindex.net/images/epv/468.png" style="text-decoration:none; border-bottom:none; color:none;"/></a> 
            </div>
            <div class="ad234 align-center">
               <script type="application/javascript">
                  var ad_idzone = "3804619",
                  ad_width = "300",
                  ad_height = "100"
               </script>
               <script src="https://a.exdynsrv.com/ads.js" type="application/javascript"></script>
            </div>
         </div>
         <div class="browse">
            <section class="col-left ad120">
               <a href="https://ttf.trmobc.com/aff_ad?campaign_id=6&amp;aff_id=460" style="text-decoration:none; border-bottom:none; color:none;"><img src="https://torrindex.net/images/epv/120.png"/></a> 
            </section>
            <script>print_search()</script>
            <section class="col-right ad120">
               <a href="https://ttf.trmobc.com/aff_ad?campaign_id=6&amp;aff_id=460" style="text-decoration:none; border-bottom:none; color:none;"><img src="https://torrindex.net/images/epv/160.png"/></a> 
            </section>
         </div>
         <div class="adblock" id="ad-bottom">
            <div class="ad728 align-center">
               <a href="https://ttf.trmobc.com/aff_ad?campaign_id=6&amp;aff_id=460" style="text-decoration:none; border-bottom:none; color:none;"><img src="https://torrindex.net/images/epv/728f.png"/></a> 
            </div>
            <div class="ad468 align-center">
               <a href="https://ttf.trmobc.com/aff_ad?campaign_id=6&amp;aff_id=460" style="text-decoration:none; border-bottom:none; color:none;"><img src="https://torrindex.net/images/epv/468.png" style="text-decoration:none; border-bottom:none; color:none;"/></a> 
            </div>
            <div class="ad234 align-center">
               <script type="application/javascript">
                  var ad_idzone = "3804621",
                  ad_width = "300",
                  ad_height = "250"
               </script>
               <script src="https://a.exdynsrv.com/ads.js" type="application/javascript"></script>
            </div>
         </div>
      </main>
      <header class="row">
         <script>print_header1()</script>
         <input id="search" name="q" placeholder="Search here..." title="Pirate Search" type="search"/>
         <script>document.getElementById("search").value = getParameterByName('q')</script>
         <script>print_header2()</script>
         <section class="col-right ad468" id="had468">
            <a href="https://ttf.trmobc.com/aff_ad?campaign_id=6&amp;aff_id=460" style="text-decoration:none; border-bottom:none; color:none;"><img src="https://torrindex.net/images/epv/468.png" style="text-decoration:none; border-bottom:none; color:none;"/></a> 
         </section>
         <section class="col-right ad234" id="had234"></section>
      </header>
      <script>
         print_footer();
         mark_selected();
      </script>
      <script>do_pop()</script>
      <script>do_interstitial()</script>
   </body>
</html>
AdeHub commented 3 months ago

have switched the default to use apibay.org, this is in develop if you want to test

Mikaciu commented 3 months ago

Hello @AdeHub, I have a cloudflare error for apibay.org, do you have it too ?

AdeHub commented 3 months ago

Hello @AdeHub, I have a cloudflare error for apibay.org, do you have it too ?

No, got logs for that?

Could also try proxy https://prbay.top/

Mikaciu commented 3 months ago

Hello @AdeHub, I have a cloudflare error for apibay.org, do you have it too ?

No, got logs for that?

Could also try proxy https://prbay.top/

Yep!

headphones-1           | 15-Aug-2024 14:09:08 - INFO :: ('CP Server Thread-4',) : Searching for "Amaranthe - The Catalyst" since it was marked as wanted
headphones-1           | 15-Aug-2024 14:09:08 - DEBUG :: ('CP Server Thread-4',) : Using search term: Amaranthe The Catalyst
headphones-1           | 15-Aug-2024 14:09:08 - INFO :: ('CP Server Thread-4',) : Searching The Pirate Bay using term: Amaranthe The Catalyst
headphones-1           | 15-Aug-2024 14:09:08 - DEBUG :: ('CP Server Thread-4',) : Requesting URL via GET method: http://apibay.org/q.php?q=Amaranthe The Catalyst&cat=101
headphones-1           | 15-Aug-2024 14:09:08 - ERROR :: ('CP Server Thread-4',) : Unable to connect to remote host. Check if the remote host is up and running.

For prbay, same result unfortunately!

headphones-1           | 15-Aug-2024 14:10:48 - INFO :: ('CP Server Thread-10',) : Searching for "Amaranthe - The Catalyst" since it was marked as wanted
headphones-1           | 15-Aug-2024 14:10:48 - DEBUG :: ('CP Server Thread-10',) : Using search term: Amaranthe The Catalyst
headphones-1           | 15-Aug-2024 14:10:48 - INFO :: ('CP Server Thread-10',) : Searching The Pirate Bay using term: Amaranthe%20The%20Catalyst
headphones-1           | 15-Aug-2024 14:10:48 - DEBUG :: ('CP Server Thread-10',) : Requesting URL via GET method: https://prbay.top/search/Amaranthe%20The%20Catalyst/0/7/101
headphones-1           | 15-Aug-2024 14:10:48 - ERROR :: ('CP Server Thread-10',) : Unable to connect to remote host. Check if the remote host is up and running.
headphones-1           | 15-Aug-2024 14:10:48 - INFO :: ('CP Server Thread-10',) : Search for wanted albums complete
Mikaciu commented 3 months ago

I tried another proxy and I have the same error: 

headphones-1           | 15-Aug-2024 14:14:13 - INFO :: ('CP Server Thread-2',) : Searching for "Amaranthe - Manifest" since it was marked as wanted
headphones-1           | 15-Aug-2024 14:14:13 - DEBUG :: ('CP Server Thread-2',) : Using search term: Amaranthe Manifest
headphones-1           | 15-Aug-2024 14:14:13 - INFO :: ('CP Server Thread-2',) : Searching The Pirate Bay using term: Amaranthe%20Manifest
headphones-1           | 15-Aug-2024 14:14:13 - DEBUG :: ('CP Server Thread-2',) : Requesting URL via GET method: https://tprbay.xyz/search/Amaranthe%20Manifest/0/7/101
headphones-1           | 15-Aug-2024 14:14:13 - ERROR :: ('CP Server Thread-2',) : Unable to connect to remote host. Check if the remote host is up and running.
headphones-1           | 15-Aug-2024 14:14:13 - INFO :: ('CP Server Thread-2',) : Search for wanted albums complete

Although if you check the URL, you can see that there is one result for this search :(

AdeHub commented 3 months ago

I am not getting that error, apibay, prbay,tprbay all working for me

TPB seems to be pretty flaky, have you looked at using Torznab with Jackett/Prowlarr

Mikaciu commented 3 months ago

Hello @AdeHub,

I reproduced locally, apibay is working fine when running Headphones using the python script. What I can't understand is why it does not want to connect when using a docker image :(

EDIT: got it! DNS issue.

WOuld you be interested in adding a dockerfile to the repo ? The one from linuxserver is outdated :(