YGG Login issues - No Form Selector found

brewkunz commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing open and closed issues

Current Behavior

Prowlarr fails to access a CloudFlare protected indexer (yggtorrent) despite receiving a successful response from FlareSolverr.

Prowlarr seems to not be able to find the login form in FlareSolverr's response.

Expected Behavior

Prowlarr should be able to actually find the login form in FlareSolverr's response to log into the indexer.

Steps To Reproduce

Add a CloudFlare protected indexer to prowlarr (yggtorrent in this case)
Add FlareSolverr as an indexer proxy for the previously added indexer.
Test the indexer, this should fail.

Environment

- OS: Ubuntu 20.04
- Prowlarr: 1.1.0.2289
- Docker Install: Yes
- Using Reverse Proxy: Yes

- FlareSolverr: 3.0.0, installed on the same machine.

What branch are you running?

Nightly

Trace Logs?

Prowlarr relevant logs :

2023-01-12 20:30:05.2|Debug|FlareSolverr|Cloudflare Detected, Applying FlareSolverr Proxy FlareSolverr to request https://www6.yggtorrent.lol/
2023-01-12 20:30:21.2|Warn|Cardigann|Unable to connect to indexer

[v1.1.0.2289] NzbDrone.Core.Indexers.Definitions.Cardigann.Exceptions.CardigannConfigException: Login failed: No form found on https://www6.yggtorrent.lol/ using form selector #user-login
   at NzbDrone.Core.Indexers.Cardigann.CardigannRequestGenerator.DoLogin() in D:\a\1\s\src\NzbDrone.Core\Indexers\Definitions\Cardigann\CardigannRequestGenerator.cs:line 253
   at NzbDrone.Core.Indexers.Cardigann.Cardigann.DoLogin() in D:\a\1\s\src\NzbDrone.Core\Indexers\Definitions\Cardigann\Cardigann.cs:line 180
   at NzbDrone.Core.Indexers.HttpIndexerBase`1.FetchIndexerResponse(IndexerRequest request) in D:\a\1\s\src\NzbDrone.Core\Indexers\HttpIndexerBase.cs:line 374
   at NzbDrone.Core.Indexers.HttpIndexerBase`1.FetchPage(IndexerRequest request, IParseIndexerResponse parser) in D:\a\1\s\src\NzbDrone.Core\Indexers\HttpIndexerBase.cs:line 285
   at NzbDrone.Core.Indexers.HttpIndexerBase`1.TestConnection() in D:\a\1\s\src\NzbDrone.Core\Indexers\HttpIndexerBase.cs:line 447

FlareSolverr relevant logs (removed response content) :

flaresolverr    | 2023-01-12 21:30:05 INFO     ReqId 140279955842816 127.0.0.1 POST http://localhost:8191/v1 200 OK
flaresolverr    | 2023-01-12 21:30:05 INFO     ReqId 140279738791680 Incoming request => POST /v1 body: {'maxTimeout': 60000, 'cmd': 'request.get', 'url': 'https://www6.yggtorrent.lol/', 'userAgent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36', 'proxy': {}}
flaresolverr    | 2023-01-12 21:30:05 WARNING  ReqId 140279738791680 Request parameter 'userAgent' was removed in FlareSolverr v2.
flaresolverr    | 2023-01-12 21:30:05 DEBUG    ReqId 140279738791680 Launching web browser...
flaresolverr    | 2023-01-12 21:30:07 DEBUG    ReqId 140279730398976 Navigating to... https://www6.yggtorrent.lol/
flaresolverr    | 2023-01-12 21:30:12 INFO     ReqId 140279730398976 Challenge detected. Title found: Just a moment...
flaresolverr    | 2023-01-12 21:30:12 DEBUG    ReqId 140279730398976 Waiting for title: Just a moment...
flaresolverr    | 2023-01-12 21:30:20 DEBUG    ReqId 140279730398976 Waiting for title: DDOS-GUARD
flaresolverr    | 2023-01-12 21:30:20 DEBUG    ReqId 140279730398976 Waiting for selector: #cf-challenge-running
flaresolverr    | 2023-01-12 21:30:20 DEBUG    ReqId 140279730398976 Waiting for selector: .ray_id
flaresolverr    | 2023-01-12 21:30:20 DEBUG    ReqId 140279730398976 Waiting for selector: .attack-box
flaresolverr    | 2023-01-12 21:30:20 DEBUG    ReqId 140279730398976 Waiting for selector: #cf-please-wait
flaresolverr    | 2023-01-12 21:30:20 DEBUG    ReqId 140279730398976 Waiting for selector: #challenge-spinner
flaresolverr    | 2023-01-12 21:30:20 DEBUG    ReqId 140279730398976 Waiting for selector: #trk_jschal_js
flaresolverr    | 2023-01-12 21:30:20 DEBUG    ReqId 140279730398976 Waiting for selector: td.info #js_info
flaresolverr    | 2023-01-12 21:30:20 DEBUG    ReqId 140279730398976 Waiting for redirect
flaresolverr    | 2023-01-12 21:30:20 INFO     ReqId 140279730398976 Challenge solved!
flaresolverr    | 2023-01-12 21:30:21 DEBUG    ReqId 140279738791680 Response => POST /v1 body: {'status': 'ok', 'message': 'Challenge solved!', 'solution': {'url': 'https://www6.yggtorrent.lol/', 'status': 200, 'cookies': [{'domain': '.yggtorrent.lol', 'expiry': 1674333019, 'httpOnly': False, 'name': 'v3_promo_details', 'path': '/', 'secure': False, 'value': 'eyJjb3VudGRvd25fZGF0ZSI6IjAxLzEyLzIwMjMgMjM6NTk6NTkiLCJ0cyI6MTY3MzU2NDM5OX0='}, {'domain': '.yggtorrent.lol', 'expiry': 1673562618, 'httpOnly': True, 'name': 'ygg_', 'path': '/', 'sameSite': 'None', 'secure': True, 'value': 'f5b6kcc7k9aiv1rv487e9a3laui9sdj4'}, {'domain': '.yggtorrent.lol', 'expiry': 1705091418, 'httpOnly': True, 'name': 'cf_clearance', 'path': '/', 'sameSite': 'None', 'secure': True, 'value': '9NfvgnfGp81Uxziyfi2Ubshjz1shY7H0IlX0eVrD2rg-1673555418-0-160'}], 'userAgent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36', 'headers': {}, 'response': '....'}, 'startTimestamp': 1673555405284, 'endTimestamp': 1673555421126, 'version': '3.0.0'}
flaresolverr    | 2023-01-12 21:30:21 INFO     ReqId 140279738791680 Response in 15.842 s
flaresolverr    | 2023-01-12 21:30:21 INFO     ReqId 140279738791680 127.0.0.1 POST http://localhost:8191/v1 200 OK

FlareSolverr relevant part of the html response :

...
<form action="/user/login" method="post" id="user-login">
...

AB#4702

AB#4703

bakerboy448 commented 1 year ago

not a flaresolverr issue
YGG changed their site so the existing definition is broke
YGG hate automation and have been / are taking additional measures to block or otherwise shutdown prowlarr and jackett....including so it's been said banning users for even asking about them or the Starr apps.
trace logs w/ indexer response enabled are needed
YGG also has blocked access from US IPs, so it is increasingly difficult for folks to troubleshoot and fix their continually created issues.

brewkunz commented 1 year ago

Indeed this is not a FlareSolverr issue. The logs I included from FlareSolverr are there to show that it returns a successful response.
They might have changed their homepage a bit but the login form in FlareSolverr's response is still using the html ids defined in the definitions (#user-login)
I included the relevant part of the returned homepage from FlareSolverr (the html form at the bottom of the logs is extracted from the full response to avoid clogging the issue)

To me it looks like there is something going on when fetching/converting the FlareSolverr's response since the login form is actually in the response Prowlarr should be able to find it (I can provide the full response if needed)

bakerboy448 commented 1 year ago

(I can provide the full response if needed)

yes the full response in prowlarr's logs, not the response from FS itself. indexer response logging enabled should have that the response and the ygg page prowlarr gets

ilike2burnthing commented 1 year ago

Unlikely to change anything, but update FlareSolverr to v3.0.2

brewkunz commented 1 year ago

Updated to FlareSolverr v3.0.2 and tried again to test the connection to the indexer. This is still failing even if FlareSolverr is returning a successful response.

Here you'll find the Prowlarr logs with trace enabled :

Edit : Logs are still too long to be included in the comment, I created a gist where you can read it.

bakerboy448 commented 1 year ago

It appears YGG is blocking every single request with Cloudflare & Flaresolverr is not able to continually bypass the blockage.

This appears to be a Flaresolverr issue. @ilike2burnthing That the same conclusion you come to?

Note that these are yet even more actions by YGG admins to block or break automation. They seem to hate automation and the YGG Admins should frankly just request they be removed from jackett and prowlarr if that's what they want - and based on their increased CF security and actions against users for discussing the Starr apps it seems to be what they'd want. It's a shame that the community still financially keep YGG supported given their continual negative behavior.

brewkunz commented 1 year ago

Reading the logs it looks like FlareSolverr resolved the challenge, actually it resolved it twice.

Once line 111 for the request to https://www6.yggtorrent.lol/engine/search?category=2145&name=&description=&file=&uploader=&sub_category=&do=search&order=desc&sort=publish_date
Then a second time line 434 to proxy a request to https://www6.yggtorrent.lol/

In both case there actually is a #form-login element in the response coming from FlareSolverr. Line 438 for the last one. But Prowlarr seems to be failing to parse the response correctly.

The element looks like this in the logs :

<form action=\"/user/login\" method=\"post\" id=(removed)"user-login\">

Regarding ygg and their admins you are absolutely right... I think they stated that CloudFlare was there because the site was frequently the target of DDOS and that CF is the only way to prevent outage. They don't care about the small minority of users like us that use automation tools... I keep using it because unfortunately that's the only one with this amount of french content.

bakerboy448 commented 1 year ago

The flaresolverr response is not what is used. My understanding is FS is only used to get the cookie which should then bypass Cloudflare, but that is not working.

You can see this as well

FS response
Prowlarr tries to auth to YGG again (w/ FS solved cookie)
YGG says 403 and blocks with cloudflare (again)

2023-01-13 12:32:07.5|Trace|HttpClient|Req: [GET] https://www6.yggtorrent.lol/
2023-01-13 12:32:07.5|Trace|ConfigService|Using default config value for 'proxyenabled' defaultValue:'False'
2023-01-13 12:32:07.6|Trace|HttpClient|Res: [GET] https://www6.yggtorrent.lol/: 403.Forbidden (8226 bytes)

brewkunz commented 1 year ago

Indeed you are right ! I completely forgot about that cookie... My bad. So FS is bypassing CF successfully but the cookie provided to Prowlarr is not usable to bypass CF again.

Could there be an issue with my environment ? Prowlarr is running on in its own docker container, and FS in his own too but in network_mode : host.

ilike2burnthing commented 1 year ago

If you haven't already, try disabling IPv6 for both Prowlarr and FlareSolverr, then reconfigure the indexer (so basically logging back in, rather than immediately running a test).

I take it you're not using a VPN or proxy for Prowlarr?

If it's still not working, then as a test you can try YGG in Jackett (again with IPv6 disabled), see if it works there.

If not, you should follow https://github.com/FlareSolverr/FlareSolverr/issues/664, although as @bakerboy448 said, this is likely an issue of how they've configured their Cloudflare IUAM protection as a result of being DDoS'd over the past month of so.

brewkunz commented 1 year ago

If you haven't already, try disabling IPv6 for both Prowlarr and FlareSolverr, then reconfigure the indexer (so basically logging back in, rather than immediately running a test).

I take it you're not using a VPN or proxy for Prowlarr?

If it's still not working, then as a test you can try YGG in Jackett (again with IPv6 disabled), see if it works there.

If not, you should follow FlareSolverr/FlareSolverr#664, although as @bakerboy448 said, this is likely an issue of how they've configured their Cloudflare IUAM protection as a result of being DDoS'd over the past month of so.

I disabled IPV6 support in my firewall and it seems to be working fine now (It's either that or ygg changed their CF config but I doubt about that) ! Thanks for the tip !

Prowlarr / Indexers