Closed ilike2burnthing closed 3 years ago
The cookies provided by FlareSolverr are not valid
I'm getting this error in Epizod
but cpasbien
is working. Try to open the sites in the browser. In my case Epizod is not working in Chrome.
FlareSolverr was able to process the request, but a captcha was detected. Message: Captcha detected but no automatic solver is configured.
I see this too in some sites, it happens when CloudFlare shows the images to pick, hChaptcha. FlareSolverr has 2 captcha providers but I just tested 1 of them and it's not working. The good thing is that the challenge of images only appears in some requests. If you get the cookie in one of them, the cookie is saved for some time.
Any way to extend the proxy settings for Jackett to FlareSolver, or add similar environment variables to FlareSolver?
I don't use proxys so I never tested it. It should be possible to configure FlareSolverr's Chrome with the same proxy.
Looks like the issue with cpasbien is a result of their Cloudflare implementation. I get the same page in browser whether I use a VPN or not - https://github.com/Jackett/Jackett/issues/10472#issuecomment-743468413 (I'm assuming it's region locked as I can access if I change my usual server to a French one). Nothing to do with FlareSolverr then.
https://wwv.epizod.tv/ and https://wwv.epizod.tv/?s= both work in browser for me.
For epizod, could you run FlareSolverr with these environment vars: LOG_LEVEL=debug
and LOG_HTML=true
and share the traces? Be careful about sensitive information.
Proxy support => #26
Hello,
I have the same error on yggtorrent :
Jackett.Common.IndexerException: Exception (yggtorrent): FlareSolverr was able to process the request, but a captcha was detected. Message: Captcha detected but no automatic solver is configured.
---> FlareSolverrSharp.Exceptions.FlareSolverrException: FlareSolverr was able to process the request, but a captcha was detected. Message: Captcha detected but no automatic solver is configured.
at FlareSolverrSharp.Solvers.FlareSolverr.<>cDisplayClass5_0.<1 worker) at FlareSolverrSharp.Solvers.FlareSolverr.Solve(HttpRequestMessage request) at FlareSolverrSharp.ClearanceHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken) at Jackett.Common.Utils.Clients.HttpWebClient2.Run(WebRequest webRequest) in /home/vsts/work/1/s/src/Jackett.Common/Utils/Clients/HttpWebClient2.cs:line 166 at Jackett.Common.Utils.Clients.WebClient.GetResultAsync(WebRequest request) in /home/vsts/work/1/s/src/Jackett.Common/Utils/Clients/WebClient.cs:line 184 at Jackett.Common.Indexers.BaseWebIndexer.RequestWithCookiesAsync(String url, String cookieOverride, RequestType method, String referer, IEnumerable
1 data, Dictionary2 headers, String rawbody, Nullable
1 emulateBrowser) in /home/vsts/work/1/s/src/Jackett.Common/Indexers/BaseIndexer.cs:line 481
at Jackett.Common.Indexers.CardigannIndexer.PerformQuery(TorznabQuery query) in /home/vsts/work/1/s/src/Jackett.Common/Indexers/CardigannIndexer.cs:line 1278
at Jackett.Common.Indexers.BaseIndexer.ResultsForQuery(TorznabQuery query, Boolean isMetaIndexer) in /home/vsts/work/1/s/src/Jackett.Common/Indexers/BaseIndexer.cs:line 369
--- End of inner exception stack trace ---
at Jackett.Common.Indexers.BaseIndexer.ResultsForQuery(TorznabQuery query, Boolean isMetaIndexer) in /home/vsts/work/1/s/src/Jackett.Common/Indexers/BaseIndexer.cs:line 377
at Jackett.Common.Indexers.BaseWebIndexer.ResultsForQuery(TorznabQuery query, Boolean isMetaIndexer) in /home/vsts/work/1/s/src/Jackett.Common/Indexers/BaseIndexer.cs:line 645
at Jackett.Common.Services.IndexerManagerService.TestIndexer(String name) in /home/vsts/work/1/s/src/Jackett.Common/Services/IndexerManagerService.cs:line 300
at Jackett.Server.Controllers.IndexerApiController.Test() in /home/vsts/work/1/s/src/Jackett.Server/Controllers/IndexerApiController.cs:line 132
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.
LOG_LEVEL=debug and LOG_HTML=true
flaresolverr@1.2.0 start
> node ./dist/index.js
2020-12-14T00:57:54.314Z INFO REQ-0 FlareSolverr v1.2.0 listening on http://0.0.0.0:8191
2020-12-14T00:58:35.642Z INFO REQ-1 Incoming request: POST /v1
2020-12-14T00:58:35.647Z INFO REQ-1 Params: {"maxTimeout":60000,"cmd":"request.get","url":"https://wwv.epizod.tv/?s=","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"}
2020-12-14T00:58:35.652Z DEBUG REQ-1 Launching headless browser...
2020-12-14T00:58:42.626Z DEBUG REQ-1 Using custom UA: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
2020-12-14T00:58:42.637Z DEBUG REQ-1 Adding custom headers: {}
2020-12-14T00:58:42.638Z DEBUG REQ-1 { headers: [Function (anonymous)] }
2020-12-14T00:58:42.653Z DEBUG REQ-1 Navigating to... https://wwv.epizod.tv/?s=
2020-12-14T00:58:42.943Z DEBUG REQ-1 {
headers: {
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36'
}
}
2020-12-14T00:58:43.802Z INFO REQ-1 Cloudflare detected
2020-12-14T00:58:43.819Z DEBUG REQ-1 No '#trk_jschal_js' challenge element detected.
2020-12-14T00:58:43.822Z DEBUG REQ-1 No '.ray_id' challenge element detected.
2020-12-14T00:58:43.826Z DEBUG REQ-1 No '.attack-box' challenge element detected.
2020-12-14T00:58:43.852Z INFO REQ-1 Successful response in 8.21 s
Nothing redacted, that's all there is.
Cloudflare hcaptcha is a little bit different than a normal hcaptcha.
We could install and enable by default hcaptcha-solver.
It work by clicking on random images hopping it will work.
More info regarding Cloudflare's hcaptcha implementation:
@ilike2burnthing cpasbien is always presenting a Cloudflare hcaptcha for me from server in France, Spain, Canada and USA
For epizod.tv,from your log, it's detecting cloudflare but nothing after, weird.
Even weirder, I had Windows Sandbox open so decided to check Epizod there as well - worked, no VPN, no FlareSolverr. Turns out the site is a stitched together mess and the indexer needs a lot of work to make any sense out of it (my stab at it - https://github.com/Jackett/Jackett/pull/10505).
I wouldn't be surprised if this isn't some clever DDoS protection, but rather it's just broken.
Ah interesting, epizod.tv is blocking any request if the user agent contain: X11; Linux x86_64
.
We will have to add a check in the code so that it return an error with a message instead of OK:
// Exception for epizod.tv
if (response.url().match(/epizod\.tv/gi) && (await page.content()).match(/error(\s)?code[^0-9]+1020/gi)) {
await page.close()
return ctx.errorResponse('Cloudflare has blocked this request (Code 1020 Detected via regex).')
}
{
"status": "ok",
"message": "",
"startTimestamp": 0,
"endTimestamp": 0,
"version": "1.2.0",
"solution": {
"url": "https://wwv.epizod.tv/?s=test",
"status": 403,
"headers": {
"status": "403",
"date": "Mon, 14 Dec 2020 04:49:26 GMT",
"content-type": "text/plain; charset=UTF-8",
"content-length": "16",
"set-cookie": "__cfduid=[random number]; expires=Wed, 13-Jan-21 04:49:26 GMT; path=/; domain=.epizod.tv; HttpOnly; SameSite=Lax; Secure",
"x-frame-options": "SAMEORIGIN",
"cache-control": "private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0",
"expires": "Thu, 01 Jan 1970 00:00:01 GMT",
"cf-request-id": "[random number]",
"expect-ct": "max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"",
"report-to": "{\"endpoints\":[{\"url\":\"https:\/\/a.nel.cloudflare.com\/report?s=[random number]\"}],\"group\":\"cf-nel\",\"max_age\":604800}",
"nel": "{\"report_to\":\"cf-nel\",\"max_age\":604800}",
"vary": "Accept-Encoding",
"server": "cloudflare",
"cf-ray": "mmm...ray"
},
"response": "<html><head></head><body><pre style=\"word-wrap: break-word; white-space: pre-wrap;\">error code: 1020</pre></body></html>",
"cookies": [
{
"name": "__cfduid",
"value": "[random number]",
"domain": ".epizod.tv",
"path": "/",
"expires": 0,
"size": 51,
"httpOnly": true,
"secure": true,
"session": false,
"sameSite": "Lax"
}
],
"userAgent": "X11; Linux x86_64"
}
}
LOG_LEVEL=debug and LOG_HTML=true
I was expecting to see more HTML source code in the traces => #30
We could install and enable by default hcaptcha-solver.
I did a little test but it wasn't able to solve the hCaptchas. I don't know if it was bad luck or the solver is not working. #31
Ah interesting, epizod.tv is blocking any request if the user agent contain: X11; Linux x86_64.
We can change the User-Agent in Jackett just for this site adding the HTTP header in the request. I don't know if it's possible with the current code but that's the way.
// Exception for epizod.tv if (response.url().match(/epizod.tv/gi)
@abeloin I don't like the idea of having exceptions for each site in the code. Is it possible to make it generic to work in other sites too?
We can change the User-Agent in Jackett just for this site adding the HTTP header in the request. I don't know if it's possible with the current code but that's the way.
As far as I can tell while debugging YGGCookies the user-agent string is cosmetic(in Cardigann) and not use at all, it use the one in BrowserUtil.cs. Some indexer in c#, for example abnormal, use a custom useragent that work
I did a little test but it wasn't able to solve the hCaptchas. I don't know if it was bad luck or the solver is not working. #31
It isn't currently working, it is outputting to word nonsense. There is a patch to fix it but not yet merged. https://github.com/JimmyLaurent/hcaptcha-solver, pull request 12.
I'll check if it is working.
With the patch, I get:
statusCode: 403,
error: {
success: false,
'error-codes': [ 'invalid-answers', 'invalid-motionData' ]
},
Which would suggest that is not enough, meaning it is not enough anymore.
I don't like the idea of having exceptions for each site in the code. Is it possible to make it generic to work in other sites too?
I didn't like it either but I wanted to be safe in case it detected the word mile a part in the page. Like error code: this is a long line 1020
Today, I realized that I can just limit it by using {}: error(\s)?code[^0-9]{0,5}?1020
As far as I can tell while debugging YGGCookies the user-agent string is cosmetic(in Cardigann) and not use at all, it use the one in BrowserUtil.cs. Some indexer in c#, for example abnormal, use a custom useragent that work
I will take a look and fix Jackett if required.
I didn't like it either but I wanted to be safe in case it detected the word mile a part in the page. Like error code: this is a long line 1020
It's safer to check the http response code than the response text. I'm getting 403 forbidden with Linux useragent in epizod.tv. If we can't find cloduflare selectors and response code = 403 i think it's safe to say the user is blocked or banned.
@ilike2burnthing I think this can be closed.
Jackett - Docker v0.17.50-ls9 amd64 FlareSolver - Docker v1.2.0 amd64
When not using a proxy/VPN in Jackett's settings, this affects:
Indexers which are working when not using a proxy/VPN:
Indexers which get
Exception (INDEXER): FlareSolverr was able to process the request, but a captcha was detected. Message: Captcha detected but no automatic solver is configured.
(so would probably work if I set one up) when not using proxy/VPN:However, the only indexers which work when using a proxy/VPN in Jackett's settings are:
The rest show that the cookies aren't valid, or that a captcha was detected. Any way to extend the proxy settings for Jackett to FlareSolver, or add similar environment variables to FlareSolver?