Cloudflare Javascript & reCaptcha challenge (I'm Under Attack Mode or IUAM) solving/bypass .NET Standard library.
Reawakening of CloudflareSolver (removed) adding the capabilities (DelegatingHandler) of CloudFlareUtilities (not working).
This can be useful if you wish to scrape or crawl a website protected with Cloudflare. Cloudflare's IUAM page currently just checks if the client supports JavaScript or requires solving captcha challenge. Fortunately, this library supports both.
For reference, this is the default message Cloudflare uses for these sorts of pages:
Checking your browser before accessing website.com.
This process is automatic. Your browser will redirect to your requested content shortly.
Please allow up to 5 seconds...
The first visit to any site with Cloudflare IUAM enabled will sleep for 4 to 5 seconds until the challenge is solved, though no delay will occur after the first request.
Full-Featured library:
PM> Install-Package CloudflareSolverRe.Captcha
Or get just javascript challenge solver without the captcha features:
PM> Install-Package CloudflareSolverRe
In case you need to use captcha solvers:
If you want to use another captcha provider, see How to implement a captcha provider?
Cloudflare regularly modifies their IUAM protection challenge and improves their bot detection capabilities.
If you notice that the anti-bot page has changed, or if library suddenly stops working, please create a GitHub issue so that I can update the code accordingly.
Before submitting an issue, just be sure that you have the latest version of the library.
A DelegatingHandler that handles the challenge solution automatically.
A type for HTTP handlers that delegate the processing of HTTP response messages to another handler, called the inner handler.
It checks on every request if the clearance is required or not, if required, it solves the challenge in background then returns the response.
Websites not using Cloudflare will be treated normally. You don't need to configure or call anything further, and you can effectively treat all websites as if they're not protected with anything.
var target = new Uri("https://uam.hitmehard.fun/HIT");
var handler = new ClearanceHandler
{
MaxTries = 3,
ClearanceDelay = 3000
};
var client = new HttpClient(handler);
var content = client.GetStringAsync(target).Result;
Console.WriteLine(content);
The internal challenge solver, that's what happens inside the ClearanceHandler, you can use it directly.
Use it when you already have a HttpClient and you want to solve the challenge manually for some specific website so you can scrape it freely.
var target = new Uri("https://uam.hitmehard.fun/HIT");
var cf = new CloudflareSolver
{
MaxTries = 3,
ClearanceDelay = 3000
};
var handler = new HttpClientHandler();
var client = new HttpClient(handler);
var result = cf.Solve(client, handler, target).Result;
if (!result.Success)
{
Console.WriteLine($"[Failed] Details: {result.FailReason}");
return;
}
// Once the protection has been bypassed we can use that HttpClient to send the requests as usual
var content = client.GetStringAsync(target).Result;
Console.WriteLine($"Server response: {content}");
Full Samples Here
To use a message handler (HttpClientHandler, SocketsHttpHandler, etc.) with ClearanceHandler, provide it as an inner handler.
To provide an inner handler, there are two ways
var handler = new ClearanceHandler
{
InnerHandler = new HttpClientHandler
{
CookieContainer = cookieContainer,
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
Proxy = proxy
},
MaxTries = 5,
ClearanceDelay = 3000
};
var httpHandler = new HttpClientHandler
{
CookieContainer = cookieContainer,
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
Proxy = proxy
};
var handler = new ClearanceHandler(httpHandler)
{
MaxTries = 5,
ClearanceDelay = 3000
};
The maximum number of challenge solving tries. Most of the time the minimum tries required are 2 or 3, so default value is 3. If you would like to override this number to make sure it always succeed, change MaxTries property.
var handler = new ClearanceHandler
{
MaxTries = 10
};
var cf = new CloudflareSolver
{
MaxTries = 10
};
Cloudflare challenges are not always javascript challenges, it may be reCaptcha challenges and this library also provides a way to solve these challenges using captcha solvers (2captcha, anti-captcha, etc.).
So, there's MaxCaptchaTries property to set the max number of captcha challenge solving tries (default is 1).
handler.MaxCaptchaTries = 2;
cf.MaxCaptchaTries = 2;
Normally, when a browser is faced with a Cloudflare IUAM challenge page, Cloudflare requires the browser to wait 4 seconds (default delay) before submitting the challenge answer. If you would like to override this delay, change ClearanceDelay property.
var handler = new ClearanceHandler
{
ClearanceDelay = 5000
};
var cf = new CloudflareSolver
{
ClearanceDelay = 5000
};
User-Agent must be the same as the one used to solve the challenge, otherwise Cloudflare will flag you as a bot.
You can set the user-agent of ClearanceHandler once and it will be used along with the handler. All requests made using that handler will have this user-agent, cannot be changed even if you set the user-agent header explicitly.
var handler = new ClearanceHandler(
userAgent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0");
If you didn't set it, a random user-agent will be generated, also cannot be changed even if you set the user-agent header explicitly.
Also, you can set the user-agent of CloudflareSolver once and it will be used to solve every challenge. But unlike the ClearanceHandler, this user-agent can be changed to a specific value or set to random.
var cf = new CloudflareSolver(
userAgent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0");
To use a random user-agent set randomUserAgent to true when calling Solve method.
var result = await cf.Solve(client, handler, target, randomUserAgent: true);
To use a specific user-agent set userAgent.
var result = await cf.Solve(client, handler, target, userAgent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0");
It's easy to integrate CloudflareSolverRe with other applications and tools. Cloudflare uses two cookies as tokens:
The __cfduid cookie is used to identify individual clients behind a shared IP address and apply security settings on a per-client basis.
Clearance cookies are like authentication cookies, but instead of being tied to an identity, they are tied to the fact that you solved a challenge sometime in the past.
To bypass the challenge page, simply include both of these cookies (with the appropriate user-agent) in all HTTP requests you make.
To retrieve working cookies and a user-agent for a specific website:
var target = new Uri("https://uam.hitmehard.fun/HIT");
var cf = new CloudflareSolver
{
MaxTries = 3,
ClearanceDelay = 3000
};
var result = await cf.Solve(target);
Can be used with a proxy:
IWebProxy proxy = new WebProxy("51.83.15.1:8080");
var result = await cf.Solve(target, proxy);
User-agent can be set to
var result = await cf.Solve(target, randomUserAgent: true);
var result = await cf.Solve(target, userAgent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0");
CancellationToken can be passed to control solving operation cancelling:
var cancellationToken = new CancellationTokenSource();
var result = await cf.Solve(target, cancellationToken: cancellationToken.Token);
Returns SolveResult struct:
public struct SolveResult
{
public bool Success; // indicates that challenge solve process succeeded
public string FailReason // indicates clearance fail reason, if failed
public DetectResult DetectResult; // indicates the protection type found (js, captcha, banned, unknown, no-protection)
public string UserAgent; // the user-agent used to solve the challenge (must be used during the session)
public SessionCookies Cookies; // the two tokens/cookies to use during the session indicating that we passed the challenge.
}
Session cookies can be retrieved as
var cookiesHeaderValue = result.Cookies.AsHeaderString(); // __cfduid=anyvalue;cf_clearance=anyvalue;
var cookieCollection = result.Cookies.AsCookieCollection();
var cookieContainer = result.Cookies.AsCookieContainer();
Remember, you must always use the same user-agent when retrieving or using these cookies.
Here is an example of integrating CloudflareSolver with WebClient. As you can see, all you have to do is pass the cookies and user-agent to the webclient headers.
var target = new Uri("https://uam.hitmehard.fun/HIT");
var cf = new CloudflareSolver
{
MaxTries = 3,
ClearanceDelay = 3000
};
var result = cf.Solve(target).Result;
if (!result.Success)
{
Console.WriteLine($"[Failed] Details: {result.FailReason}");
return;
}
var client = new WebClient();
client.Headers.Add(HttpRequestHeader.Cookie, result.Cookies.AsHeaderString());
client.Headers.Add(HttpRequestHeader.UserAgent, result.UserAgent);
var content = client.DownloadString(target);
Console.WriteLine($"Server response: {content}");
Here is an example of integrating CloudflareSolver with HttpWebRequest. As you can see, all you have to do is pass the cookies and user-agent to the HttpWebRequest headers.
var target = new Uri("https://uam.hitmehard.fun/HIT");
var cf = new CloudflareSolver
{
MaxTries = 3,
ClearanceDelay = 3000
};
var result = cf.Solve(target).Result;
if (!result.Success)
{
Console.WriteLine($"[Failed] Details: {result.FailReason}");
return;
}
var request = (HttpWebRequest)WebRequest.Create(target);
request.Headers.Add(HttpRequestHeader.Cookie, result.Cookies.AsHeaderString());
request.Headers.Add(HttpRequestHeader.UserAgent, result.UserAgent);
var response = (HttpWebResponse)request.GetResponse();
var content = new StreamReader(response.GetResponseStream()).ReadToEnd();
Console.WriteLine($"Server response: {content}");
To use captcha solving capabilities, you can install CloudflareSolverRe.Captcha package which supports the following captcha providers.
If you want to use another captcha solver, see How to implement a captcha provider?
var handler = new ClearanceHandler(new TwoCaptchaProvider("YOUR_API_KEY"))
{
MaxTries = 3,
MaxCaptchaTries = 2
};
var cf = new CloudflareSolver(new TwoCaptchaProvider("YOUR_API_KEY"))
{
MaxTries = 3,
MaxCaptchaTries = 1
};
Implement ICaptchaProvider interface.
public interface ICaptchaProvider
{
string Name { get; }
Task<CaptchaSolveResult> SolveCaptcha(string siteKey, string webUrl);
}
Example AntiCaptchaProvider
public class AntiCaptchaProvider : ICaptchaProvider
{
public string Name { get; } = "AntiCaptcha";
private readonly AntiCaptcha antiCaptcha;
public AntiCaptchaProvider(string apiKey) => antiCaptcha = new AntiCaptcha(apiKey);
public async Task<CaptchaSolveResult> SolveCaptcha(string siteKey, string webUrl)
{
var result = await antiCaptcha.SolveReCaptchaV2(siteKey, webUrl);
return new CaptchaSolveResult
{
Success = result.Success,
Response = result.Response,
};
}
}