search

Awful / Awful.apk

Native Android client for the Something Awful forums
109 stars 66 forks source link

Issues with Cloudflare Captchas #776

Closed tazjin closed 11 months ago

tazjin commented 1 year ago

In non-Western countries, the Cloudflare DDoS protection that SA is behind prompts for captchas regularly (once every 1-2 hours). In Awful, this currently means that the app can not be used without an active VPN connection to a Western country, as the captcha pages just lead to errors.

I'm not sure how exactly the captchas work, but they probably leave behind some browser state (cookies?) to identify the "blessed" session. It might be possible for Awful to detect the Cloudflare captchas, display the captcha page (probably by necessity using the same user-agent etc. as the app itself), and then persist whatever browser state change happens as a result.

tazjin commented 1 year ago

Some more information, it seems like there's a "managed captcha" thing for mobile users: https://forums.somethingawful.com/showthread.php?threadid=4012198

This doesn't seem to work with the Awful app at the moment, might be something we have to actively add support for somewhere.

tazjin commented 1 year ago

As mentioned in the thread, I'm starting to look into this in more detail. Relevant information so far:

  1. A captcha-request can be detected through the cf-mitigated: challenge header in the response.
  2. After going through a captcha, the user receives a cf_clearance cookie which authenticates (for a short period) that particular user-agent/IP combination.

I think we can detect the broken captcha responses and then show a web view to do the challenge. I don't know a lot about Android stuff, but there's probably a way to set the user-agent of the web view to match whatever Awful is sending.

Edit: Well, that got stopped in its tracks quickly. I have a fiber cut at home today and even just getting started with Android development requires something like a 5GB download which I'm not patient enough to wait out over a mobile network. I'll give it another shot again in the future.

tazjin commented 1 year ago

Persisting with some blind hacking on this anyways. We can detect this problem in AwfulRequest (I've added an example commit above that does this).

Because this can occur on every request to SA, detecting it in that class is probably sensible, but I'm not 100% sure how to proceed. Can an AwfulRequest be retried?

At the point where we receive that captcha we have the HTML required to fill in the captcha web view (it's part of that response), so it would seem like what we want is something like:

  1. When encountering a captcha request during an AwfulRequest, render a web view with the captcha and let the user solve it.
  2. The solved captcha somehow sets a cookie. I'm not sure if this happens in JS on the client (probably??), or if there's an additional request/response involved.
  3. Based on the answer to 2, we somehow have to extract this cookie and set it for all other requests.
  4. Finally retry the original AwfulRequest.

I have a very vague understanding of how this Android stuff works, so not sure if this makes sense. I don't know what intents and actions are so far, so it's a bit confusing.

tazjin commented 1 year ago

The cookie setting is done "transparently", what that means is that after solving the captcha the user is redirected by CF to the original URL and that response contains a set-cookie header with cf_clearance.

To avoid overcomplicating this, we might want the web view that deals with this to just open /index.php and scrape the cookie out of that somehow.

tazjin commented 1 year ago

If I'm understanding this correctly, it's actually quite difficult to launch a web view from inside an AwfulRequest, as that is not an "activity" and the web view is.