reCAPTCHA blocked in China

JacksonWuxs commented 6 years ago

A pypi ID is necessary while Iwould like to share and upload a package on pypi. Unfortunnatly, there was something wrong with the website of "https://pypi.org/account/register/". After writing the all information, an ERROR was arise named "Recaptcha error.". I'm not sure if that is caused by China Internet?

ewdurbin commented 6 years ago

It looks like our Register page may be blocking the "additional screening" required by the recaptcha on occasion. Generally a modal that pops over and requests that a user click on objects that look like a "street sign" or "storefront"

@di @nlhkabu any ideas?

dstufft commented 6 years ago

Might be something related to the CSP policy.

di commented 6 years ago

@JacksonWuxs Are you seeing a reCAPTCHA like this on the register page?

UlionTse commented 6 years ago

@di @ewdurbin I also don't see CAPTCHA in China. See below,

dstufft commented 6 years ago

Oh interesting, I wonder if China is blocking recaptcha.

dstufft commented 6 years ago

Okay, so according to @reaperhulk Recaptcha doesn't work in China, that is... unfortunate.

dstufft commented 6 years ago

Some more information https://github.com/google/recaptcha/issues/87

UlionTse commented 6 years ago

@dstufft I see, thanks.

dstufft commented 6 years ago

Not sure what (if any) milestone we should add this to, but getting this fixed is super important since currently all of China can't interact with anything on Warehouse that requires a captcha. /cc @brainwane

JacksonWuxs commented 6 years ago

I'm sorry about writing back so late. In fact I didn't see this register page. Here is what I got after clicking the "Create Account" button.

JacksonWuxs commented 6 years ago

I will try to find the answer from the blog what you gave me. In any case, I am moved by your attention to this issue.

By the way, Is the user who upload the library to Pypi really important? If not, could you help me upload my lib to Pypi please? I can't wait to share my library. hahaha... Here is my lib address: https://github.com/JacksonWuxs/datapy .The address includes CODES, README and EXAMPLES.

If you need more information about this issue, please let me know. I will try my best to collect information to help you.

di commented 6 years ago

@JacksonWuxs If you let me know the email address and username you'd like to use, I can register an account on your behalf, and then we can do a password reset so you can get access to it.

JacksonWuxs commented 6 years ago

I'm really happy to hear that. I hope you could help me register an account and my information as follow: E-Mail:wuxsmail@163.com User Name: Jackson Woo Real Name: Xuansheng Wu

di commented 6 years ago

@JacksonWuxs Done, you should have a password reset email in your inbox.

Leaving this issue open until we resolve the larger problem of using reCAPTCHA in China.

JacksonWuxs commented 6 years ago

Done, I have finished to reset my password. Thanks, so much!

JacksonWuxs commented 6 years ago

I'm sorry I need to bother you again. Unfortunately, when I was uploading my program with command as "twine upload dist/*", I raised another error. The error said: "HTTPError: 403 Client Error: The user 'JacksonWoo' is not allowed to upload to project 'datapy'." How can I do?

di commented 6 years ago

@JacksonWuxs No apology necessary. The datapy project already exists, you'll need to choose a new name.

JacksonWuxs commented 6 years ago

Oh it so sad... Could you tell me which web site will show the name list? I mean the token names.

di commented 6 years ago

@JacksonWuxs You can search here: https://pypi.org/search/, for example https://pypi.org/search/?q=datapy.

JacksonWuxs commented 6 years ago

I upload my package to Pypi successful. But I faced another bigger problem I thought. After I install successful, it still couldn't be import? I have screenshot of the wrong process. Thank you! On 3/12/2018 22:02，Dustin Ingramnotifications@github.com wrote：

@JacksonWuxs No apology necessary. The datapy project already exists, you'll need to choose a new name.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

brainwane commented 6 years ago

I'd love advice on how we should address this. Right now I believe this is the biggest blocker stopping us from going to the beta stage and doing our publicity push.

dstufft commented 6 years ago

We're probably going to have to use a different captcha solution, at least in china if not everywhere. At least that's the only thing I can think of.

ncoghlan commented 6 years ago

Some basic notes at https://stackoverflow.com/questions/23780387/recaptcha-availability-in-china that sum up to "Don't use recaptcha if you have users in China".

tylerdave commented 6 years ago

It's been a while but I've used this with success when I needed a captcha solution that worked in China in the past: https://captcha.com/ (note that there's no Python backend though.)

catrope commented 6 years ago

@brainwane asked me for input on this, so I read the entire upstream issue (google/recaptcha#87). In summary:

reCAPTCHA was reported blocked in China in January 2016. I believe this is because parts of it are hosted under https://www.google.com/recaptcha/, and since SSL encrypts the URL being requested, this is indistinguishable from any other request to www.google.com from the Great Firewall's perspective
There's a lot of confusing discussion and back-and-forth where people try to figure out a solution that proxies reCAPTCHA on their domain. This is harder than it sounds because reCAPTCHA serves JS that makes further requests to reCAPTCHA, so a proxy has to rewrite URLs embedded in the responses to point to itself instead of to Google.
Proxy solutions that people claim work for them are this one for nginx, and either this one, or this one with this addendum, for Apache. However, the latter is reported to have some issues, and I wouldn't be surprised if other proxy solutions had similar problems. Rewriting a third party's JS on the fly to get it to send all its requests to your domain is tricky and fragile.
Someone claimed that https://recaptcha.net would work, because that domain is not blocked according to greatfire.org, and its SSL certificate is Google's misc services cert (so it looks legit). However, someone else reported that recaptcha.net still makes a request to www.google.com, which would fail in China.

My recommendation would be to first try to get Google to fix their own proxy (recaptcha.net) first, since it appears to almost work (if it is indeed broken; that's probably worth independently verifying). If that fails, you could try setting up your own proxy, but that is likely to be buggy and require frequent maintenance, since routine changes made by Google to the reCAPTCHA code are likely to break the content rewrite rules from time to time. When such a proxy does break, it could break captcha functionality for all users, not just those in China. As an alternative, it might be less trouble to use a different captcha service, as @tylerdave suggests.

stevepiercy commented 6 years ago

Assuming the purpose of a CAPTCHA is to prevent malicious users from registering accounts, there are other proven techniques that do not require it in any form. I've successfully implemented many of the techniques described in this article: https://webaim.org/blog/spam_free_accessible_forms/

These techniques are transparent to the legitimate user (no extra click, selection of specific objects in squares, or transcription of wiggly alphanumerics), are accessible (this is extremely important), and they effectively eliminate most automated attacks.

Please consider this strategy instead of *CAPTCHA. Thank you!

sfdye commented 6 years ago

It's very unfortunate to see this. IMO we shouldn't rely on any work-around on reCAPTCHA itself, you don't know China's policy when it comes to google. Trust me.

di commented 6 years ago

Looking at https://webaim.org/blog/spam_free_accessible_forms/ I'm not sure those techniques are going to help us with registration spam:

Detect spam-like content within submitted form elements

There is not really enough content on the submitted fields to do this

Detect content within a hidden form element

I don't think this is applicable here.

Validate the submitted form values

Already do this, and blacklist disposable email domains.

Search for the same content in multiple form elements

I don't think this is applicable here.

Generate dynamic content to ensure the form is submitted within a specific time window or by the same user

Already do this with CSRF tokens.

Create a multi-stage form or form verification page

I don't think we want to do this for our registration form.

Ensure the form is posted from your server

Checking the referrer is something we can do, but this is easy to forge.

di commented 6 years ago

I think I'm going to try switching to https://captcha.com/, it's both available in China and touts it's accessibility so it shouldn't be much different than reCAPTCHA for our users.

di commented 6 years ago

Hmm, it seems like https://captcha.com/ does not offer a hosted solution (probably why it works so well in China) so the lack of a Python backend will be a problem, and requires a license.

stevepiercy commented 6 years ago

I hope y'all decide to abandon CAPTCHAs. If so, then I'd ask that you reconsider these three tactics. They have worked very well for me, second only to rigorous submitted value validation and CSRF protection, to stop automated evildoers.

Detect spam-like content within submitted form elements
Detect content within a hidden form element # This "honeypot" catches many spambots for me
Search for the same content in multiple form elements

Of course, stopping attacks further upstream would be even better, but that's above my pay grade and for systems and network administrators. 😉

ncoghlan commented 6 years ago

For ease of reference: the approach @di implemented in #3339 is @stevepiercy's suggestion of "add a hidden form element that's required to be empty".

For ordinary users, that's hidden and will always be empty, but a spam bot scraping the HTML without looking at the CSS will see a text field that it can attempt to add links to.

iseki0 commented 5 years ago

I think you can use https://recaptcha.net for a instead. This url looks not been blocked by China government.

pradyunsg commented 5 years ago

Someone claimed that recaptcha.net would work, because that domain is not blocked according to greatfire.org, and its SSL certificate is Google's misc services cert (so it looks legit). However, someone else reported that recaptcha.net still makes a request to www.google.com, which would fail in China.

That's already been mentioned in this thread @cpdyj.

batou-mtcapthca commented 5 years ago

reCaptcha v2 is not guaranteed to work in china due to its dependencies on google.com which is blocked. One can look at https://captcha.com/ to host their own, or https://www.mtcaptcha.com for a paid for recaptcha alternative. Both would work in China.

Dorro101 commented 5 years ago

Hi

I am currently residing in Nanjing, China. I was temporarily unable to create an account because of the reCAPTCHA issue and the blanket blocking of google.com and its APIs. I was, however, able to get around this using the audio function by clicking on the speaker/volume (sic) icon and entering a code provided by audio playback.

JacksonWuxs commented 5 years ago

Well there was long time after I rose the issue. Finally, I solved this problem with a proxy software which let me go through the fire wall of China Internet.

Due to the reason that China government locked most of responses from abroad, including APIs from Google, the reCAPTCHA issue happened. If anyone face the same issue in the future, the best solution is buying a proxy. At that moment, you can not only create an account, you also search papers or more information with Google conveniencely.

ShikiSuen commented 4 years ago

use HCaptcha instead, period.

di commented 1 year ago

Unfortunately, we needed to re-enable reCAPTCHA in #13232 due to the changes in #3339 eventually becoming insufficient to prevent spam signups. For any user that is affected by this, your best option at this time would be to use a proxy. Our sincere apologies for this inconveneince!

ewdurbin commented 1 year ago

New guidance: https://developers.google.com/recaptcha/docs/faq#can-i-use-recaptcha-globally

With an additional step not noted in googles docs regarding the need to add gstatic.cn to CSP: https://stackoverflow.com/a/57855838

di commented 1 year ago

Looking at the comments there, it seems like this doesn't work anymore, however it might be worth doing anyways and the "proxying" solution there might be something we could implement instead.

I'm going to unlock this issue to give our users in China the ability to tell us if they are still encountering this.

demonguy commented 1 year ago

@di I got the same issue here, cannot see the recaptcha on regsiter page.

BTW, my problem is slightly different here. I got proxy to access so websites outside of China. I can access Google Facebook and so on. And i can also see recaptcha on those websites. when i open console of the browser, i didn't see any http status_code but the request failed because of csp

And i tried to disable CSP of chrome, and it works. I think that's some sort of security update? my Chrome version: 111.0.5563.111

miketheman commented 1 year ago

@demonguy thanks for the report, can you please provide more details on the specific blocked request? I’m looking for the full url the browser is attempting to load? Clicking on the blocked line in the inspector ought to display more details. Also clicking on the Console tab should show an error detail as well.

Teddy-van-Jerry commented 1 year ago

@demonguy Same situation and issue with you on Chrome. Recapture displays correctly on Safari. (When setting global proxy I can see Recapture on Chrome too, but with some latency.)