ReCaptcha is broken on prod

[easoncxz]

Clicking "Sign in with Twitter" leads to a long wait followed by an error response:

Reports of similar issues:

• https://groups.google.com/g/recaptcha/c/THvvdHPoEHE?pli=1

Official docs:

• https://developers.google.com/recaptcha/docs/verify

Server logs: (journalctl -u twitanalysis)

HttpExceptionRequest Request {
  host                 = "www.google.com"
  port                 = 443
  secure               = True
  requestHeaders       = [("Content-Type","application/x-www-form-urlencoded")]
  path                 = "/recaptcha/api/siteverify"
  queryString          = ""
  method               = "POST"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 ConnectionTimeout

Diagnosis

First guess is that it's something to do with IP addresses -- did Google just block my server's IP? Perhaps the extra remoteip param should be passed?

[easoncxz]

It seems very likely that I have somehow gotten my IP blocked by Google. From my server:

➜  ~ ping google.com
PING google.com (216.58.199.78) 56(84) bytes of data.
^C
--- google.com ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4086ms

➜  ~ curl -i https://www.google.com
^C
➜  ~ time curl -i https://www.google.com
^C
curl -i https://www.google.com  0.01s user 0.00s system 0% cpu 12.168 total

[easoncxz]

Resolution: it turns out that I have somehow manually listed an IP for google.com in my /etc/hosts, and the listed IP has become bad: that IP address does not seem to be Google's server's anymore.

• using my laptop, ping/ping6 the IPs hard-wired on my server results in timeouts. This suggests that hard-wired IP is bad.
• using my server, ping/ping6 on the IPs obtained via dig on my laptop results in good responses. This suggests that Google did not block my server's IP.

After deleting the bad lines from my /etc/hosts and flushing DNS cache (sudo systemd-resolve --flush-caches, ReCaptcha is once again working on my server.