PageSpeed Insights always returns ERRORED_DOCUMENT_REQUEST

[drauch]

PageSpeed Insights always returns ERRORED_DOCUMENT_REQUEST for the web site "https://rauch.io". It tells me:

Lighthouse returned error: ERRORED_DOCUMENT_REQUEST. Lighthouse was unable to reliably load the page you requested. Make sure you are testing the correct URL and that the server is properly responding to all requests. (Status code: 403)

Status code 403 is Forbidden, but my web site loads just fine when I open it with Google Chrome. Is there a catch? Do I have to allow PageSpeed Inisghts somehow to measure my web site?

Best regards, D.R.

[devtools-bot]

Thanks chief! Appreciate you filing this bug. :clap:

This is a known issue, most well described in #2784. So, we'll automatically close this as a duplicate.

However, if you believe your bug is different than the cases described there, please comment here with "necessarily-wide-alpaca" and I'll reopen this bug. :robot: Beep beep boop.

[drauch]

necessarily-wide-alpaca

None of the situations described in #2784 match my page. 1) It returns a valid HTML document. 2) There is a HTTP-to-HTTPS redirect in place. 4) Nope, not about:blank. 5) Nope. 6) Nope, seems valid in Chrome.

[patrickhulce]

@drauch 403 forbidden is a status code that is meant to indicate that the user is not authorized to view the page. You can probably see it because you're an authorized user, but the API which is loading it away from your machine is not.

This is what I see and what PSI is likely seeing.

You can audit your page locally as a workaround (DevTools, Extension, CLI, etc), but if you'd like to audit using PSI you'll have to publish a version of your site that doesn't reject unauthorized users.

[drauch]

What? This is very weird. I can open an incognito window and surf https://rauch.io/ just fine. Even the non-HTTPS version loads just fine. No login or anything else required.

Where are you from?

Google can also index my page:

[patrickhulce]

Interesting.

I'm located in Dallas, TX, USA. Using a proxy browser spoofing Europe I was able to load your page, so it appears your server is denying certain IPs at least and PSI's is among them.

[drauch]

My WordPress statistics definitely show traffic from US pages as well. I'm unsure why Google or you can't visit my page ... maybe it has to do with Cloudflare which is in-between?

[patrickhulce]

In case it helps, here's my curl output trying to access your page.

*   Trying 104.27.176.236...
* TCP_NODELAY set
* Connected to www.rauch.io (104.27.176.236) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: OU=Domain Control Validated; OU=PositiveSSL Multi-Domain; CN=sni136550.cloudflaressl.com
*  start date: Apr 21 00:00:00 2019 GMT
*  expire date: Oct 28 23:59:59 2019 GMT
*  subjectAltName: host "www.rauch.io" matched cert's "*.rauch.io"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO ECC Domain Validation Secure Server CA 2
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f8537009400)
> GET / HTTP/2
> Host: www.rauch.io
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 403
< date: Fri, 26 Apr 2019 18:07:56 GMT
< content-type: text/html; charset=iso-8859-1
< set-cookie: __cfduid=dc47dc6a0ceb53ae9efac2c9298719c001556302076; expires=Sat, 25-Apr-20 18:07:56 GMT; path=/; domain=.rauch.io; HttpOnly; Secure
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< server: cloudflare
< cf-ray: 4cda7fcaaeafb3a8-IAH
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /
on this server.</p>
<p>Additionally, a 403 Forbidden
error was encountered while trying to use an ErrorDocument to handle the request.</p>
</body></html>

Best of luck to ya!