htmltest - 503, curl - 200 for all pages of https://www.php.net

[mmattel]

Describe the bug

When htmltest checks eg: https://www.php.net/manual/en/book.pcntl.php it returns 503 (true for all links pointing somewhere to https://www.php.net)

If I do a manual check with curl, I get HTTP/2 200

Btw, result also differs when having the google play store, example: https://play.google.com/store/apps/details?id=com.azure.authenticator&hl=en&gl=US returning a 404 but curl a 200

Versions

• OS: Ubuntu 20.04
• htmltest: 0.15.0

[arranf]

I've seen similar behaviour for lots of other sites (normally a 403).

These requests work fine using curl or other tools even with the same User-Agent.

A good example of this is reddit.com

[wjdp]

Merging this issue into #165 as seems to be same fault.