internetstandards / Internet.nl

Internet standards compliance test suite
https://internet.nl
172 stars 35 forks source link

Changed to common crawler User-Agent #1257

Closed bwbroersma closed 6 months ago

bwbroersma commented 7 months ago

Fixes https://github.com/internetstandards/Internet.nl/issues/1224

Note that during development the INTERNETNL_DOMAINNAME is internet.test, and the VERSION is 0.0.0-dev0 (or at least in my fork), so the result is:

Mozilla/5.0 (compatible; internetnl/0.0.0-dev0; +https://internet.test/)

With INTERNETNL_BRANDING it defaults to https://internet.nl/about/. Note it can be set with USER_AGENT_URL.

Some things I'm not sure about:

mxsasha commented 7 months ago
* instead of the if branding construct in the default value of `USER_AGENT_URL`, it could be simplified by using the [internet.nl.dist.env](https://github.com/internetstandards/Internet.nl/blob/main/internetnl/internet.nl.dist.env)?

The improvement with the current code is that all requests from our dev and batch environments will point to internet.nl, which is a bit cleaner.