Closed brawer closed 3 years ago
The toolforge nginx proxy injects a deny-everything configuration when the backend server returns a 404 for /robots.txt. To make the tool discoverable in search, allow crawling the site by sending back a static string.
The toolforge nginx proxy injects a deny-everything configuration when the backend server returns a 404 for /robots.txt. To make the tool discoverable in search, allow crawling the site by sending back a static string.