Automatic robots.txt generation

[qjhart]

For the production deployment the robots.txt file should be:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://library.ucdavis.edu/wp-sitemap.xml

for all other deployments, the file should be:

User-agent: *
Disallow: /

The default management of this via the wordpress management console doesn't seem like a complete solution. It removes the site map, but doesn't explicitly Disallow: indexing.

In the current setup. Developers need to proactively disable indexing in their sites, (eg sandbox and stage) by adding a robots.txt file.

dc exec wordpress bash
# on wordpress
echo -e 'User-agent: *\nDisallow: /' > robots.txt

A better setup would be one of:

• a default disallowing robots.txt in the image that is removed on the initialization of wordpress when the SITE_URL is library.ucdavis.edu
• A php program that dynamically returns the proper verison of the robots.txt file
• something else

[spelkey-ucd]

Deployed main-wp-website@v3.2.0