search

leekelleher / umbraco-robotstxt-editor

An Umbraco package to edit your robots.txt file
https://our.umbraco.org/projects/developer-tools/robotstxt-editor/
MIT License
8 stars 5 forks source link

Feature Request - Add support for multi-site different robots.txt files #20

Open NikRimington opened 4 years ago

NikRimington commented 4 years ago

Not sure how this would work yet, but if you have multiple sites in a single umbraco instance, you might need to have different robots.txt files served up and handled. Particularly for the different site maps.

leekelleher commented 4 years ago

Thanks @NikRimington. We had a discussion around this, (at the beginning of the rebuild). Came to the conclusion that if you're building multi-tenancy sites, you probably wouldn't be using this package.

Leaving this open for discussion though... @LottePitcher @emmaburstow?

LottePitcher commented 4 years ago

Multi-tenancy sites I've worked with have used dynamically generated site maps, to avoid the need for physical files. So yes it does look like that would be beyond the scope of this package.

NikRimington commented 4 years ago

I've had a multi-tenant Umbraco site that had multiple robots.txt files on disk with different names and a controller to determine which was picked. It would be cool if this package could be used to manage them. It may well be out of scope, was just an idea :-)

Nicholas-Westby commented 3 years ago

One easy way to address this would be to allow for special tokens, which would allow for the domain to be swapped. For example, a person could enter this line:

sitemap: https://{{domain}}/sitemap.xml

Then if a person views the robots.txt at https://www.site.com/robots.txt, it would render as:

sitemap: https://www.site.com/sitemap.xml

So long as the XML sitemap URL is consistent across domains, this would work fine.

As somebody who built a Robots.txt package for Umbraco 7: https://our.umbraco.com/packages/developer-tools/robotnik/

I can say with some experience that I wouldn't really recommend having a different physical file for each domain. It gets cumbersome real quick (especially in my case where the web.config had to be updated for each new domain), and for no real good reason (usually just the sitemap URL changes, or the entire file changes for dev/stage/prod to allow/disallow crawling).

Having dynamic tokens would make this much easier.