search

jazzband / django-robots

A Django app for managing robots.txt files following the robots exclusion protocol
https://django-robots.readthedocs.io
BSD 3-Clause "New" or "Revised" License
457 stars 97 forks source link

Host #25

Closed apshenychnyj closed 8 years ago

apshenychnyj commented 9 years ago

Hello. Please, if possible, add the Host field. SEO specialists is ask.

     User-agent: *
     Disallow: /admin/
     + --> Host: example.com
     Sitemap: http://example.com/sitemap.xml
jezdez commented 9 years ago

Can you provide information about that new field? You could easily override the template to add it, no?

yakky commented 8 years ago

@jezdez https://en.wikipedia.org/wiki/Robots_exclusion_standard#Host Tagging as feature as it looks legit (while still addressable in the template as suggested, as a workaround)

jezdez commented 8 years ago

Ah, cool, hadn't seen that one yet!

dborisov commented 8 years ago

It is already been implemented but declined here https://github.com/jazzband/django-robots/pull/1 I think it is good solution because it helps to avoid human factor. There is some chance to forget to add Host directive, specially on multisite setup like I have. It will be very painfull if 100+ sites will be mirrored in SERP

yakky commented 8 years ago

@dborisov thanks, will merge #1 soonish