robots.txt shoud feature full URL to site.

eudicots / Cactus

Static site generator for designers. Uses Python and Django templates.

BSD 3-Clause "New" or "Revised" License

3.46k stars 314 forks source link

Open martinvol opened 6 years ago

martinvol commented 6 years ago

Currently cactus displays robots.txt as this:

User-agent: *
Disallow:

Sitemap: sitemap.xml

But according to the standar the full URL to site is needed.