Closed Myoldmopar closed 6 years ago
Robots:
From: http://www.robotstxt.org/robotstxt.html
To allow all robots complete access User-agent: * Disallow:
From https://techstricks.com/adding-robots-txt-to-your-django-project/, add this in urls:
from django.views.generic import TemplateView urlpatterns = [ #... your project urls url(r'^robots.txt$', TemplateView.as_view(template_name="robots.txt", content_type="text/plain"), name="robots_file") ]
Very similar for humans.txt http://humanstxt.org/Standard.html
Robots:
From: http://www.robotstxt.org/robotstxt.html
From https://techstricks.com/adding-robots-txt-to-your-django-project/, add this in urls: