[add][xs] Added robots.txt from theme /public folder

datopian / frontend-v2

CKAN / Data Portal frontend as microservice in pure Javascript (Node).

http://tech.datopian.com/frontend/

MIT License

38 stars 18 forks source link

[add][xs] Added robots.txt from theme /public folder #185

Closed gavram closed 3 years ago

gavram commented 3 years ago

@anuveyatsu Please can you review this. This is URGENT to be implemented so we can apply this to stop the Bots on Montreal frontend. I added that if we have robots.txt configured in theme /public folder to use it. In oposite we will use default User Agent: * Allow: /