search

prestaconcept / PrestaSitemapBundle

A symfony bundle that provides tools to build a rich application sitemap. The main goals are : simple, no databases, various namespace (eg. google image), respect constraints etc.
MIT License
347 stars 100 forks source link

Add /robots.txt route #330

Closed tacman closed 5 months ago

tacman commented 5 months ago

Description

This bundle gives a route to /sitemap.xml, I'd like to also see a basic response for /robots.txt (which of course includes a link to the sitemap).

yann-eugone commented 5 months ago

Hello, and thank you for the suggestion.

I'm not sure if it's a good idea to implement this. For the same reason it's recommended to rely on a sitemap dump, the robots.txt should be a file at in your public dir. There is very few reason for this file to change, and there is nothing sensitive, so it is safe to add it to your versioned files.

Also, the robots.txt file can contains much more than just a link to the sitemap, and it's not this bundle responsibility to have all that complexity.

tacman commented 5 months ago

Well, yes, but often I forget to have it until a bot tries to access it and I get a 404 error in my logs.

And in fact I do want it to change, if it's a staging or dev version, I want to exclude bots, for example.

So mostly I was looking for a super-simple way to get that functionality, and since you already have somethat that handles sitemap, I thought it'd be worth mentioning. I have some other places I can add it, and now I'll look to see if this bundle is installed to properly generate the sitemap key.