search

wuwanahq / wuwana2

🇪🇺 Wuwana v2 - A webapp to help users find local suppliers in Spain.
Mozilla Public License 2.0
2 stars 2 forks source link

Serve "robots.txt" for search engines #154

Closed Nils85 closed 3 years ago

Nils85 commented 3 years ago

A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. Every single day errors appear in PHP logs because search engines requested this file...

levogirar commented 3 years ago

I will take a look and see if I can upload one.

levogirar commented 3 years ago

@Nils85 I created a very simple one from what I read on Google and Shopify. However, I am not sure if it should be on the root of the project or in the folder WebRoot. From what I've read, it has to be on the root of the website.

Resource: https://developers.google.com/search/docs/advanced/robots/create-robots-txt

Nils85 commented 3 years ago

Well... It should be in the WebRoot folder. But actually I was thinking more about the sitemap.xml instead of the robot.txt. Because the file robot.txt is useful only to declare the sitemap. So I will close this issue to use this one #155