The majority of e-commerce and content platforms already have a robots.txt file included with the site. However, capistrno-ash is now setting a pretty much empty robots.txt file. This can cause issues with search engines trying to index parts of the site not worth indexing.
If at all possible in production only, it should be checked to see if there is already a robots.txt file in place and, if not, then write the general file. Either that, or just do not write a robots.txt file at all in production.
The majority of e-commerce and content platforms already have a robots.txt file included with the site. However, capistrno-ash is now setting a pretty much empty robots.txt file. This can cause issues with search engines trying to index parts of the site not worth indexing.
If at all possible in production only, it should be checked to see if there is already a robots.txt file in place and, if not, then write the general file. Either that, or just do not write a robots.txt file at all in production.
Thanks!