torchbox / careers

Torchbox careers site
https://torchbox.com/careers
4 stars 1 forks source link

`robots.txt` missing from `public` #97

Closed jhancock532 closed 1 year ago

jhancock532 commented 1 year ago

https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt#file-location-and-range-of-validity

Closing this issue as Google won't check for this file in this location. ChatGPT doesn't think it's a bad idea though:

Yes, if you have a separate website that is being forwarded to a subfolder on the main website, it's a good idea to include a robots.txt file on the separate website.
This is because the robots.txt file is used to give instructions to web crawlers about which pages or sections of your site should not be crawled.
It helps to prevent search engines from indexing pages on your site that you don't want to be publicly available, like pages with sensitive information or duplicate content.
Even though the separate website is being forwarded to a subfolder on the main website, it would be best to have a robots.txt file for the separate website as well, as it would give more fine-grained control over which pages are indexed.