Robots.txt is text file that resides in the website root directory and contains the instructions
for various robots (mainly search engine robots) for how to crawl and index your website for
their webpage. Robots.txt contains the search bots or others bot names, the directory list
allowed or disallowed to be indexed, crawling for bots, time delay for bots to crawl and index,
and even the sitemap URL. Full access or a full restriction or customized access or restriction
can be imposed through robots.txt.
Robots.txt is very important for SEO. Your website directories will be crawled and indexed on
search engines according to robots.txt instructions. So, add a robots.txt file in your website
root directory. Write it properly including your content-enriched pages and other public pages
and exclude any pages which contain sensitive information. Remember robots.txt instruct
restricts access to the sensitive information of your page if it is not formidable on webpage
security grounds.
Robots.txt is text file that resides in the website root directory and contains the instructions for various robots (mainly search engine robots) for how to crawl and index your website for their webpage. Robots.txt contains the search bots or others bot names, the directory list allowed or disallowed to be indexed, crawling for bots, time delay for bots to crawl and index, and even the sitemap URL. Full access or a full restriction or customized access or restriction can be imposed through robots.txt.
Robots.txt is very important for SEO. Your website directories will be crawled and indexed on search engines according to robots.txt instructions. So, add a robots.txt file in your website root directory. Write it properly including your content-enriched pages and other public pages and exclude any pages which contain sensitive information. Remember robots.txt instruct restricts access to the sensitive information of your page if it is not formidable on webpage security grounds.
http://www.robotstxt.org/robotstxt.html