Description
The website currently does not have a robots.txt file, which is essential for controlling how search engine bots crawl and index the site. The absence of a robots.txt file can lead to undesirable indexing of pages, potential SEO issues, and increased server load.
Steps to Reproduce
Navigate to the root directory of the website https://tourwise.org/robots.txt
Observe that no robots.txt file is present.
Expected Behavior
A robots.txt file should exist in the root directory of the website, specifying the crawling and indexing rules for search engine bots.
Actual Behavior
There is no robots.txt file present in the root directory of the website.
Suggested Solution
Create a robots.txt file in the root directory of the website with appropriate directives for search engine bots. Below are examples of how to implement different rules using the robots.txt file.
Description The website currently does not have a robots.txt file, which is essential for controlling how search engine bots crawl and index the site. The absence of a robots.txt file can lead to undesirable indexing of pages, potential SEO issues, and increased server load.
Steps to Reproduce Navigate to the root directory of the website https://tourwise.org/robots.txt Observe that no robots.txt file is present.
Expected Behavior A robots.txt file should exist in the root directory of the website, specifying the crawling and indexing rules for search engine bots.
Actual Behavior There is no robots.txt file present in the root directory of the website.
Suggested Solution Create a robots.txt file in the root directory of the website with appropriate directives for search engine bots. Below are examples of how to implement different rules using the robots.txt file.