Robots.txt - Githubissues

DarrachBarneveld / ci-swag

Project 5 for Code Institute - Diploma in Full Stack Software Development (E-commerce Applications)

0 stars 2 forks source link

Robots.txt #56

Closed DarrachBarneveld closed 7 months ago

DarrachBarneveld commented 7 months ago

As a developer, I can create a robots.txt file so that I can control search engine crawler access to specific parts of my website.

Acceptance Criteria:

The robots.txt file effectively communicates directives to search engine crawlers.
Crawlers are properly instructed on which parts of the website to access and which to exclude.
The robots.txt file is correctly formatted according to the Robots Exclusion Protocol (REP).
The robots.txt file is successfully uploaded to the root directory of the website and is accessible to search engine crawlers.

Tasks:

[x] Define the pages to be crawled or excluded.
[x] Identify user-agent directives for search engine crawlers.
[x] Decide on allow or disallow rules for each user-agent.
[x] Create the robots.txt file using a text editor.
[x] Test the robots.txt file using online validation tools.
[x] Upload the robots.txt file to the root directory of the website.

DarrachBarneveld commented 7 months ago

Robots.txt file declared and disallowed paths allowed. Linked to sitemap