Added comments to the file robotstxt.py

scrapy / scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

https://scrapy.org

BSD 3-Clause "New" or "Revised" License

50.99k stars 10.34k forks source link

Added comments to the file robotstxt.py #6287

Closed josegicar closed 1 month ago

josegicar commented 1 month ago

Added comments regarding the robots exclusion standard, facilitating its understanding and explaining how does it work to people who are not very familiar with robots.txt.