Explanation of the robots.txt exclusion standard in DownloaderMiddleware.robotstxt.py

[mery16q]

In the project's code, in the part of downloaderMiddleware, there is a class called RobotsTxtMiddleware, which a first sight I did not know what it did. Then, after reading the documentation , I understood it, but it did not clarify me about the robots.txt exclusion standard. So, I suggest to put the explanation of that, like a comment in the code of the class, I mean, explaning what the class does and what is the robots.txt exclusion standard stand for.

[wRAR]

The docs already link to https://www.robotstxt.org/ , is that not enough?