Closed kingRodian closed 7 years ago
There was an error in the parsing of robots.txt which prohibited the bot from being polite. It is now fixed (I believe)
There was an error in the parsing of robots.txt which prohibited the bot from being polite. It is now fixed (I believe)