Open varna9000 opened 1 month ago
If I use the --parserobots argument the crawler simply crawls the main domain but doesn't follow any links dispite the allow directives.
--parserobots
You can try it on https://google.com to see the behaviour.
https://google.com
If I use the
--parserobotsargument the crawler simply crawls the main domain but doesn't follow any links dispite the allow directives.You can try it on
https://google.comto see the behaviour.