-
Is scrapy-splash not compatible with obeying robots.txt? Everytime I make a query it attempts to download the robots.txt from the docker instance of scrapy-splash. The below is my settings file. I'm t…
-
```
This isn't really a problem, I'm just not sure where else to post. I got bored
and added a picture drawing extension onto the bot & thought other people might
like it.
It requires PIL, accepts…
-
Avoiding other robots requires resources to track them. Most of them aren't that interesting. We're not hitting them that hard anyway.
-
Domain: https://nv.gov
Hi, looks like your scan did not pick up our Sitemap or Robots. If your crawler is following the robots.txt, I just need to know what to allow so you can crawl the rest of th…
-
This is part of the GoLive Checklist activities
-
It is not served and apparently we should.
-
Does it lie in the scope of this project to support robots.txt?
-
Source URL: https://dmccreary.gitbooks.io/coderdojo-robots/content/
Source Dojo: Twin Cities
Author: Dan McCreary
Author URL (e.g. web page, Twitter, etc.) https://www.gitbook.com/@dmccreary
Short Des…
-
I disagree with the classification of R2D2 and C3PO. OK, yeah, threepio *sounds* middle class, but droids in the Star Wars universe appear to have no rights, no recognition, no respect. Droids are bas…
-
i'm not sure what's going on, but i think that when a robot is mining and moving in lazy chunks, the robot picks itself up. I came back and saw the particle effects like a robot unable to move, but t…