-
I tried to index www.democracynow.org and it reproducibly fails with the message:
Crawling of "https://www.democracynow.org" failed. Reason: scraper cannot load URL: REJECTED EMPTY RESPONSE BODY 'HTT…
-
In 5e you use half of your movement to get up. A keyboard shortcut to use half of the remaning movement without actually moving could be usefull.
Related to this, hold-key and drag combination to m…
-
when crawling the files group, they are sorted alphabetically, moving index.js file at which ever place. I would be helpful to move the index.js (if found) to the top of each group, so its the first t…
acatl updated
10 years ago
-
for example:
```
crawler.crawl({
url: "http://localhost:8080/locations/",
selector: ".main-content"
```
would only follow the links found inside `.main-content`
this way i don't have to keep cr…
duggi updated
9 years ago
-
## What happened?
Crawler never finishes when crawling `http://www.ohioattorneygeneral.gov/`.
## What should have happened?
There should be a maxRedirects property, which limits the amount of red…
-
This is more for research as we can serve different landing for mobile which is more tap friendly
-
While crawling [Web Animations Level 2](https://drafts.csswg.org/web-animations-2/), the following links to other specifications were detected as pointing to non-existing anchors:
* [ ] https://draft…
-
Hello
I was wondering if you could share the books corpus as the crawling takes pretty long. I was using it
https://github.com/soskek/bookcorpus