amoilanen js-crawler issues

amoilanen / js-crawler

Web crawler for Node.JS

MIT License

253 stars 55 forks source link

issues

Newest

Newest Most commented Recently updated Oldest Least commented Least recently updated

Link crawling gets stuck in Wordpress sites

#61 sandysh opened 4 years ago
0
Crawler completes then cancels the output of "crawledUrls"?

#60 sbr2567 opened 4 years ago
0
Crawler stopped without reason and any error

#59 rafaelwdornelas opened 4 years ago
1
Crawler stopped without reason and any error

#58 pittersnider opened 5 years ago
0
How to deal with basic auth?

#57 pittersnider opened 5 years ago
1
Crawler is not a function

#56 atomixstar opened 5 years ago
3
freeze and defrost for saving and resuming a big crawl? enhancement

#55 Shane-Neeley opened 6 years ago
5
Update request dependency

#54 ghost closed 6 years ago
2
I think shouldCrawl code example is incorrect

#53 JeffreyConnected closed 6 years ago
2
Is it possible to just crawl images using this package?

#52 FritzAgency opened 6 years ago
1
accept custom headers in "configure"

#51 JediRhymeTrix closed 6 years ago
3
`knownUrls` processing logic is incorrectly using underscore

#50 Nysosis opened 6 years ago
3
Pair js-crawler with PhantomJS

#49 Tynael opened 7 years ago
1
Usage

#48 zaszlo closed 3 years ago
2
How to deal with ETIMEDOUT error and pending forever?

#47 fanzijian opened 7 years ago
5
Basics

#46 peterweter1 closed 7 years ago
3
How to deal with shortened URLs

#45 lukasIO opened 7 years ago
3
Follow redirects

#44 roccomuso opened 7 years ago
1
robots.txt

#43 Muneem opened 7 years ago
1
stop crawling

#42 Muneem opened 7 years ago
5
path in variable

#41 Muneem closed 7 years ago
2
forgetCrawled method

#40 Muneem closed 7 years ago
2
Added shouldSpider to supplement shouldCrawl

#39 jankcat closed 7 years ago
3
Added shouldSpider to supplement shouldCrawl

#38 jankcat closed 7 years ago
1
Add referer directly to page object, support for single quote hrefs

#37 jankcat closed 7 years ago
0
Add referer directly to page object

#36 jankcat closed 7 years ago
0
Page that linked to current page

#35 jankcat closed 7 years ago
4
Evaluate selectors

#34 BrunoQuaresma opened 7 years ago
3
getting unknown encoding error on some pages

#33 stamanuel closed 7 years ago
2
Add <base> tag support for relative urls

#32 sedran closed 7 years ago
2
bug empty response

#31 bymaximus closed 7 years ago
5
shouldCrawl doesnt call onAllFinished

#30 bymaximus closed 7 years ago
3
Is it can use by only JavaScript?

#29 mqliutie closed 7 years ago
6
Publishing latest fixes?

#28 mledwards closed 7 years ago
3
Can we promisfy js-crawler

#27 shekarls opened 8 years ago
2
How to assign encoding of response content?

#26 winglight opened 8 years ago
6
Getting every type of url from the page source

#25 shekarls opened 8 years ago
1
Fix a tiny but disastrous bug to make depth works as it should be

#24 tibetty closed 8 years ago
1
When run asynchronous by Executor, depth lost its scope

#23 tibetty closed 8 years ago
3
Feature to crawl up to a limited number of pages

#22 ArielAleksandrus opened 8 years ago
1
js-crawler seems to crawl the same url multiple times

#21 JulienBonnin-ABTasty closed 7 years ago
3
A small change to omit comments in html body

#20 tibetty closed 8 years ago
1
Minor changes were made to handle binary resources

#19 tibetty closed 8 years ago
1
What the content exactly is when the requested resource are binary, e.g., images or pdf file?

#18 tibetty closed 8 years ago
9
Adds Referer header to the request custom HTTP Headers

#17 plebon closed 8 years ago
2
Ajax crawling

#16 edouardkombo closed 8 years ago
2
Would be awesome to apply a selector to limit scope of crawled links

#15 duggi opened 9 years ago
1
Fixed undefined reference to onAllFinished callback in _requestUrl

#14 mathewbergt closed 9 years ago
1
The "depth" for crawling a website completely

#13 rahul-desai3 closed 9 years ago
4
How can we crawl local websites?

#12 Redsandro closed 9 years ago
2