issues
search
jculvey
/
roboto
A web crawler/scraper/spider for nodejs
67
stars
24
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Pass error on finish event on exit due to error
#20
bogusfocused
opened
7 years ago
0
Provide external logger
#19
bijoor
opened
8 years ago
0
Question about roboto and memory usage
#18
joshterrill
opened
8 years ago
0
Unable to crawl a page with encoding other than utf8
#17
yogiatpozen
opened
8 years ago
0
syntax error
#16
olavtenbosch
closed
8 years ago
0
get stats for each request
#15
cryptoBOOOOM
opened
9 years ago
0
use the obeyRobotsTxt option
#14
halfnelson
closed
9 years ago
1
hashbang urls are discarded
#13
Morriz
opened
9 years ago
0
infinite crawl
#12
martingg88
opened
9 years ago
4
Cant crawl Facebook.com
#11
webb365
opened
9 years ago
0
Removed node_modules
#10
ArsalanDotMe
closed
9 years ago
1
Crawl any page, parse only whitelisted and not blacklisted pages
#9
ArsalanDotMe
opened
9 years ago
2
Remove folder npm_modules and have a .gitignore
#8
ArsalanDotMe
closed
9 years ago
0
Different Parsers for different domains
#7
ArsalanDotMe
opened
9 years ago
0
Allow Query Parameters
#6
ArsalanDotMe
closed
9 years ago
2
Custom queing logic?
#5
ArsalanDotMe
opened
9 years ago
2
specifying specific urls to crawl
#4
prateeksachan
closed
9 years ago
0
Add path for the item issue and obeyNofollow option
#3
f1ames
closed
9 years ago
1
Stop/resume
#2
f1ames
opened
9 years ago
2
Add path for the item
#1
f1ames
closed
9 years ago
0