issues
search
chrisakroyd
/
robots-txt-parser
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
MIT License
12
stars
8
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Handing empty robots.txt string
#18
tcz
opened
3 weeks ago
0
sitemaps links are not returned for some websites (ex: https://www.sainsburys.co.uk)
#17
kvamsij
opened
1 month ago
0
canCrawl stalls in nodejs but not browser
#16
salivity
opened
3 months ago
1
canCrawl() returns incorrect value for middle paths
#15
ssaumya1711
opened
8 months ago
0
Project maintained?
#14
abhisheksurve45
closed
1 year ago
2
parse localhost with port
#13
chantorak
opened
1 year ago
0
Valid URL results in error
#12
dskvr
opened
1 year ago
0
chore: enforce linting
#11
Trott
opened
1 year ago
0
chore: update mocha to 10.2.0
#10
Trott
opened
1 year ago
0
fix: match paths only at start
#9
Trott
opened
1 year ago
0
canCrawl() returns incorrect result when matching middle of path
#8
Trott
opened
1 year ago
0
canCrawl() returns wrong value for partial path matches without wildcards in robots.txt
#7
Trott
closed
1 year ago
2
Update simple-get 2.4.0 to 4.0.1.
#6
Trott
closed
2 years ago
4
Bug (in `useRobotsFor()`?) causes `canCrawl()` to sometimes return incorrect result
#5
Trott
opened
2 years ago
0
Add support for TypeScript
#4
russellsteadman
opened
2 years ago
0
Fixes/1.1.0
#3
chrisakroyd
closed
3 years ago
0
Save/Restore robots.txt cache.
#2
chrisakroyd
closed
3 years ago
0
Wildcards not supported?
#1
sanderheilbron
closed
3 years ago
3