issues
search
fizx
/
robots
robots.txt parser
40
stars
17
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
WSNG-319: ability to accept a robots.txt file io as an argument so check does not have to download hte file from the domain
#11
pyellambalase
closed
5 years ago
0
Changed file permission of lib/robots.rb, addressing issue #6.
#10
Romulus10
closed
7 years ago
6
Only test the first element of status
#9
postmodern
opened
8 years ago
1
Please don't use open-uri
#8
postmodern
opened
8 years ago
0
Parsing error: user-agent label is case sensitive and collapses user-agent specific settings
#7
jasonkim
opened
12 years ago
1
No such file to load -- robots
#6
RichardStokes
opened
13 years ago
4
Rakefile
#5
xambroz
closed
13 years ago
2
add URI to timeout message
#4
rb2k
opened
14 years ago
0
Error in Parser?
#3
rb2k
opened
14 years ago
2
robots non functional at the moment?
#2
rb2k
closed
14 years ago
3
Parsing Error
#1
rb2k
closed
14 years ago
2