issues
search
temoto
/
robotstxt
The robots.txt exclusion protocol implementation for Go language
MIT License
269
stars
55
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
TestAgent and Test (for the same user-agent) gives different results in case of temporary error when fetching the robots.txt file
#40
masonlouchart
opened
10 months ago
0
Add CodeQL workflow for GitHub code scanning
#39
lgtm-com[bot]
closed
1 year ago
1
List allow & disallow
#38
TheUltimateCookie
opened
1 year ago
2
Mov to latest go
#37
DoryGuy
closed
2 years ago
0
Make the `AllowAll` and `DisallowAll` instances public
#36
thewizardplusplus
closed
3 months ago
5
ci: missed coverage report flags
#35
temoto
closed
2 years ago
0
Initialize field 'Agent' of struct Group
#34
JVMerkle
closed
2 years ago
8
Parser not getting sitemap from robots.txt
#33
TheUltimateCookie
closed
2 years ago
2
Yet more files now parse
#32
DoryGuy
closed
2 years ago
2
Yet more files now parse
#31
DoryGuy
closed
2 years ago
1
Fix defaults to for valid files
#30
DoryGuy
closed
2 years ago
14
Use strings.TrimRightFunc instead of strings.TrimRight to omit internal rune comparison function allocation
#29
moredure
closed
3 years ago
13
sitemaps?
#28
hackermondev
closed
3 years ago
1
optimize Group.findRule
#27
moredure
closed
3 years ago
10
Expose rules Api
#26
jakesylvestre
closed
4 years ago
4
Regexp should match full string
#25
msama
closed
4 years ago
3
Parse rules for a given user agent
#24
yields
opened
5 years ago
1
compatible for https://github.com/google/robotstxt
#23
exfly
closed
1 year ago
2
Add support for Crawl-Delay
#22
moredure
closed
5 years ago
5
Incorrect usage of $ symbols
#21
lmas
closed
5 years ago
6
panic: runtime error: invalid memory address or nil pointer dereference
#20
crosscue
closed
5 years ago
9
Duplicated user agent groups merge
#19
sergebezborodov
closed
7 years ago
7
uhmaybe
#18
temoto
closed
7 years ago
0
Speeding up Parse function
#17
nielsole
closed
7 years ago
2
Change import statement or change package name
#16
sudotliu
closed
8 years ago
3
Different behavior on Google Webmaster Tools robots.txt checker and robotstxt-go
#15
uforic
closed
8 years ago
3
'Host' directive parsing added. It's often used in Yandex to specify …
#14
brainm
closed
9 years ago
1
Removed duplication of code when finding a rule
#13
ujjwalt
closed
9 years ago
1
Crash with unicode symbols
#12
vad
closed
10 years ago
2
removed annoying print line
#11
ghost
closed
10 years ago
1
No error is returned by FromResponse() when the file is incorrect
#10
alexander-bauer
closed
11 years ago
4
Separate DNS resolve from connect
#9
temoto
closed
11 years ago
1
Tiny optimization in scanner.go
#8
mynameisfiber
closed
11 years ago
4
Fixes #6
#7
mna
closed
11 years ago
1
Groups are not handled correctly
#6
mna
closed
11 years ago
13
Reduce memory footprint of multiple logically same RobotsData acquired from different inputs
#5
temoto
opened
12 years ago
2
empty disallow is allow
#4
igrigorik
closed
13 years ago
1
fix allow rule parsing
#3
igrigorik
closed
13 years ago
1
parse non-empty robots.txt
#2
temoto
closed
14 years ago
1
fix '//' formatting in README
#1
temoto
closed
14 years ago
1