issues
search
scrapy
/
protego
A pure-Python robots.txt parser with support for modern conventions.
BSD 3-Clause "New" or "Revised" License
54
stars
28
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Wrong handling of rule with wildcard in path
#51
kox-solid
opened
3 weeks ago
0
Update README.rst
#50
baotlake
closed
3 months ago
1
Release notes for 0.3.1
#49
Gallaecio
closed
5 months ago
1
Add equal-sign to parse safe characters
#48
jeroenseegers
closed
5 months ago
3
Use `pyre2` as optional dependency for RegExp speedup.
#47
rtb-zla-karma
opened
6 months ago
0
Add Python 3.12 to supported versions.
#46
wRAR
closed
10 months ago
1
fixed leading double slash issue
#45
s-ferri-fortop
opened
11 months ago
0
URL starting with double-slash are misinterpreted
#44
s-ferri-fortop
opened
11 months ago
1
Add a changelog and cover version 0.3.0
#43
Gallaecio
closed
1 year ago
1
Add pre commit rules
#42
VMRuiz
closed
1 year ago
2
Convert user-agent to lowercase on _get_matching_rule_set
#41
VMRuiz
closed
1 year ago
1
Add visit time parsing feature
#40
VMRuiz
closed
1 year ago
1
Support for visit-time
#39
rasmusson
closed
1 year ago
1
Drop Python 3.7 support.
#38
wRAR
closed
1 year ago
1
fix(tests): `assertEquals` → `assertEqual`
#37
Laerte
closed
1 year ago
2
python-protego fails to build with Python 3.12: AttributeError: 'TestProtego' object has no attribute 'assertEquals'.
#36
rosset
closed
1 year ago
0
Update for RFC 9309 Robots Exclusion Protocol
#35
Gallaecio
opened
1 year ago
0
Only prefix absolute URLs with /
#34
Gallaecio
closed
1 year ago
1
Add Python 3.11 support
#33
Laerte
closed
1 year ago
2
Remove `six` package dependency
#32
Laerte
closed
1 year ago
1
six usage
#31
pgajdos
closed
1 year ago
1
Simplify PyPy installation, bump actions versions
#30
Laerte
closed
1 year ago
2
Protego differs from reppy in handling of wildcards for GET-params
#29
gertjanol
opened
1 year ago
4
Non-encoding of URL queries causing Regex robots.txt rule matches to be missed
#28
jack-kaiasm
opened
2 years ago
3
Raise exception if input is not string
#27
Laerte
closed
2 years ago
2
fix: Code coverage
#26
Laerte
closed
2 years ago
1
Drop support for Python 2.7, Switch CircleCI → GitHub Actions
#25
Laerte
closed
2 years ago
3
Select applied rule by longest pattern length
#24
sseveran
closed
1 year ago
5
Automatically deploy new Protego releases to conda-forge
#23
anubhavp28
closed
2 years ago
1
Document Python version support in the comparison table of the README
#22
Gallaecio
opened
2 years ago
0
Fix readme content type and Travis deploy step
#21
anubhavp28
closed
2 years ago
2
Specify project url and support Python - 3.8, 3.9 and 3.10
#20
anubhavp28
closed
2 years ago
5
Fix disallow not working when no path is provided
#19
Laerte
closed
2 years ago
1
robots.txt can now be read as bytes
#18
Eeelco
closed
2 years ago
8
Disallowing / does not work when the target URL path is missing
#17
Gallaecio
closed
2 years ago
0
Accept robots.txt as bytes
#16
Gallaecio
closed
2 years ago
0
Handle sitemaps that come before any user agent directives
#15
sseveran
closed
3 years ago
0
Replace ':' with '_' in test_data file names
#14
tjlaboss
closed
3 years ago
0
Colons in file names prevent installation on NTFS
#13
tjlaboss
closed
3 years ago
2
Update URLs for prebuilt pypy
#12
whalebot-helmsman
closed
3 years ago
0
Correct a typo in protego.py
#11
felixonmars
closed
3 years ago
1
Prepare for wheel distributions
#10
akx
closed
4 years ago
5
Add some improvements
#9
noviluni
closed
4 years ago
0
Enforce path only in Alllow/Disallow
#8
whalebot-helmsman
closed
4 years ago
0
Add code coverage support
#7
Gallaecio
closed
4 years ago
1
Add a comparison table to the README
#6
Gallaecio
closed
4 years ago
0
Support absolute URLs
#5
Gallaecio
closed
4 years ago
2
Cannot fetch non-disallowed domain
#4
Gallaecio
closed
4 years ago
1
Test for malformed disallow directives
#3
maramsumanth
closed
4 years ago
0
Demo
#2
anubhavp28
closed
4 years ago
6
Next