search

scrapy / protego

A pure-Python robots.txt parser with support for modern conventions.
BSD 3-Clause "New" or "Revised" License
54 stars 28 forks source link

Add visit time parsing feature #40

Closed VMRuiz closed 1 year ago

VMRuiz commented 1 year ago

Fixes https://github.com/scrapy/protego/issues/39

Added a new Namedtuple VisitTime with fields start_time, start_time that contains the allowed visit time if any.

I didn't want automatically validation of Visit-Time to can_fetch as it wouldn't be backward compatibility, so for now, it has to be manually by the user.

codecov[bot] commented 1 year ago

Codecov Report

Merging #40 (e69947b) into master (907d62f) will decrease coverage by 0.49%. The diff coverage is 92.85%.

@@            Coverage Diff             @@
##           master      #40      +/-   ##
==========================================
- Coverage   98.35%   97.87%   -0.49%     
==========================================
  Files           1        1              
  Lines         304      329      +25     
  Branches       78       83       +5     
==========================================
+ Hits          299      322      +23     
- Misses          3        4       +1     
- Partials        2        3       +1
Files Changed Coverage Δ
src/protego.py 97.87% <92.85%> (-0.49%) :arrow_down: