search

t1gor / Robots.txt-Parser-Class

Php class for robots.txt parse
MIT License
83 stars 31 forks source link

Crawl-Delay not supported #4

Closed troelskn closed 10 years ago

troelskn commented 10 years ago

Hi,

Nice class. I doesn't recognise the directive for Crawl-Delay though. (See: http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive and https://ahrefs.com/robot/)

t1gor commented 10 years ago

Hi. Thanks for the suggestion.

I will look over this and update the code shortly. Alternatively - you could submit a pull-request.

Thanks, Igor

t1gor commented 10 years ago

It would also be very helpful if you could provide an example of the file that you are trying to parse, to know what should we expect.

troelskn commented 10 years ago

The ahrefs link shows an example. I'll have a look at making a pull request.

troelskn commented 10 years ago

Opened pull request #5