Added code for dealing with globing inside allow/disalow rules

ekalinin / robots.js

Parser for robots.txt for node.js

MIT License

66 stars 21 forks source link

Closed XooR closed 12 years ago

XooR commented 12 years ago

I thought this is needed, there is some web sites that use this... Also I added two tests for this feature.

Also I removed repetition of case: allow/disalow... and fixed mispelling error

Also I think it is better to use trim function on strings instead replace magic...

Please tell me if there is something else I should do, or if you disagree on adding globing support.

ekalinin commented 12 years ago

Hi @XooR,

Thanks for patch. I think it will be very useful. Applyed.

Best regards, Eugene.