canCrawl() returns incorrect result when matching middle of path

chrisakroyd / robots-txt-parser

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

MIT License

12 stars 8 forks source link

Open Trott opened 1 year ago

Trott commented 1 year ago

robots.txt:

User-agent: *
Disallow: /rss
Allow: /

canCrawl() thinks this means /home/rssa cannot be crawled but that is incorrect.