search

chrisakroyd / robots-txt-parser

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
MIT License
12 stars 8 forks source link

Wildcards not supported? #1

Closed sanderheilbron closed 3 years ago

sanderheilbron commented 6 years ago

Several search engine bots support wildcards.

I've checked this for the following robots.txt files, and noticed that robots-txt-parser seems like to ignore this for the following robots txt files: https://www.zalando.nl/robots.txt https://www.bcc.nl/robots.txt

But supports it for the following robots txt file: https://www.karwei.nl/robots.txt

Hope you could do a check to see what is causing this behaviour.

chrisakroyd commented 6 years ago

Thanks for bringing this to my attention, I'm looking into it now and I should hopefully have a fix out over the coming days.

keyndark commented 6 years ago

Hi. did you have fix for this? Thanks!

chrisakroyd commented 6 years ago

Hey keyndark, I managed to get tests running that confirmed the issue but I haven't managed to fix one edge case. I'm aiming to have a fix done for Friday this week but that timeline may change.