search

samclarke / robots-parser

NodeJS robots.txt parser with support for wildcard (*) matching.
MIT License
148 stars 19 forks source link

Support for URL object #11

Closed gsouf closed 3 years ago

gsouf commented 5 years ago

Hi,

With version we could use url objects that way:

let myUrl = new URL('http://foo.bar');
robotsTxt.isAllowed(myUrl);

With version 2 it's no longer possible and for this reason existing code breaks when upgrading.

The reason is that URL.parse(url) that was used in version 1 accepts URL object as parameter when new URL.URL(url) does not.

Can we consider adding back the support for url objects?

samclarke commented 5 years ago

That's a great idea! I hadn't realised there was a difference between the old node url.parse method and the new URL constructor but it would make sense to support URL objects.

samclarke commented 5 years ago

I've tested this and it seems to be working with Node 8 and in browsers. I think the URL is being converted into a string and then parsed correctly. What platform are you getting the error on?