canCrawl stalls in nodejs but not browser

chrisakroyd / robots-txt-parser

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

MIT License

11 stars 7 forks source link

Open salivity opened 2 weeks ago

salivity commented 2 weeks ago

I created a project at https://davidclews.com/article/158.html and the robots.txt for https://www.homebase.co.uk/robots.txt appears a parse correctly but in node 20 when running canCrawl on a url the function stalls with no exception. What could this be?

salivity commented 2 weeks ago

did further checks seems to be this part

User-agent: *
Disallow: /*elysium.search***
Allow:  *******/*IFP=true*