search

chrisakroyd / robots-txt-parser

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
MIT License
12 stars 8 forks source link

Handing empty robots.txt string #18

Open tcz opened 2 months ago

tcz commented 2 months ago

I got bugs when the robots.txt was empty on the server, possibly due to it missing altogether.

TypeError: string.match is not a function or its return value is not iterable
  at robustSplit (src/parser.js:23:50)
  at parser (src/parser.js:62:13)
  at Context.<anonymous> (test/parser/can-parse-test-files.js:72:25)
  at process.processImmediate (node:internal/timers:478:21)