Open midnightfreddie opened 6 years ago
When providing a string user agent to the constructor or allowing the default user agent string, it is built as
"headers":{"userAgent":"<user agent string>"}
Servers do not log this header; they show the user agent field as "-". The option needs to be styled as
"headers":{"User-Agent":"<user agent string>"}
Workaround: Pass an object to constructor instead:
var parser = new robots.RobotsParser( 'http://nodeguide.ru/robots.txt', { "headers": { "User-Agent": "My User Agent String" } }, after_parse );
When providing a string user agent to the constructor or allowing the default user agent string, it is built as
Servers do not log this header; they show the user agent field as "-". The option needs to be styled as
Workaround: Pass an object to constructor instead: