search

ekalinin / robots.js

Parser for robots.txt for node.js
MIT License
66 stars 21 forks source link

Fix user agent header not actually being sent #34

Closed rune-cats closed 3 years ago

rune-cats commented 3 years ago

The way user agent string is added to header requests in nodejs http module has changed. This change reflects that.