HTTP Agent with keepAlive

bda-research / node-crawler

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

MIT License

6.69k stars 876 forks source link

HTTP Agent with keepAlive #466

Closed AlithAnar closed 8 months ago

AlithAnar commented 8 months ago

Hi, is it possible to support using a custom HTTP agent e.g. with keepAlive enabled for requests? Currently, I have an issue with crawling on some GCP infrastructure, where I can't establish many TCP connections, and I'd rather share some connection pool.

AlithAnar commented 8 months ago

nvm, there is a way to pass custom agent as crawler options