search

anshulrgoyal / imdb-scrapper

This is small project it provide many good api for search ,scrapping ,trending
39 stars 32 forks source link

Support for proxies #33

Closed T-vK closed 3 years ago

T-vK commented 3 years ago

I would like to use this library, but I need to access the Internet through a socks4 proxy. Requests can be proxies through a socks proxy using the "agent" option of the http or https modules. E.g.

import https from 'https'
import SocksProxyAgent from 'socks-proxy-agent'

const proxyAgent = new SocksProxyAgent("socks4://1.2.3.4:35618")

https.get('https://ifconfig.io', {agent: proxyAgent}, (res) => {
    console.log(res.headers)
    res.pipe(process.stdout)
})

Is there any chance we can add an option to allow people to specify a custom agent? And if so, do you have any thoughts on how you would like to see this implemented?

Edit: Being able to set the lookup option as well would be nice because this would allow using a custom DNS server, which in some environments is necessary.

Maybe it would make sense to allow passing any option you like to this library.

anshulrgoyal commented 3 years ago

HI @T-vK you can import all the scrapper functions and create your scrap function with any HTTP client.

anshulrgoyal commented 3 years ago

https://github.com/anshulrgoyal/imdb-scrapper/blob/9f9c7e342526f2aa6f1c78453ca66b5b148352a8/index.js#L51