spinlud
commented
3 years ago Hi @omri08 !
This seems a cool idea, I've just release a @next version with a proxies option in the constructor:
const scraper = new LinkedinScraper({
headless: false,
slowMo: 100,
args: [
"--lang=en-GB",
],
proxies: [
'<proxy1_address>:<proxy1_port>',
'<proxy2_address>:<proxy2_port>',
// ...
]
});To be honest I know very little about proxy usage, I've tried the following fast attempts for testing:
- Using package http-proxy (example taken from here):
const http = require('http'), net = require('net'), httpProxy = require('http-proxy'), url = require('url'), util = require('util');
var proxy = httpProxy.createServer();
var server = http.createServer(function (req, res) { console.log('Receiving reverse proxy request for:' + req.url); var parsedUrl = url.parse(req.url); var target = parsedUrl.protocol + '//' + parsedUrl.hostname; proxy.web(req, res, {target: target, secure: false}); }).listen(8888);
server.on('connect', function (req, socket) { console.log('Receiving reverse proxy request for:' + req.url);
var serverUrl = url.parse('https://' + req.url);
var srvSocket = net.connect(serverUrl.port, serverUrl.hostname, function() {
socket.write('HTTP/1.1 200 Connection Established\r\n' +
'Proxy-agent: Node-Proxy\r\n' +
'\r\n');
srvSocket.pipe(socket);
socket.pipe(srvSocket);
});});
* Using [docker-tinyproxy](https://github.com/monokal/docker-tinyproxy):
```sh
docker run -d --name='tinyproxy' -p 8888:8888 monokal/tinyproxy:latest 'ANY'Both failed. I see some requests forwarded and page loaded but at some point it stops working. Honestly I don't know if this is a problem on the proxy (likely) or on Puppeteer/puppeteer-page-proxy side.
If you want to spend some time diving into it that's great! 😎
You can install this new version using npm install linkedin-jobs-scraper@next.
Let me know if you find something we have to fix to make it working properly 🙃
omri08
Hi @spinlud 😄 What do you think about the option to add a proxy? I saw this library or: https://github.com/Cuadrix/puppeteer-page-proxy for adding a proxy per page/request.
I think the best use case for this option will be in when we getting
429 too many requests. We can add a proxy server from a list of proxy servers the user has and if this proxy server getting also 429 we will update again the proxy server.we can maybe do something like this:
I don't have experience with puppeteer but if you think its something that can be done I don't mind dive into it 😄