search

spinlud / linkedin-jobs-scraper

151 stars 41 forks source link

Works locally but can't make it work on remote server #3

Closed stoodkev closed 4 years ago

stoodkev commented 4 years ago

Hi, it works perfectly locally but I ve tried on both Amazon Linux 2 and Ubuntu 18 lts and I cant make it work. I ve downloaded all the related dependencies and ran your examples, it runs without error but it doesnt fetch any job.

On my custom implementation of your code I get this error although it works fine locally:

`Error: Execution context was destroyed, most likely because of a navigation. scraper:error at rewriteError (/var/ftjet-api/node_modules/puppeteer/lib/ExecutionContext.js:167:15) scraper:error -- ASYNC -- scraper:error at ExecutionContext. (/var/ftjet-api/node_modules/puppeteer/lib/helper.js:111:15) scraper:error at DOMWorld.evaluate (/var/ftjet-api/node_modules/puppeteer/lib/DOMWorld.js:112:20) scraper:error -- ASYNC -- scraper:error at Frame. (/var/ftjet-api/node_modules/puppeteer/lib/helper.js:111:15) scraper:error at Page.evaluate (/var/ftjet-api/node_modules/puppeteer/lib/Page.js:860:43) scraper:error at Page. (/var/ftjet-api/node_modules/puppeteer/lib/helper.js:112:23)

Any idea what could the issue be? Anyways, thanks for your work! `

EDIT : It seems that whenever I m trying to run this on AWS, linkedin redirects all links to the authwall. Have you managed to get this to work on a remote server?

spinlud commented 4 years ago

Hi! I think the problems you are having are caused by the absence of gtk-related libraries on the Amazon Linux AMI. If you install chrome dependencies on the machine Puppeteer should fetch pages correctly. I've tested the library on an EC2 running Ubuntu Server 18.04 LTS (free tier eligible) and it worked without problems.

Hope this helps, cheers

stoodkev commented 4 years ago

Hi and thanks for your answer . It's weird I found out the reason it's not working on my end is that when I'm on a remote server , LinkedIn knows it and redirects me to an authwall.

On Thu, Mar 5, 2020, 18:06 spinlud notifications@github.com wrote:

Hi! I think the problems you are having are caused by the absence of gtk-related libraries on the Amazon Linux AMI. If you install chrome dependencies on the machine Puppeteer should fetch pages correctly. I've tested the library on an EC2 running Ubuntu Server 18.04 LTS (free tier eligible) and it worked without problems.

Hope this helps, cheers

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/spinlud/linkedin-jobs-scraper/issues/3?email_source=notifications&email_token=AHXAF3MKELGV3VAANH5SB4TRF52Q7A5CNFSM4KYHXXRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEN4SEFI#issuecomment-595141141, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHXAF3N5LHKPCFAENO6FLYDRF52Q7ANCNFSM4KYHXXRA .