spinlud / linkedin-jobs-scraper

147 stars 40 forks source link

Update Readme to reflect anonymous mode on GCP & Docker #19

Closed ppetruneac closed 1 year ago

ppetruneac commented 3 years ago

@spinlud I just tested the scraper on Docker container on my laptop and the same anonymous mode I get. It might be good to update the description to reflect the same on GCP and Docker containers in case other users stumble across.

By default the scraper will run in anonymous mode (no authentication required). In some environments (e.g. AWS or Heroku; also tested on GCP & Docker containers - run locally) this may be not possible though. You may face the following error message:

Error on running on Docker container:

  scraper:error Error: Protocol error (Target.setDiscoverTargets): Target closed.
  scraper:error     at /node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:71:63
  scraper:error     at new Promise (<anonymous>)
  scraper:error     at Connection.send (/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:70:16)
  scraper:error     at Function.create (/node_modules/puppeteer/lib/cjs/puppeteer/common/Browser.js:95:26)
  scraper:error     at ChromeLauncher.launch (/node_modules/puppeteer/lib/cjs/puppeteer/node/Launcher.js:99:56)
  scraper:error     at async PuppeteerExtra.launch (/node_modules/puppeteer-extra/dist/index.cjs.js:132:25) +0ms
Error: Protocol error (Target.setDiscoverTargets): Target closed.
    at /node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:71:63
    at new Promise (<anonymous>)
    at Connection.send (/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:70:16)
    at Function.create (/node_modules/puppeteer/lib/cjs/puppeteer/common/Browser.js:95:26)
    at ChromeLauncher.launch (/node_modules/puppeteer/lib/cjs/puppeteer/node/Launcher.js:99:56)
    at async PuppeteerExtra.launch (/node_modules/puppeteer-extra/dist/index.cjs.js:132:25)
spinlud commented 3 years ago

@jitboo Probably your container doesn't have the required dependencies to run puppeteer. You can use this image as reference. Also make sure that you are running in headless mode.