sruusk / esea-scraper

A consumable Node package to scrape data from ESEA API
https://www.npmjs.com/package/esea-scraper
Other
3 stars 2 forks source link

Out of curiosity, how do you run this in a deployed web app? #191

Closed saddiqs1 closed 1 year ago

saddiqs1 commented 1 year ago

I'm currently trying to run this within a vercel deployment, but am struggling to get ulixee working. Any ideas/suggestions on how to get it to work?

sruusk commented 1 year ago

There's a couple things that might cause you problems.

  1. You need x86 architecture CPU to run the browser and you may not be able to install it on a managed deployment.
  2. You can't run it from an datacenter IP, as cloudflare will block you based on the IP address.
  3. You may need to run an install script to setup the browser.

As for what I'm running:

Here's my options for the scraper:

new eseaScraper.EseaScraper({
        useLocalHero: false,
        heroOverrides: {
            connectionToCore: {
                host: 'ws://OpenVPNTunnel:7007',
            }
        },
        timeout: 30 * 1000
});

Instead of the OpenVPN Tunnel, you could use an upstream proxy

saddiqs1 commented 1 year ago

For my needs I think this isn't suitable, so i've decided to go with a completely different route altogether now. Thanks for the response though!