search

dcts / opensea-scraper

Scrapes nft floor prices and additional information from opensea. Used for https://nftfloorprice.info
MIT License
184 stars 73 forks source link

Not working when deployed (Heroku) #39

Open jervalles opened 2 years ago

jervalles commented 2 years ago

This is not working for me when I deploy the app on heroku but working fine in local. Is that the same for you?

image

Wanna use this package because a collection can have different categories of NFT and OpeanSea's API don't provide to filter by trait

dcts commented 2 years ago

Deploying scrapers to heroku or other providers (google cloud, AWS) is always trickier than running from local environment, because opensea might be able to detect IP addresses of those providers.

I use google cloud (firebase functions) and it seems to work well with puppeteer, I will quickly test my setup and let you know.

jervalles commented 2 years ago

Thank you for the fast answer!

Your package is just amazing and would love to work with it! if you have a solution I will be happy

dcts commented 2 years ago

I could not make it work on firebase cloud function as well (for now). What is weird is that I have a setup that is deployed and working, but could not reproduce. Sorry I can't give a more helpful answer as of now, maybe with a fresh mind I will be able to figure out and will post here.

If you find out something else please share, might be interesting for folks using heroku too.

MY error when hosted on firebase cloud functions (for future reference):

{
  status: 'error',
  errorMessage: "Evaluation failed: TypeError: Cannot read properties of null (reading 'innerText')\n" +
    '    at <anonymous>:2:65',
  errorStack: "Error: Evaluation failed: TypeError: Cannot read properties of null (reading 'innerText')\n" +
    '    at <anonymous>:2:65\n' +
    '    at ExecutionContext._evaluateInternal (/workspace/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:221:19)\n' +
    '    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n' +
    '    at async ExecutionContext.evaluate (/workspace/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:110:16)\n' +
    '    at async Object.rankings (/workspace/node_modules/opensea-scraper/src/functions/rankings.js:50:25)\n' +
    '    at async /workspace/index.js:9:23'
}
jervalles commented 2 years ago

Where is your working setup deployed? Cause I've just tried heroku but if I can use something else...

jervalles commented 2 years ago

tried to put a node server to a dedicated NAS with your solution and I got this as error. My local my and this server got the same project (from git)

image

jervalles commented 2 years ago

related to this: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#running-puppeteer-on-heroku

How can I add "no--sandbox" on your package?

edit : well I forked and added this. without succcess (same error)

dcts commented 2 years ago

Thanks for reporting, it seems to be hard to deploy on any cloud infrastructure currently, also google cloud (see #40 ). This is not a problem specific to this package, but a general challenge when deploying scrapers on cloud infrastructure. We can keep this open in case someone finds a solution.