search

jmrchelani / scrap-this-web

API to scrap HTML CSS and JS of a website
9 stars 1 forks source link

Timeout for large web page #1

Open jmrchelani opened 3 years ago

jmrchelani commented 3 years ago

The api fails to scrap a large web pages. Reason: Request timeouts.

2021-01-14T11:33:36.769049+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path="/?url=https://github.com" host=scrap-this-web.herokuapp.com request_id=d0a2bb59-3479-44b0-9a8c-187e483ea360 fwd="182.182.28.37" dyno=web.1 connect=1ms service=30000ms status=503 bytes=0 protocol=https

jmrchelani commented 3 years ago

@slimcoders Have a look at this issue

MrSlimCoder commented 3 years ago

@jmrchelani we need to create multiple instances of our node.js server. I will suggest to use pm2 within your node.js backend.

jmrchelani commented 3 years ago

But it works totally fine on my local deployment- problem seems to be with heroku..

jmrchelani commented 3 years ago

@jmrchelani we need to create multiple instances of our node.js server. I will suggest to use pm2 within your node.js backend.

Alright i will have a look at PM2 tomorrow