search

my8100 / scrapyd-cluster-on-heroku

Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO :point_right:
https://scrapydweb.herokuapp.com/
GNU General Public License v3.0
122 stars 94 forks source link

Published projects disappear and I have to deploy again #16

Open Suchana34 opened 4 years ago

Suchana34 commented 4 years ago

As mentioned in cons , Heroku server by default restarts every 24 hours and removes all published projects done using the ‘scrapyd-deploy’ command. How can I keep the projects published? Please help.

my8100 commented 4 years ago

You can put the egg files here before deploying the Scrapyd app on heroku. https://github.com/my8100/scrapyd-cluster-on-heroku-scrapyd-app/tree/master/eggs

Scrapyd config: https://github.com/my8100/scrapyd-cluster-on-heroku-scrapyd-app/blob/b117252bd8bcabd73b01967ea7da46036e9ce3d1/scrapyd.conf#L3