search

my8100 / scrapyd-cluster-on-heroku

Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO :point_right:
https://scrapydweb.herokuapp.com/
GNU General Public License v3.0
122 stars 94 forks source link

[Question] Postgresql setup #12

Open andreierdoss opened 4 years ago

andreierdoss commented 4 years ago

I added the Heroku Postgres add on, which creates the DATABASE_URL

Once I restarted the scrapydweb server I got this error:

2020-05-23T07:13:33.128467+00:00 heroku[web.1]: Starting process with command `scrapydweb`
2020-05-23T07:13:35.447284+00:00 app[web.1]: Traceback (most recent call last):
2020-05-23T07:13:35.447303+00:00 app[web.1]: File "/app/.heroku/python/bin/scrapydweb", line 5, in <module>
2020-05-23T07:13:35.447478+00:00 app[web.1]: from scrapydweb.run import main
2020-05-23T07:13:35.447489+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/scrapydweb/__init__.py", line 14, in <module>
2020-05-23T07:13:35.447642+00:00 app[web.1]: from .common import handle_metadata
2020-05-23T07:13:35.447652+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/scrapydweb/common.py", line 15, in <module>
2020-05-23T07:13:35.447771+00:00 app[web.1]: from .models import Metadata, db
2020-05-23T07:13:35.447781+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/scrapydweb/models.py", line 8, in <module>
2020-05-23T07:13:35.447901+00:00 app[web.1]: from .vars import STATE_RUNNING
2020-05-23T07:13:35.447911+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/scrapydweb/vars.py", line 61, in <module>
2020-05-23T07:13:35.448037+00:00 app[web.1]: results = setup_database(DATABASE_URL, DATABASE_PATH)
2020-05-23T07:13:35.448047+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/scrapydweb/utils/setup_database.py", line 37, in setup_database
2020-05-23T07:13:35.448172+00:00 app[web.1]: setup_postgresql(*m_postgres.groups())
2020-05-23T07:13:35.448194+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/scrapydweb/utils/setup_database.py", line 134, in setup_postgresql
2020-05-23T07:13:35.448345+00:00 app[web.1]: conn = psycopg2.connect(host=host, port=int(port), user=username, password=password)
2020-05-23T07:13:35.448355+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.7/site-packages/psycopg2/__init__.py", line 127, in connect
2020-05-23T07:13:35.448492+00:00 app[web.1]: conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
2020-05-23T07:13:35.448526+00:00 app[web.1]: psycopg2.OperationalError: FATAL:  database "wrzusiwvpowsmj" does not exist

What else do I need to do to get this working?

my8100 commented 4 years ago

What’s the value of DATABASE_URL? How did you install scrapydweb, via pip or git?