dmclain / scrapy-heroku

BSD 3-Clause "New" or "Revised" License
68 stars 36 forks source link

pscycopg2 Operational Error #2

Closed summera closed 9 years ago

summera commented 9 years ago

Consistently getting a pscycopg2 exception: "SSL connection has been closed unexpectedly"

After reading this (http://stackoverflow.com/questions/26792943/heroku-rails-pg-activerecordstatementinvalid-pgconnectionbad-pqconsum) I figured it was due to having a free tier Hobby Dev DB. I then tried a Hobby Basic and same thing happened. Any ideas for how to solve this and/or what the cause may be?

Full Log Exception:

2014-12-09T12:26:04.194140+00:00 app[web.1]:      File "/app/.heroku/python/lib/python2.7/site-packages/twisted/internet/defer.py", line 1237, in unwindGenerator
2014-12-09T12:26:04.194185+00:00 app[web.1]:        return len(self.q)
2014-12-09T12:26:04.194142+00:00 app[web.1]:        return _inlineCallbacks(None, gen, Deferred())
2014-12-09T12:26:04.194186+00:00 app[web.1]:      File "/app/.heroku/python/lib/python2.7/site-packages/scrapy_heroku/spiderqueue.py", line 80, in __len__
2014-12-09T12:26:04.194143+00:00 app[web.1]:      File "/app/.heroku/python/lib/python2.7/site-packages/twisted/internet/defer.py", line 1099, in _inlineCallbacks
2014-12-09T12:26:04.194188+00:00 app[web.1]:        result = self._execute(q)[0][0]
2014-12-09T12:26:04.194145+00:00 app[web.1]:        result = g.send(result)
2014-12-09T12:26:04.194190+00:00 app[web.1]:      File "/app/.heroku/python/lib/python2.7/site-packages/scrapy_heroku/spiderqueue.py", line 36, in _execute
2014-12-09T12:26:04.194176+00:00 app[web.1]:      File "/app/.heroku/python/lib/python2.7/site-packages/scrapyd/poller.py", line 21, in poll
2014-12-09T12:26:04.194191+00:00 app[web.1]:        cursor.execute(q, args)
2014-12-09T12:26:04.194177+00:00 app[web.1]:        c = yield maybeDeferred(q.count)
2014-12-09T12:26:04.194193+00:00 app[web.1]:    psycopg2.OperationalError: terminating connection due to administrator command
2014-12-09T12:26:04.194129+00:00 app[web.1]: 2014-12-09 12:26:03+0000 [-] Unhandled Error
2014-12-09T12:26:04.194179+00:00 app[web.1]:    --- <exception caught here> ---
2014-12-09T12:26:04.194194+00:00 app[web.1]:    SSL connection has been closed unexpectedly
2014-12-09T12:26:04.194134+00:00 app[web.1]:    Traceback (most recent call last):
2014-12-09T12:26:04.194180+00:00 app[web.1]:      File "/app/.heroku/python/lib/python2.7/site-packages/twisted/internet/defer.py", line 139, in maybeDeferred
2014-12-09T12:26:04.194196+00:00 app[web.1]:    
2014-12-09T12:26:04.194136+00:00 app[web.1]:      File "/app/.heroku/python/lib/python2.7/site-packages/twisted/internet/defer.py", line 139, in maybeDeferred
2014-12-09T12:26:04.194182+00:00 app[web.1]:        result = f(*args, **kw)
2014-12-09T12:26:04.194138+00:00 app[web.1]:        result = f(*args, **kw)
2014-12-09T12:26:04.194183+00:00 app[web.1]:      File "/app/.heroku/python/lib/python2.7/site-packages/scrapy_heroku/spiderqueue.py", line 130, in count
2014-12-09T12:26:04.194197+00:00 app[web.1]: 

Is it possible to catch the exception and try to reconnect when the next poll happens? Thanks!

Cheers, Ari