scrapinghub / frontera

A scalable frontier for web crawlers
BSD 3-Clause "New" or "Revised" License
1.29k stars 215 forks source link

ERROR:sqlalchemy.queue:A transaction is already begun. Use subtransactions=True to allow subtransactions. #378

Open yevgenpapernyk opened 5 years ago

yevgenpapernyk commented 5 years ago

I'm using PostgreSQL as backend for the workers.

The worker config is:

from __future__ import absolute_import
from .common import *

BACKEND = 'frontera.contrib.backends.sqlalchemy.Distributed' 
SQLALCHEMYBACKEND_ENGINE = 'postgresql://postgres:example@localhost'
MAX_NEXT_REQUESTS = 2048 
NEW_BATCH_DELAY = 3.0

The requirements.txt is:

Scrapy>=0.24.4
psycopg2
SQLAlchemy>=0.9.8
msgpack
frontera[sql,hbase,logging,tldextract,kafka,distributed,strategies]

After a while I'm getting this error periodically:

ERROR:sqlalchemy.queue:A transaction is already begun.  Use subtransactions=True to allow subtransactions.
Traceback (most recent call last):
  File "/home/yp/FronteraFromScratch/venv/lib/python3.7/site-packages/frontera/contrib/backends/sqlalchemy/components.py", line 26, in func_wrapper
    return func(self, *args, **kwargs)
  File "/home/yp/FronteraFromScratch/venv/lib/python3.7/site-packages/frontera/contrib/backends/sqlalchemy/components.py", line 213, in schedule
    self.session.commit()
  File "/home/yp/FronteraFromScratch/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1027, in commit
    self.transaction.commit()
  File "/home/yp/FronteraFromScratch/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 506, in commit
    self.close()
  File "/home/yp/FronteraFromScratch/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 588, in close
    self.session.begin()
  File "/home/yp/FronteraFromScratch/venv/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 941, in begin
    "A transaction is already begun.  Use "
sqlalchemy.exc.InvalidRequestError: A transaction is already begun.  Use subtransactions=True to allow subtransactions.