SQLAlchemy Scoped Session - ValueError: generator already executing

[bobbydams]

Hello,

In part of our code we use scoped sessions from SQLAlchemy. Frequently we run into the error ValueError: generator already executing (see screenshot). The only workaround we've found at the moment is to set auto_enabling_integrations=False in the sentry.init function.

Let me know if any additional information is needed. Thanks!

Environment

How do you use Sentry?

on-premise v21.5.1

Which SDK and version?

Python v3.8, Sentry SDK 1.1.0

Steps to Reproduce

Creating a scoped session

scoped_session = db.create_scoped_session()

This bug seems to be triggered by this bit of code:

    def get_files_from_blob(self, files, text=False):
        """get files content in thread pool in bytes or text format"""
        start = time.time()
        pool = ThreadPool(4)
        results = pool.map(partial(self._get_file, text=text), files)

        # close the pool and wait for the work to finish
        pool.close()
        pool.join()
        logger.debug("DEBUG time to get in pool: %d", time.time() - start)
        return results

    def _get_file(self, file, text):
        """..."""
        blob_file = BlobFile(file.filename)
        if text:
            data = blob_file.read_from_blob_to_text().replace("\r", "")
        else:
            data = blob_file.read_from_blob_to_bytes()
        return file, data

Expected Result

There should be no errors thrown by the sentry-sdk

Actual Result

Traceback (most recent call last):
  File "/src/helpers/processing.py", line 165, in instrumented_upload_processing
    processed_data = processor.run_processing()
  File "/src/helpers/processing.py", line 670, in run_processing
    outputs += self.load_hmd()
  File "/src/helpers/processing.py", line 785, in load_hmd
    data = self.get_files_from_blob(self.hmd_files, text=True)
  File "/src/helpers/processing.py", line 556, in get_files_from_blob
    results = pool.map(partial(self._get_file, text=text), files)
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/src/helpers/processing.py", line 566, in _get_file
    blob_file = BlobFile(file.filename)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 294, in __get__
    return self.impl.get(instance_state(instance), dict_)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 725, in get
    value = state._load_expired(state, passive)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/state.py", line 652, in _load_expired
    self.manager.deferred_scalar_loader(self, toload)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 1006, in load_scalar_attributes
    result = load_on_ident(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 200, in load_on_ident
    return load_on_pk_identity(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 286, in load_on_pk_identity
    return q.one()
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3490, in one
    ret = self.one_or_none()
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3459, in one_or_none
    ret = list(self)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
    return self._execute_and_instances(context)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1514, in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1281, in _execute_context
    self.dispatch.after_cursor_execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/event/attr.py", line 322, in __call__
    fn(*args, **kw)
  File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/sqlalchemy.py", line 77, in _after_cursor_execute
    ctx_mgr.__exit__(None, None, None)
  File "/usr/local/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
ValueError: generator already executing

Dependencies

[packages]
Fiona = ">=1"
Flask = "==1.1.1"
Flask-Migrate = "~=2.7.0"
Flask-Executor = "==0.9.1"
Flask-Script = ">=2"
GeoAlchemy2 = "==0.6.3"
Jinja2 = "==2.10.3"
Werkzeug = "==0.16.0"
alembic = "==1.2.1"
azure-mgmt-managementpartner = "==0.1.1"
flask-restplus = "==0.12.1"
geopandas = "==0.4.0"
kubernetes = "==12.0.1"
ray = ">=1"
requests = "==2.25.0"
sendgrid = ">=6"
treelib = "==1.5.5"
gunicorn = ">=20"
numpy = "==1.19.1"
psycopg2 = "==2.8.2"
azure-storage-blob = "==2.1.0"
xlrd = "==1.2.0"
pandas = "==0.25.3"
pytruth = "*"
shapely = "*"
newrelic = "*"
opencensus-ext-azure = "*"
sentry-sdk = {extras = ["flask"], version = "*"}
python-dotenv = "*"
flask-sqlalchemy = "==2.4.4"
sqlalchemy = "==1.3.24"

[sp1rs]

Any update on this?

[github-actions[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[antonpirker]

Hello @bobbydams and @sp1rs !

Could you please verify if this problem still exists with the newest Python SDK and can you give us a minimal example project where we could reproduce this?

Thanks a lot!

[sp1rs]

@antonpirker auto_enabling_integrations=False worked for me. I will try the latest version of sentry and let you know.

[vlmaksimuk]

Faced exactly the same problem. As a result of the research, the case turned out to be in the context manager based on the generator. The generator in python is not thread safe. As a result, this error appears. As a solution, I propose to replace the context manager with a Class-based context manager. Checked on my project, everything works fine

[sl0thentr0py]

ok I spent an afternoon trying to repro this with locust in this sample app but failed to do so. As I pointed out in #1368, the actual problem seems to be setting the variable on conn which is not thread-safe, but if the class based context manager 'fixes' the problem, I'm happy to merge it in.

[sl0thentr0py]

'fixed' by #1368 for now, pls reopen if issue persists.