AttributeError: 'AnnotatedValue' object has no attribute 'scramble_native_password'

[cnicodeme]

Hi!

I'm having this exception popping up on my code. I initially thought there was some kind of versioning issue but after restarting everything, the issue still happens.

Here's the full stack:

raised unexpected: AttributeError("'AnnotatedValue' object has no attribute 'scramble_native_password'")
Traceback (most recent call last):
  File "/var/www/project/env/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/var/www/project/env/lib/python3.9/site-packages/sentry_sdk/integrations/celery.py", line 231, in _inner
    reraise(*exc_info)
  File "/var/www/project/env/lib/python3.9/site-packages/sentry_sdk/_compat.py", line 60, in reraise
    raise value
  File "/var/www/project/env/lib/python3.9/site-packages/sentry_sdk/integrations/celery.py", line 226, in _inner
    return f(*args, **kwargs)
  File "/var/www/project/www/extensions/celery_worker.py", line 167, in __call__
    self.retry(exc=e)
  File "/var/www/project/env/lib/python3.9/site-packages/celery/app/task.py", line 717, in retry
    raise_with_context(exc)
  File "/var/www/project/www/extensions/celery_worker.py", line 160, in __call__
    loop.run_until_complete(self._async_run(*args, **kwargs))
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/var/www/project/www/extensions/celery_worker.py", line 135, in _async_run
    await await_if_needed(Task.__call__, self, *args, **kwargs)
  File "/var/www/project/www/utils/__init__.py", line 273, in await_if_needed
    return await result
  File "/var/www/project/www/apps/webhooks/tasks.py", line 52, in process_incoming_email
    raise e
  File "/var/www/project/www/apps/webhooks/tasks.py", line 47, in process_incoming_email
    await process_incoming_email_sub(bucket_name, object_key, delivered_to, remote_ip, return_path, is_spam)
  File "/var/www/project/www/apps/webhooks/tasks.py", line 227, in process_incoming_email_sub
    channel = await Channel.find_by_email(delivered_to)
  File "/var/www/project/www/apps/organizations/models.py", line 855, in find_by_email
    return await query.first()
  File "/var/www/project/www/surge/database/models.py", line 36, in first
    return (await self.execute()).scalars().first()
  File "/var/www/project/www/surge/database/models.py", line 33, in execute
    return await db.read.execute(self)
  File "/var/www/project/www/surge/database/database.py", line 107, in execute
    return await super().execute(statement, parameters)
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/ext/asyncio/session.py", line 313, in execute
    result = await greenlet_spawn(
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 165, in greenlet_spawn
    result = context.throw(*sys.exc_info())
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2229, in execute
    return self._execute_internal(
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2114, in _execute_internal
    conn = self._connection_for_bind(bind)
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1981, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
  File "<string>", line 2, in _connection_for_bind
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 137, in _go
    ret_value = fn(self, *arg, **kw)
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1108, in _connection_for_bind
    conn = bind.connect()
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3251, in connect
    return self._connection_cls(self)
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
    self._dbapi_connection = engine.raw_connection()
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3275, in raw_connection
    return self.pool.connect()
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 455, in connect
    return _ConnectionFairy._checkout(self)
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 1271, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 719, in checkout
    rec = pool._do_get()
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 283, in _do_get
    return self._create_connection()
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 396, in _create_connection
    return _ConnectionRecord(self)
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 681, in __init__
    self.__connect()
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 906, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 901, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 636, in connect
    return dialect.connect(*cargs, **cparams)
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 580, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/dialects/mysql/asyncmy.py", line 279, in connect
    await_only(self.asyncmy.connect(*arg, **kw)),
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 102, in await_only
    return current.driver.switch(awaitable)  # type: ignore[no-any-return]
  File "/var/www/project/env/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 160, in greenlet_spawn
    value = await result
  File "asyncmy/connection.pyx", line 1299, in _connect
    await conn.connect()
  File "asyncmy/connection.pyx", line 550, in connect
    raise e
  File "asyncmy/connection.pyx", line 528, in asyncmy.connection.Connection.connect
    await self._request_authentication()
  File "asyncmy/connection.pyx", line 748, in _request_authentication
    authresp = auth.scramble_native_password(self._password, self.salt)
AttributeError: 'AnnotatedValue' object has no attribute 'scramble_native_password'

Let me know if you need any other details! (Oddly, restart Celery kind of resolve the issue for a while, then it appears again !)

[cnicodeme]

Update: Again, I got the error but now it's not on Celery and on the web server directly (so definitely not related to Celery)

[cnicodeme]

I'm suspecting that somehow, at some point, the auth module is modified to be an AnnotatedValue by who-knows-what, causing this lib to stop functioning properly.

Does that ring something?

[cnicodeme]

I made some progress about the issue, and it is related to ... Sentry ! It wraps the object, causing further issues down the line:

The auth object becomes:

<sentry_sdk.utils.AnnotatedValue object at 0x7fb3a32abc40>
<class 'sentry_sdk.utils.AnnotatedValue'>

[cnicodeme]

I think I can close the issue here, since it's not related to Asyncmy but to Sentry