asyncio: #8 consider contextvars

[delfick]

Make it so everything gets executed in the same asyncio context

I've never worked with contextvars before and what I did here was very non obvious.

So before I write more tests and docs, can you give this a shot please @andredias ?

Thanks.

[delfick]

originally I tried to make it so that each test got their own context, but it's too difficult to know what should stick around from fixtures of different scopes so I've made one context for everything.

[andredias]

Thanks! Please, give me a couple of days.

[delfick]

cool. No rush :)

[delfick]

did you still need this @andredias ?

[andredias]

no, I don't. Thank you very much!

Em qui, 6 de out de 2022 00:28, Stephen Moore @.***> escreveu:

did you still need this @andredias https://github.com/andredias ?

— Reply to this email directly, view it on GitHub https://github.com/delfick/alt-pytest-asyncio/pull/9#issuecomment-1269260153, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG4LDGU6BKVC523KCOG3CDWBZBOVANCNFSM5GDMXHHQ . You are receiving this because you were mentioned.Message ID: @.***>

[delfick]

mmkay, I'm gonna close this then. If you (or anyone else!) wants this, add a comment and I'll make it work again :)

[andredias]

It works for the simplest example I gave before. Unfortunately, when it gets a bit more complicated, it fails:

from collections.abc import AsyncIterable
from pathlib import Path

from databases import Database
from pytest import fixture
from sqlalchemy import text

@fixture(scope="session")
async def db() -> AsyncIterable[Database]:
    db: Database = Database("sqlite:///example.db")
    await db.connect()
    query = """
create table produto (
    id integer primary key,
    name text not null,
    email text not null,
    unique(email)
)
"""
    await db.execute(query)
    try:
        yield db
    finally:
        await db.disconnect()
        Path("exammple.db").unlink()

@fixture
async def trans(db: Database) -> AsyncIterable[Database]:
    async with db.transaction(force_rollback=True):
        yield db

async def test_db(trans: Database) -> None:
    query = """insert into produto (id, name, email) values (1, 'Fulano', 'fulano@email.com')"""
    await trans.execute(text(query))

The error is:

$ pytest 
=================================== test session starts ===================================
platform linux -- Python 3.12.3, pytest-8.2.1, pluggy-1.5.0
rootdir: /tmp/test_alt
plugins: alt-pytest-asyncio-0.7.2
collected 1 item                                                                          

tests/test_database.py .E                                                           [100%]

========================================= ERRORS ==========================================
______________________________ ERROR at teardown of test_db _______________________________
  + Exception Group Traceback (most recent call last):
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 341, in from_call
  |     result: Optional[TResult] = func()
  |                                 ^^^^^^
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 241, in <lambda>
  |     lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
  |     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
  |     raise exception.with_traceback(exception.__traceback__)
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
  |     teardown.throw(exception)  # type: ignore[union-attr]
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/_pytest/threadexception.py", line 92, in pytest_runtest_teardown
  |     yield from thread_exception_runtest_hook()
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook
  |     yield
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
  |     teardown.throw(exception)  # type: ignore[union-attr]
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/_pytest/unraisableexception.py", line 95, in pytest_runtest_teardown
  |     yield from unraisable_exception_runtest_hook()
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook
  |     yield
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
  |     teardown.throw(exception)  # type: ignore[union-attr]
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/_pytest/logging.py", line 857, in pytest_runtest_teardown
  |     yield from self._runtest_for(item, "teardown")
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/_pytest/logging.py", line 833, in _runtest_for
  |     yield
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
  |     teardown.throw(exception)  # type: ignore[union-attr]
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/_pytest/capture.py", line 883, in pytest_runtest_teardown
  |     return (yield)
  |             ^^^^^
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
  |     res = hook_impl.function(*args)
  |           ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 188, in pytest_runtest_teardown
  |     item.session._setupstate.teardown_exact(nextitem)
  |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 559, in teardown_exact
  |     raise BaseExceptionGroup("errors during test teardown", exceptions[::-1])
  | ExceptionGroup: errors during test teardown (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 546, in teardown_exact
    |     fin()
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/_pytest/fixtures.py", line 1020, in finish
    |     raise exceptions[0]
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/_pytest/fixtures.py", line 1009, in finish
    |     fin()
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/alt_pytest_asyncio/async_converters.py", line 269, in finalizer
    |     _run_and_raise(ctx, loop, info, generator, async_finalizer())
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/alt_pytest_asyncio/async_converters.py", line 161, in _run_and_raise
    |     _raise_maybe(func, info)
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/alt_pytest_asyncio/async_converters.py", line 143, in _raise_maybe
    |     raise_error()
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/alt_pytest_asyncio/async_converters.py", line 138, in raise_error
    |     raise info["e"]
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/alt_pytest_asyncio/async_converters.py", line 310, in async_runner
    |     return await func(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/tmp/test_alt/tests/test_database.py", line 26, in db
    |     Path("exammple.db").unlink()
    |   File "/home/andre/.pyenv/versions/3.12.3/lib/python3.12/pathlib.py", line 1342, in unlink
    |     os.unlink(self)
    | FileNotFoundError: [Errno 2] No such file or directory: 'exammple.db'
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 546, in teardown_exact
    |     fin()
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/_pytest/fixtures.py", line 1020, in finish
    |     raise exceptions[0]
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/_pytest/fixtures.py", line 1009, in finish
    |     fin()
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/alt_pytest_asyncio/async_converters.py", line 269, in finalizer
    |     _run_and_raise(ctx, loop, info, generator, async_finalizer())
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/alt_pytest_asyncio/async_converters.py", line 161, in _run_and_raise
    |     _raise_maybe(func, info)
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/alt_pytest_asyncio/async_converters.py", line 143, in _raise_maybe
    |     raise_error()
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/alt_pytest_asyncio/async_converters.py", line 138, in raise_error
    |     raise info["e"]
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/alt_pytest_asyncio/async_converters.py", line 310, in async_runner
    |     return await func(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/tmp/test_alt/tests/test_database.py", line 31, in trans
    |     async with db.transaction(force_rollback=True):
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/databases/core.py", line 426, in __aexit__
    |     await self.rollback()
    |   File "/tmp/test_alt/.venv/lib/python3.12/site-packages/databases/core.py", line 471, in rollback
    |     assert self._connection._transaction_stack[-1] is self
    |            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^
    | IndexError: list index out of range
    +------------------------------------

Isn't creating a backport for Runner to run on older versions feasible? It feels like the best solution.

[delfick]

@andredias ok, so the first problem is your test has a typo and says "exammple.db" instead of "example.db"

The second problem is that when you let it work out which connection to use, it'll use one per asyncio.Task, which doesn't work because we create new asyncio.Tasks all the time in this plugin.

So you want something more like

from collections.abc import AsyncIterable
from pathlib import Path

from databases import Database
from databases.core import Connection
from pytest import fixture
from sqlalchemy import text

@fixture(scope="session")
async def db() -> AsyncIterable[Database]:
    db: Database = Database("sqlite:///example.db")
    await db.connect()
    query = """
create table if not exists produto (
    id integer primary key,
    name text not null,
    email text not null,
    unique(email)
) 
"""
    await db.execute(query)
    try:
        yield db
    finally:
        await db.disconnect()
        Path("example.db").unlink(missing_ok=True)

@fixture
async def nondurable_conn(db: Database) -> AsyncIterable[Connection]:
    async with db.connection() as connection:
        async with connection.transaction(force_rollback=True):
            yield connection

async def test_db(nondurable_conn: Database) -> None:
    query = """insert into produto (id, name, email) values (1, 'Fulano', 'fulano@email.com')"""
    await nondurable_conn.execute(text(query))

[andredias]

Hi, Stephen,

Thanks for your response!

You're right: I could pass a connection to other functions that need it as we usually do with SQLAlchemy. However, one of the features I like most in encode/databases is not doing that as it relies on contextvars to get the right one ( https://www.encode.io/databases/connections_and_transactions/#transactions). That keeps interfaces much simpler.

Regards,

André

Em seg., 27 de mai. de 2024 às 09:18, Stephen Moore < @.***> escreveu:

@andredias https://github.com/andredias ok, so the first problem is your test has a typo and says "exammple.db" instead of "example.db"

The second problem is that when you let it work out which connection to use, it'll use one per asyncio.Task, which doesn't work because we create new asyncio.Tasks all the time in this plugin.

So you want something more like

from collections.abc import AsyncIterablefrom pathlib import Path from databases import Databasefrom databases.core import Connectionfrom pytest import fixturefrom sqlalchemy import text

@fixture(scope="session")async def db() -> AsyncIterable[Database]: db: Database = Database("sqlite:///example.db") await db.connect() query = """create table if not exists produto ( id integer primary key, name text not null, email text not null, unique(email)) """ await db.execute(query) try: yield db finally: await db.disconnect() Path("example.db").unlink(missing_ok=True)

@fixtureasync def nondurable_conn(db: Database) -> AsyncIterable[Connection]: async with db.connection() as connection: async with connection.transaction(force_rollback=True): yield connection

async def test_db(nondurable_conn: Database) -> None: query = """insert into produto (id, name, email) values (1, 'Fulano', @.***')""" await nondurable_conn.execute(text(query))

— Reply to this email directly, view it on GitHub https://github.com/delfick/alt-pytest-asyncio/pull/9#issuecomment-2133363150, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG4LDHXFAES3FN2GVZLD6DZEMQDFAVCNFSM5GDMXHH2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJTGMZTMMZRGUYA . You are receiving this because you were mentioned.Message ID: @.***>

[delfick]

You're right: I could pass a connection to other functions that need it as we usually do with SQLAlchemy. However, one of the features I like most in encode/databases is not doing that as it relies on contextvars to get the right one

Well, the contextvars part works fine. The part that doesn't work is that it caches which connection is available based on the current asyncio.Task. And fundamentally the way this plugin works is by creating new asyncio.Task objects whenever it runs a test or fixture.

That keeps interfaces much simpler.

unsolicited advice, but certainly my default opinion would be that for anything more than a random script, you would find that convenience very limiting in the future in a way that would be very difficult to reverse.

[andredias]

I have been doing some async programming but am no expert. After reading PEP 555 Context-local variables (https://peps.python.org/pep-0555/) -- which has a good rationale but was withdrawn--, and PEP 567 – Context Variables (https://peps.python.org/pep-0567/), I understand that contextvars is a standard module with legitimate uses in asynchronous programming to provide context-local variables.

The problem with the plugin is that the fixtures and tests don't share the same context, right? For example, the database fixture in the example caches a connection in its context but that context isn't shared with tests.

Asyncio.Runner would be a good solution according to this ( https://github.com/python/cpython/blob/d4680b9e17815140b512a399069400794dae1f97/Lib/asyncio/runners.py#L39 ):

This can be useful for interactive console (e.g. IPython),
unittest runners, console tools, -- everywhere when async code
is called from existing sync framework and where the preferred single
asyncio.run() call doesn't work.

The issue is that requires Python 3.11 or greater. Couldn't it be backported? At least for Python 3.8 as Python 3.6 is not supported anymore ( https://devguide.python.org/versions/)?

Regards,

André

Em qui., 30 de mai. de 2024 às 19:03, Stephen Moore < @.***> escreveu:

You're right: I could pass a connection to other functions that need it as we usually do with SQLAlchemy. However, one of the features I like most in encode/databases is not doing that as it relies on contextvars to get the right one

Well, the contextvars part works fine. The part that doesn't work is that it caches which connection is available based on the current asyncio.Task. And fundamentally the way this plugin works is by creating new asyncio.Task objects whenever it runs a test or fixture.

That keeps interfaces much simpler.

unsolicited advice, but certainly my default opinion would be that for anything more than a random script, you would find that convenience very limiting in the future in a way that would be very difficult to reverse.

— Reply to this email directly, view it on GitHub https://github.com/delfick/alt-pytest-asyncio/pull/9#issuecomment-2140929162, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG4LDA37EFOS4AWTCLBXODZE6OZZAVCNFSM5GDMXHH2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJUGA4TEOJRGYZA . You are receiving this because you were mentioned.Message ID: @.***>

[delfick]

well, this PR makes the library 3.11+ so I get rid of the problem of needing at least python 3.11

As I said I definitely don't have time to use asyncio.Runners. Also I don't think that will solve your problem. This PR supports contextvars fine, and I'm gonna merge it now and release a new version.

The problem is this code https://github.com/encode/databases/blob/0.9.0/databases/core.py#L89. It is able to find the connections it stores, but it's finding them based on the current task. And the way this plugin runs everything as async is by running each function as it's own asyncio.Task (necessary for the timeout stuff and error handling)

[delfick]

I've released version 0.8.0.

Please don't be afraid to keep seeking assistance :)