"Event loop is closed" exception raised during shutdown

[ErikKalkoken]

Description

I am current building a persistent queue library based on aiosqlite and think I might have found a bug.

When a task is still trying to perform SQL queries during shutdown it will raise the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/home/erik/python/projects/aiodiskqueue/venv/lib/python3.11/site-packages/aiosqlite/core.py", line 121, in run
    get_loop(future).call_soon_threadsafe(set_exception, future, e)
  File "/usr/lib/python3.11/asyncio/base_events.py", line 806, in call_soon_threadsafe
    self._check_closed()
  File "/usr/lib/python3.11/asyncio/base_events.py", line 519, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

In order to allow for a graceful shutdown of a service I think it should log the SQL statement that failed, instead of producing this exception.

Here is a minimal code example to reproduce this behavior:

import asyncio
import aiosqlite

async def consumer(db_path):
    async with aiosqlite.connect(db_path, isolation_level=None) as db:
        for num in range(100):
            message = f"message {num + 1}"
            await db.execute("INSERT INTO messages (message) VALUES (?);", (message,))

async def main():
    db_path = "event_loop_bug.sqlite"
    async with aiosqlite.connect(db_path, isolation_level=None) as db:
        await db.execute("DROP TABLE IF EXISTS messages;")
        await db.execute("CREATE TABLE messages (message TEXT);")
    asyncio.create_task(consumer(db_path))

asyncio.run(main())

Details

• OS: Ubuntu 22.04
• Python version: 3.11
• aiosqlite version: 0.19.0
• Can you repro on 'main' branch? yes
• Can you repro in a clean virtualenv? yes

[davidandreoletti]

@amyreese Currently experiencing this on py312 / aiosqlite v0.20.0 too. @ErikKalkoken PS: On py311 / aiosqlite v0.20.0, I did not experience this error.

Reviewing the code, the event flows is this:

• User thread's asyncio event loop exists and is not closed.
  • Let's call this loop loop1.
• User thread creates a Connection instance.
  • Connection's own thread has also an asyncio event loop. Let's call this loop loop2
• User thread execute a statement on the Connection instance
  • statement is executed on loop2
• User thread closes it own loop1 for (so far) unknown reason.
  • aiosqlite await for the statement's result running on loop2.
  • once available, aiosqlite set the statement's result as the future's result.
    • future's result can't be set (via function set_result / set_exception) since the function can't be executed on the future's binded loop loop1 because loop1 is closed.
• This behaviour yields the exception "Event loop closed" in loop1's execution context

Therefore, any transaction queue item results must forwarded to loop2 before await connection.close() returns.