asyncio functionality is mostly unusable after async handling was applied

[pmeier]

Reporting on my own failures here. In #90 we added automatic async handling and did so by using asyncio.run:

https://github.com/smarie/mkdocs-gallery/blob/8a16972a3d1dd9728a5406bb83f4fe631b9ad4c6/src/mkdocs_gallery/gen_single.py#L782-L783

The problem is, and I didn't know this before, that asyncio.run is meant to be only called once. This is what the documentation says about this:

This function should be used as a main entry point for asyncio programs, and should ideally only be called once.

However, we call it for every code block that needs async handling. That in itself is not an issue as the gallery here builds just fine. Unfortunately, it breaks down when other libraries that are part of the documentation building also need to use some async functionality.

For example, panel creates an asyncio.Lock on import. When building some async galleries before the import, we effectively have the following situation

import asyncio
import asyncio.events

async def foo():
    pass

asyncio.run(foo())

loop = asyncio.events.get_event_loop()

And the get_event_loop fails with

RuntimeError: There is no current event loop in thread 'MainThread'.

because we have previously used asyncio.run.

My absolute crude workaround for now is the following snippet that has to be placed before any further asyncio functionality is used:

import asyncio

asyncio.get_event_loop_policy()._local._set_called = False

Please note that this works for my case, but I have no idea if this screws up other cases as we are messing with low-level private attributes here.

I'll look on how to restructure the async handling to avoid using asyncio.run to avoid this mess all together.

[pmeier]

A simple solution is to replace

https://github.com/smarie/mkdocs-gallery/blob/8a16972a3d1dd9728a5406bb83f4fe631b9ad4c6/src/mkdocs_gallery/gen_single.py#L783

with

__loop__ = __asyncio__.new_event_loop()
try:
    __async_wrapper_locals__ = __loop__.run_until_complete(__async_wrapper__())
finally:
    __loop__.close()

For our purpose this is equivalent to asyncio.run, but doesn't set the _set_called flag that prevents more event loops from being created.

We could also have one global event loop instead of recreating it every time for the async handling. I'll have a look at how difficult it would be to fit in.

[pmeier]

Here is what IPython does. They have a function that resembles asyncio.get_event_loop (deprecated in 3.12) to potentially create and return a singleton global event loop. This event loop is then used to just call run_until_complete on it.

We could potentially vendor this function and always insert it into the globals of the galleries. That way we can just call

__get_asyncio_loop__().run_until_complete(__async_wrapper__())

in our wrapper.