Async with connection pools

[remy-sl]

I think connection pools do not work well with async. I may be doing something wrong of course. Here goes nothing:

import asyncio
import bonsai

client = bonsai.LDAPClient("ldap://opendj:1389")
client.set_credentials(mechanism="SIMPLE", user="cn=Directory Manager", password="password")
pool = bonsai.pool.ConnectionPool(client, minconn=1, maxconn=2, is_async=True)
pool.open()

async def test():
    conn = await pool.get()
    pool.put(conn)
    conn = await pool.get()

loop = asyncio.get_event_loop()
result = loop.run_until_complete(test())

This results in

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-11-07b69a060a6c> in <module>
     14
     15 loop = asyncio.get_event_loop()
---> 16 result = loop.run_until_complete(test())

/usr/local/lib/python3.8/asyncio/base_events.py in run_until_complete(self, future)
    614             raise RuntimeError('Event loop stopped before Future completed.')
    615
--> 616         return future.result()
    617
    618     def stop(self):

<ipython-input-11-07b69a060a6c> in test()
     10     conn = await pool.get()
     11     pool.put(conn)
---> 12     conn = await pool.get()
     13
     14

RuntimeError: cannot reuse already awaited coroutine

What also makes me think that the async implementation was not designed to work with connection pools is the fact that pool.spawn does not work at all because it is wrapped in contextlib.contextmanager and obviously lacks __aenter__ and __aexit__ methods so it fails when used with async with.

Any advice on how to bite this? I am designing an application for quite high throughput.

EDIT:

Okay, I think I have hacked my way around it like this

import asyncio
from contextlib import asynccontextmanager

import bonsai

class AsyncConnectionPool(bonsai.pool.ConnectionPool):
    def __init__(self, *args, **kwargs):
        super().__init__(is_async=True, *args, **kwargs)

    @asynccontextmanager
    async def spawn(self, *args, **kwargs):
        try:
            if self._closed:
                self.open()
            conn = self.get(*args, **kwargs)
            if conn.closed:
                await conn.open()
            yield conn
        finally:
            self.put(conn)

client = bonsai.LDAPClient("ldap://opendj:1389")
client.set_credentials(mechanism="SIMPLE", user="cn=dm", password="cangetinds")
pool = AsyncConnectionPool(client, minconn=1, maxconn=2)

async def test():
    async with pool.spawn() as conn1:
        print(await conn1.whoami())
        async with pool.spawn() as conn2:
            print(await conn2.whoami())
    async with pool.spawn() as conn1:
        print(await conn1.whoami())

loop = asyncio.get_event_loop()
result = loop.run_until_complete(test())

This works fine, but now I get this ugly warning

/home/remy/.local/lib/python3.8/site-packages/bonsai/asyncio/aioconnection.py:70: RuntimeWarning: coroutine 'AIOLDAPConnection._poll' was never awaited
  self.__open_coro = super().open(timeout)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

not sure what to do with this and what the side effects are.

[noirello]

There's an AIOConnectionPool for using asyncio connections:

from bonsai.asyncio import AIOConnectionPool

async with pool.spawn() as conn:
    res = await conn.whoami()
    print(res)

It's listed in the API docs, but documentation lacks of examples about using it. Every method of AIOConnectionPool is awaitable. The spawn method uses a custom async context manger wrapper, because asynccontextmanager is a new feature that's only available from 3.7.

[remy-sl]

Oh snap.. I suddenly feel very stupid that I missed it. Anyway, thanks a lot!

[TheCheeseDev]

Hello noirello,

I have tried to find the documentation referring to AIConnectionPool and I can't seem to find it. Would you be willing to point me towards documentation or give me a more in-depth example of this new functionality?

I am trying to create an AI Connection pool that will create new connections with different credentials each time.

Thank you, The Cheese Dev

[noirello]

Here's in the docs and it's inherited from ConnectionPool, where you can find the implemented methods. But it's designed to use the same credentials for every connection.

[TheCheeseDev]

Ah gotcha, do you have any advice for my desired functionality within your library?