noirello / bonsai

Simple Python 3 module for LDAP, using libldap2 and winldap C libraries.
MIT License
116 stars 32 forks source link

Question: what is the recommended way to do a concurrent search? #48

Closed programus closed 3 years ago

programus commented 3 years ago

I found bonsai recently and I am trying to change the LDAP searching part in my project from ldap3 to bonsai since ldap3 is too slow especially establishing the first connection.

I need to search many data at the same time, so I wonder what is the recommended way to do this concurrent by using bonsai?

I worked out a coroutine way and a thread way:

# use coroutine

async def search(pool, mail):
    print(f'start search {mail}...')
    con = await pool.get()
    result = await con.search(
        BASE, bonsai.LDAPSearchScope.SUBTREE,
        f'(mail={mail})', attrlist=['uid']
    )
    await pool.put(con)
    print(f'end search {mail}!')
    return result

async def main():
    ml = ['one@mail.com', 'two@address.com', 'three@something.com']
    client = bonsai.LDAPClient(LDAP_URL)
    pool = AIOConnectionPool(client, minconn=1, maxconn=5)
    await pool.open()
    rl = await asyncio.gather(*[search(pool, m) for m in ml])
    return rl

loop = asyncio.get_event_loop()
result = loop.run_until_complete(main())
# use thread

client = bonsai.LDAPClient(LDAP_URL)
pool = bonsai.pool.ThreadedConnectionPool(client, 1, 5)
pool.open()

def search(mail):
    print(f'start search {mail}...')
    con = pool.get()
    result = con.search(
        BASE, bonsai.LDAPSearchScope.SUBTREE,
        f'(mail={mail})', attrlist=['uid']
    )
    pool.put(con)
    print(f'end search {mail}!')
    return result

with multiprocessing.pool.ThreadPool(5) as p:
     result = p.map(search, ml)

but I am not sure which one is better and whether there is a better way than both these two.

noirello commented 3 years ago

There's no recommended way. I think it depends on the situation which one is preferable.

But running multiple LDAP requests concurrently shouldn't be different than SQL or HTTP requests, if you're interested in the pros and cons of async IO and threaded solutions.