abcsys / libem

Compound AI toolchain for fast and accurate entity matching, powered by LLMs.
https://libem.org
Apache License 2.0
19 stars 3 forks source link

feat: async model call and match (old) #32

Closed daiwaid closed 4 months ago

zenodflow commented 4 months ago

Thanks @daiwaid. The async implementation looks good to me overall. Two proposals:

zenodflow commented 4 months ago

For naming: let's name all async methods explicitly with an async_ prefix, e.g., async_match. Its sync wrapper should name as-is.

We should get an idea of performance (tput) improvement with async first, also the overhead of asyncio loop init in the sync call.

zenodflow commented 4 months ago

Note that vs. #34 (query batching, async support is needed because query batching does not work well when there is tool use (e.g., browsing) or more complex match outputs.

zenodflow commented 4 months ago

With #34 in, we should decide the necessity for async match (async model call is still well motivated, though) -- it should further improve the speed exploiting request-level parallelism.

daiwaid commented 4 months ago

Continued in #37.