Open AdrianLsk opened 12 months ago
I don't think the current langchain
wrapper support async calls, but it shouldn't be to hard to add, as the model.stream()
call already unlocks the GIL internally while generating tokens. But you would have to ensure that the model is never used in parallel as that would probably create some sort of memory access problems or simply crash if you offloaded your model onto a gpu.
Do you know what function needs to be implemented by langchains
LLM
class to enable async processing?
I am getting no reponses when using with langchain callback AsyncIteratorCallbackHandler?
It gives only this warning