Open akashAD98 opened 10 months ago
i tried mistral & llama7b from ctransofrmer & getting this issue,is there any way to add support for this? how can we implement it with websocket?
streaming_llm = CTransformers(model='TheBloke/Mistral-7B-v0.1-GGUF', model_file='mistral-7b-v0.1.Q4_K_M.gguf',model_type="mistral") streaming_llm = CTransformers(model="llama-2-7b.ggmlv3.q5_0.bin",model_type="llama", config={'max_new_tokens':128,'temperature':0.01})
can we use TGI here for doing this ? by passing the url link will it support?
Mistral 7B rulez
Imho it's the langchain that is outdated on this project that's causing an issue
Did you find a workaround or a different solution? I too am in need to proxy SSE over to Websocket in my app..
i tried mistral & llama7b from ctransofrmer & getting this issue,is there any way to add support for this? how can we implement it with websocket?
can we use TGI here for doing this ? by passing the url link will it support?