Closed ernestp closed 1 year ago
@ernestp exactly, code generation result quite often has errors
I'm still trying different ways of having more reliable code generation, I think just the base langchain Python REPL is a bit disappointing with local models. A couple of PoCs I tried recently to address this:
About your request: unfortunately, I wouldn't invest time into using text_generation_web_ui
embeddings.
I'm currently leaning more towards exploring solutions that leverage the guidance library, I think it has more potential than just langchain for local models - and sadly guidance can't be fully used with text_generation_web_ui
- there are features only available through Hugging Face API.
So I'm slowly migrating away from text_generation_web_ui
in my private work.
Also note that there's an example using Sentence Transformers embeddings: https://github.com/paolorechia/learn-langchain#qa-with-sentence-transformer--vicuna
If you want to try the chuck norris joke with code-it
, you can run this example:
@paolorechia Thanks for the explanation and examples.
My current goal is to leverage langchain with local quantized models (GPTQ) which are currently not supported in langchain and HuggingFace pipeline.
It seems embeddings generation is supported in text_generation_web_ui
API. I will try to use it and will create a pull request if succeeded. Or perhaps I will try to use your local server.
Regarding the guidance library, can you share links to API or manual?
Or perhaps I don't need embeddings from text_generation_web_ui
at all. I just tried HuggingFaceInstructEmbeddings
and it seems it runs on GPU and not using a lot of VRAM. I tried to load both 4bit/WizardLM-13B-Uncensored-4bit-128g
and HuggingFaceInstructEmbeddings
simultaneously and they fit well in 16GB VRAM. You can close the issue.
Cool, thanks for the update. Official documentation for guidance: https://github.com/microsoft/guidance
chuck_norris_test_web_generation_textui.py seem work fine, except of the fact that it generates code that not complies (missing import json)
would be great to see an example of using
embeddings
from text_generation_web_ui