Embeddings example with text_generation_web_ui

[ernestp]

chuck_norris_test_web_generation_textui.py seem work fine, except of the fact that it generates code that not complies (missing import json)

would be great to see an example of using embeddings from text_generation_web_ui

[paolorechia]

@ernestp exactly, code generation result quite often has errors

I'm still trying different ways of having more reliable code generation, I think just the base langchain Python REPL is a bit disappointing with local models. A couple of PoCs I tried recently to address this:

1. Running multiple prompts and installing packages in a virtualenv: https://github.com/paolorechia/code-it
2. Generating docstrings using lighter open source models in VSCode with guidance library: https://github.com/paolorechia/oasis

About your request: unfortunately, I wouldn't invest time into using text_generation_web_ui embeddings.

I'm currently leaning more towards exploring solutions that leverage the guidance library, I think it has more potential than just langchain for local models - and sadly guidance can't be fully used with text_generation_web_ui - there are features only available through Hugging Face API.

So I'm slowly migrating away from text_generation_web_ui in my private work.

[paolorechia]

Also note that there's an example using Sentence Transformers embeddings: https://github.com/paolorechia/learn-langchain#qa-with-sentence-transformer--vicuna

If you want to try the chuck norris joke with code-it, you can run this example:

https://github.com/paolorechia/learn-langchain/blob/main/langchain_app/executor_tests/chuck_norris_joke.py

[ernestp]

@paolorechia Thanks for the explanation and examples.

My current goal is to leverage langchain with local quantized models (GPTQ) which are currently not supported in langchain and HuggingFace pipeline.

It seems embeddings generation is supported in text_generation_web_ui API. I will try to use it and will create a pull request if succeeded. Or perhaps I will try to use your local server.

Regarding the guidance library, can you share links to API or manual?

[ernestp]

Or perhaps I don't need embeddings from text_generation_web_ui at all. I just tried HuggingFaceInstructEmbeddings and it seems it runs on GPU and not using a lot of VRAM. I tried to load both 4bit/WizardLM-13B-Uncensored-4bit-128g and HuggingFaceInstructEmbeddings simultaneously and they fit well in 16GB VRAM. You can close the issue.

[paolorechia]

Cool, thanks for the update. Official documentation for guidance: https://github.com/microsoft/guidance