Custom LLM and Embedding

[usmanxia]

Hi! Can we use custom LLM and Embedding models with it?

Thanx

[uripeled2]

I recently published a package llm-client that can be very helpful in enabling the support to run other LLM models, including OpenAI, Google, AI21, HuggingfaceHub, Aleph Alpha, Anthropic, Local models with transformers.

[limcheekin]

Thanks for sharing the codes that have very user-friendly simplified APIs.

First of all, why created the ChromaDB class when LangChain have Chroma class? You can find more information at https://python.langchain.com/docs/modules/data_connection/vectorstores/integrations/chroma

By the way, I forked the repo and the following points are written near the top of the readme: https://github.com/limcheekin/embedchain

• This is the fork of Embedchain to run with OpenAI API compatible llama-cpp-python Web Server. Hence, you can run the Embedchain with any LLMs supported by llama-cpp-python package.
• The notebook embedchain.ipynb is created to quick test the integrations is working fine. It is tested with orca-mini-7b.ggmlv3.q4_0.
• Use of HuggingFaceEmbeddings(model_name="intfloat/e5-large-v2") instead of OpenAIEmbeddings as the embedding endpoint of the llama-cpp-python Web Server is too slow and using too much compute resources to be usable. Hence, you need to install additional dependency, the sentence_transformers package in order to run the notebook successfully.

You can review all my changes in a single commit. I'm happy to work out a proper pull request if you're interested to merge my changes.

Thanks.

[taranjeet]

• Can the LLMs be replaced with Open commercial LLMs like Dolly in place of OpenAI? (from Ranganathan Rajkumar on linkedin)

linkedin comment link

[limcheekin]

Yes, you can replace the OpenAI to any open source commercial friendly LLMs listed at https://github.com/eugeneyan/open-llms, including Dolly.

Besides that, you need to replace the text embedding model of OpenAI such as text-embedding-ada-002 to the open source alternatives listed at https://huggingface.co/spaces/mteb/leaderboard.

That's what I did for the chatbot I'm working on currently.

The challenges is find the right models that fulfilled with your use cases and requirements with acceptable quality for given constraints in resources such as computes, memory and storage.

(I replied to the linkedin link too)

[limcheekin]

For you information, I created a project at https://github.com/limcheekin/open-text-embeddings.

The following is short description of the project extracted from the readme:

The goal of this project is to create an OpenAI API-compatible version of the embeddings endpoint, which serves open source models such as intfloat/e5-large-v2, sentence-transformers/all-MiniLM-L6-v2, sentence-transformers/all-mpnet-base-v2, and other sentence-transformers models supported by the LangChain's HuggingFaceEmbeddings class.

Please see the readme on how to use it in your project. By the way, I just added support for https://tfhub.dev/google/universal-sentence-encoder-large/5, please checkout the universal_sentence_encoder branch of the repo.

[funkytaco]

I'm trying https://github.com/oobabooga/text-generation-webui/ as a backend since it supports LLaMa-2 and exposes it at http://0.0.0.0:5001/v1

Now how do I point embedchain at it?

I only see in README to do:

import os

  from embedchain import Llama2App

  os.environ['REPLICATE_API_TOKEN'] = "REPLICATE API TOKEN"

  zuck_bot = Llama2App()

  # Embed your data
  zuck_bot.add("youtube_video", "https://www.youtube.com/watch?v=Ff4fRgnuFgQ")
  zuck_bot.add("web_page", "https://en.wikipedia.org/wiki/Mark_Zuckerberg")

[deshraj]

We now support most of the popular llm providers so will close this issue now.