Support for custom/local models

ansarizafar commented 1 year ago

Details

Is it possible to use custom LLM like https://github.com/EvanZhouDev/bard-ai with Llamaindex?

sweep-ai[bot] commented 1 year ago

Here's the PR! https://github.com/run-llama/LlamaIndexTS/pull/56.

⚡ Sweep Free Trial: I used GPT-4 to create this ticket. You have 5 GPT-4 tickets left. For more GPT-4 tickets, visit our payment portal.

Step 1: 🔍 Code Search

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I looked at (click to expand). If some file is missing from here, you can mention the path in the ticket description.

https://github.com/run-llama/LlamaIndexTS/blob/ec12633ae04912f391a84616c34103a48900a301/apps/docs/docs/api/classes/OpenAI.md#L75-L183 https://github.com/run-llama/LlamaIndexTS/blob/ec12633ae04912f391a84616c34103a48900a301/apps/docs/docs/api/interfaces/LLM.md#L1-L61 https://github.com/run-llama/LlamaIndexTS/blob/ec12633ae04912f391a84616c34103a48900a301/apps/docs/docs/api/classes/CondenseQuestionChatEngine.md#L6-L148 https://github.com/run-llama/LlamaIndexTS/blob/ec12633ae04912f391a84616c34103a48900a301/apps/docs/docs/api/classes/LlamaDeuce.md#L1-L214 https://github.com/run-llama/LlamaIndexTS/blob/ec12633ae04912f391a84616c34103a48900a301/apps/docs/docs/api/interfaces/ChatEngine.md#L1-L54

I also found the following external resources that might be helpful:

Summaries of links found in the content:

https://github.com/EvanZhouDev/bard-ai

The page is about a lightweight library called "bard-ai" that provides access to Google Bard. The library is available on GitHub and can be found at the following URL: https://github.com/EvanZhouDev/bard-ai. The library is described as being free and easy to use, with minimal setup required. It is built to provide AI capabilities through Google Bard. The page also mentions that bard-ai is smaller and faster compared to another JavaScript Bard API called googlebard. The library has features such as tree-shakeability, small size, fast performance, and a straightforward API. The documentation for bard-ai can be found at bard-ai-docs.vercel.app. The page includes code snippets demonstrating how to use bard-ai and how it compares to googlebard in terms of performance.

Step 2: 🧐 Snippet Analysis

From looking through the relevant snippets, I decided to make the following modifications:

File Path Proposed Changes

packages/core/src/llm/LLM.ts Add a new class that extends the LLM interface. This class should implement the chat and complete methods of the LLM interface. The implementation of these methods should be adapted to work with the local LLM. For example, if we're adding support for the bard-ai model, the chat method could call the generate method of the bard-ai model, and the complete method could call the complete method of the bard-ai model.

packages/core/src/ChatEngine.ts Modify the code where the LLM is instantiated. Add a condition to check if a local LLM is being used. If a local LLM is being used, instantiate the new class that we created instead of the existing LLM class.

File Path	Proposed Changes
`packages/core/src/llm/LLM.ts`	Add a new class that extends the LLM interface. This class should implement the `chat` and `complete` methods of the LLM interface. The implementation of these methods should be adapted to work with the local LLM. For example, if we're adding support for the bard-ai model, the `chat` method could call the `generate` method of the bard-ai model, and the `complete` method could call the `complete` method of the bard-ai model.
`packages/core/src/ChatEngine.ts`	Modify the code where the LLM is instantiated. Add a condition to check if a local LLM is being used. If a local LLM is being used, instantiate the new class that we created instead of the existing LLM class.

Step 3: 📝 Planning

I have created a plan for writing the pull request. I am now working my plan and coding the required changes to address this issue. Here is the planned pull request:

Support for custom/local models sweep/feature/custom-local-models

Description

This PR adds support for custom or local Language Learning Models (LLMs) in the LlamaIndexTS framework. It addresses issue #50 by extending the existing LLM interface and modifying the code where the LLM is instantiated.

Summary of Changes

Added a new class LocalLLM that extends the LLM interface to support local LLMs.

Implemented the chat and complete methods in the LocalLLM class to work with the specific local LLM.

Modified the code in ChatEngine.ts to instantiate the LocalLLM class when a local LLM is being used.

These changes allow users to use custom or local LLMs, such as the bard-ai model mentioned in the issue, with minimal changes to the existing codebase.

Please review and provide feedback. Thank you!

Step 4: ⌨️ Coding

I have finished coding the issue. I am now reviewing it for completeness.

Step 5: 🔁 Code Review

Success! 🚀

I'm a bot that handles simple bugs and feature requests but I might make mistakes. Please be kind! ^{Join Our Discord}

yisding commented 1 year ago

@ansarizafar Yes this should be doable and is on our roadmap. Currently probably the easiest thing to do would be to extend the LLM interface to another class to support the local LLM.

export interface LLM {
  /**
   * Get a chat response from the LLM
   * @param messages
   */
  chat(messages: ChatMessage[], parentEvent?: Event): Promise<ChatResponse>;

  /**
   * Get a prompt completion from the LLM
   * @param prompt the prompt to complete
   */
  complete(prompt: string, parentEvent?: Event): Promise<CompletionResponse>;
}

Would you be interested in giving it a try?

kevinlu1248 commented 1 year ago

Sweep: try again with the additional context from @yisding

joaocarloscardoso commented 1 year ago

Do you think you can implement local models following the logic described in the tweet about the new version of LlamaIndex? Thanks

yisding commented 1 year ago

Will be in the next version.

richardtape commented 10 months ago

Hi @yisding

First, thank you so much for contributing this. It's remarkable work.

I noticed in the closed PR #56 you mention that you want to support local models in a different way and that in this issue you were hoping it would be part of the next version. Is this something that is still on the radar? Or is it implemented (and if so, can you point me where I could find it?)

Again, many thanks!

EmanuelCampos commented 7 months ago

Hi @yisding

First, thank you so much for contributing this. It's remarkable work.

I noticed in the closed PR #56 you mention that you want to support local models in a different way and that in this issue you were hoping it would be part of the next version. Is this something that is still on the radar? Or is it implemented (and if so, can you point me where I could find it?)

Again, many thanks!

Currently we have ollama that support local embeddings

EmanuelCampos commented 7 months ago

Closing due inactivity and for cleaning

Feel free to open again

run-llama / LlamaIndexTS