generic codebase?

[DiTo97]

Feature request

It would be nice to write code for any generic LLM to use in a given chain, being able to specify its provider and parameters in a config file outside the source code.

Is there a built-in way to do so? What major roadblocks would you see in doing that?

Motivation

Most examples that I see in the docs or over the Internet tend to specify from code whether the LLM to be used in a given chain has to be from openAI, anthropic, LLaMA, etc., resulting in different codebases, while it would be great to write one single unified codebase and compare the performance of different open-source or API LLMs by simply changing one line of config.

This would be especially relevant when the specific LLM provider is not set or known from the get-go and multiple ones should be compared to find the most suitable one in terms of perfomance, efficiency and cost for any given LLM-driven application.

Your contribution

I don't have time for a PR now, but since I have been doing a similar thing for our private codebase at the laboratory, combining openAI, anthropic and LLaMA under a single unified spell without using the langchain framework, I may be interested in supporting that in the future as the framework already has lots of interesting features in terms of tooling, memory and storage.

[dosubot[bot]]

🤖

Hello @DiTo97! Great to meet you. I'm Dosu, a helpful bot here to assist you with your queries and concerns about LangChain while we wait for a human maintainer. I'm here to make your contribution journey smoother and answer any questions you might have. Let's work together to make LangChain better!

Thank you for your feature request and for sharing your motivation behind it. I understand that you're looking for a way to write code for any generic Language Model (LLM), specifying its provider and parameters in a config file, instead of hardcoding them. This would indeed be a great way to facilitate the comparison of different LLMs' performance.

The LangChain framework does provide a built-in method to write code for any generic LLM, specifying its provider and parameters in a config file, instead of hardcoding them. This is facilitated by the get_type_to_cls_dict() function in the libs/langchain/langchain/llms/__init__.py file. This function returns a dictionary where the keys are the names of the LLMs and the values are the corresponding import functions. This allows for dynamic selection and usage of LLMs based on the configuration provided, which can be useful for comparing the performance of different LLMs. You can find the relevant code in the LangChain repository.

In terms of roadblocks, one potential challenge could be ensuring that all LLMs are compatible with the unified codebase. Different LLMs may have different interfaces, requirements, or behaviors, and these differences would need to be accounted for in the unified codebase. However, the LangChain framework is designed to handle a wide variety of LLMs, so this should be manageable.

There have been similar discussions and solutions in the LangChain repository. For instance, the Langchain AutoLLM class issue discusses a similar topic. Another relevant issue is how to use a model loaded from HuggingFace transformers?, which provides an example of creating a custom LLM class. Lastly, the [Design Proposal] Standardized interface for LLM provider usage issue proposes a standardized interface for LLM provider usage in the LangChain project.

I hope this information is helpful. If you have any further questions or need more clarification, please don't hesitate to ask.

Sources

• Langchain AutoLLM class
• how to use a model loaded from HuggingFace transformers?
• [Design Proposal] Standardized interface for LLM provider usage
• libs/langchain/langchain/llms/init.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[DiTo97]

@dosu-bot

Thanks for the clarification. The module you shared is a great starting point for unifying the way LLM libraries are imported.

The autoLLM discussion that you mentioned (#9869) pointed in the same direction with a minimal, yet interesting, implementation, but it has been closed without further notice. Any reason why it was not supported?

Now that I look at more examples, e.g., conversation summary buffer, I see that the same idea would translate to memory, extending further than LLM only, making it far easier to experiment with different LLM (and memory, if needed) settings.

[dosubot[bot]]

🤖

Regarding the closure of the AutoLLM discussion (#9869), I'm afraid I couldn't find any specific reason within the repository. It's possible that the discussion was closed due to a variety of reasons such as the feature being implemented in a different way, the feature being deprioritized, or the discussion becoming inactive. Without more information, it's hard to say for sure.

As for your idea of creating a unified codebase for LangChain that allows for the use of any generic Language Model (LLM), with the provider and parameters specified in a config file, it sounds like a great idea. The get_type_to_cls_dict() function you mentioned is indeed a good starting point for this. You could extend this function to read from a config file and dynamically import the necessary modules based on the specified LLM.

Here's a rough idea of how you could implement this:

import json

def get_type_to_cls_dict() -> Dict[str, Callable[[], Type[BaseLLM]]]:
    with open('config.json') as f:
        config = json.load(f)
    llm_type = config['llm_type']
    return {
        llm_type: globals()[f'_import_{llm_type}'],
    }

In this example, the config.json file would look something like this:

{
    "llm_type": "ai21"
}

This would allow you to specify the LLM in the config file and the get_type_to_cls_dict() function would dynamically import the necessary modules based on this.

As for extending this idea to memory as well, it's definitely possible. You could follow a similar approach and have a function that reads from the config file and dynamically imports the necessary modules based on the specified memory settings. However, without more information on how memory is currently handled in LangChain, it's hard to provide a more specific implementation.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

Sources

• libs/langchain/langchain/llms/init.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[DiTo97]

Anyone from the contrib team (not you, Dosu bot) that can share insights on what's the direction for this?

I will just try to name a few from commits history: @mgoin, @efriis, @nfcampos.

After the 0.1.x release and the structuring in langchain, langchain-core and -community, many common abstract classes for model, memory, embedding and vectorstore seem to be present, according to the API docs. Therefore, the ability to configure a chain with one or more agents, each with a model and optional memory and toolkits, interacting with one or more embeddings or vectorstores, fully from a config file where pieces can be swapped in and out does not seem to be a long shot away.

[dosubot[bot]]

🤖

The get_type_to_cls_dict() function indeed provides a good starting point for creating a unified codebase that allows for the use of any generic Language Model (LLM). As you've correctly identified, this function maps string identifiers of different language models to their corresponding import functions, allowing for dynamic import of the required language model based on its string identifier.

Regarding the AutoLLM discussion (#9869), I'm afraid I couldn't find specific information on why it was closed without further notice within the repository. It's possible that the discussion was moved to a different platform or the idea was incorporated into the codebase in a different form. I recommend checking the commit history around the time the discussion was closed for any relevant changes.

As for extending this idea to memory, your observation is correct. After the 0.1.x release and the structuring in langchain, langchain-core, and -community, many common abstract classes for model, memory, embedding, and vectorstore are present. This indeed makes it feasible to configure a chain with one or more agents, each with a model and optional memory and toolkits, interacting with one or more embeddings or vectorstores, fully from a config file where pieces can be swapped in and out.

However, please note that while this is technically feasible, it might require significant changes to the existing codebase and careful consideration of various factors such as performance, compatibility, and maintainability. It's also important to ensure that the changes don't break existing functionality or introduce new bugs.

I hope this helps! If you have any further questions or need clarification on anything, feel free to ask.

Sources

• libs/langchain/langchain/llms/init.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[mgoin]

Hey @DiTo97 I'm just a contributor for the DeepSparse LLM backend and haven't worked on anything else/large-scale on LangChain - try contacting @baskaryan since he is a maintainer

[DiTo97]

Hi @baskaryan. Any ups on this?