zilliztech / GPTCache

Semantic cache for LLMs. Fully integrated with LangChain and llama_index.
https://gptcache.readthedocs.io
MIT License
7.06k stars 500 forks source link

LangChainLLMs cache_obj update #508

Closed keenborder786 closed 1 year ago

keenborder786 commented 1 year ago

If one were required to pass a custom cache_obj, it would necessitate passing the cache_obj during inference time, potentially leading to repetitive passes of the cache_obj. However, with this PR, such repetition will no longer be necessary:

from gptcache import cache
from gptcache.core import Cache
from gptcache.processor.pre import get_prompt,get_messages_last_content
from gptcache.processor.pre import get_messages_last_content
from langchain.llms import OpenAI
from gptcache.adapter.langchain_models import LangChainLLMs
import os
os.environ['OPENAI_API_KEY']="KEY"

# init gptcache
llm_cache = Cache()
llm_cache.init(
    pre_embedding_func=get_prompt,
)

# run llm with gptcache
llm = LangChainLLMs(llm=OpenAI(temperature=0) , cache_obj = llm_cache) ## passing cache_obj once
llm("Hello world") ## Not during inference
llm("How are you?") ## Not during inference

Furthermore, the current limitation of not being able to pass cache_obj during the construction of LangChainLLMs prevents you from using a custom cache_obj in other sequential chains like SQLDatabaseSequentialChain, where only the global cache object can be used. This restriction arises because there is no provision to pass a custom cache_obj when SQLDatabaseSequentialChain calls the predict method.

However, with the implementation of my update, you will have the ability to initialize LangChainLLMs with a custom cache_obj. As a result, when you pass this LangChainLLMs instance into the chain, custom cache_obj will function correctly:

from gptcache import cache
from gptcache.core import Cache
from gptcache.processor.pre import get_prompt,get_messages_last_content
from gptcache.processor.pre import get_messages_last_content
from langchain.llms import OpenAI
from gptcache.adapter.langchain_models import LangChainLLMs
from langchain_experimental.sql.base import (
    SQLDatabaseSequentialChain,
)
import os

os.environ['OPENAI_API_KEY']="KEY"
llm_cache = Cache()
llm_cache.init(
  pre_embedding_func=get_prompt,
)
# run llm with gptcache
llm = LangChainLLMs(llm=OpenAI(temperature=0) , cache_obj = llm_cache)
SQLDatabaseSequentialChain.from_llm(
chain_llm,
database=chain_db,
query_prompt=prompt,
decider_prompt=decider_prompt
) 

I hope this clarfies the use case and why this will improve the usability of the package.

sre-ci-robot commented 1 year ago

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: keenborder786 To complete the pull request process, please assign xiaofan-luan after the PR has been reviewed. You can assign the PR to them by writing /assign @xiaofan-luan in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/zilliztech/GPTCache/blob/dev/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
keenborder786 commented 1 year ago

/assign @xiaofan-luan

SimFG commented 1 year ago

@keenborder786 thanks your improvement