multi_retrieval_qa.py hardwires ChatOpenAI() at the _default_chain

[vecorro]

System Info

System: langchain 0.0.321 Python 3.10

Hi,

I get an error related to a missing OpenAI token when trying to reuse the code to dynamically select from multiple retrievers at the MultiRetrievalQAChain initialization step,

multi_retriever_chain = MultiRetrievalQAChain.from_retrievers(
    llm=llm, # Llama2 served by vLLM via VLLMOpenAI
    retriever_infos=retrievers_info,
    verbose=True)

I get an error because I'm not using OpenAI LLMs and the LangChain code (multi_retrieval_qa.py) hard wires ChatOpenAI() as the LLM for the _default_chain.

_default_chain = ConversationChain(
                llm=ChatOpenAI(), prompt=prompt, input_key="query", output_key="result"
            )

I think you need to assign the llm variable to the llm provided when initializing the class.

Who can help?

No response

Information

• [X] The official example notebooks/scripts
• [ ] My own modified scripts

Related Components

• [X] LLMs/Chat Models
• [ ] Embedding Models
• [ ] Prompts / Prompt Templates / Prompt Selectors
• [ ] Output Parsers
• [ ] Document Loaders
• [ ] Vector Stores / Retrievers
• [ ] Memory
• [ ] Agents / Agent Executors
• [ ] Tools / Toolkits
• [ ] Chains
• [ ] Callbacks/Tracing
• [ ] Async

Reproduction

Including the Python code and the PDF files that get loaded by the retrievers.

#%% 
import torch
from langchain.llms import VLLMOpenAI
from langchain.document_loaders import PyPDFLoader

# Import for retrieval-augmented generation RAG
from langchain import hub
from langchain.chains import RetrievalQA, MultiRetrievalQAChain
from langchain.vectorstores import Chroma
from langchain.text_splitter import SentenceTransformersTokenTextSplitter
from langchain.embeddings.sentence_transformer import SentenceTransformerEmbeddings
#%% 
# URL for the vLLM service
INFERENCE_SRV_URL = "http://localhost:8000/v1"

def setup_chat_llm(vllm_url, max_tokens=512, temperature=0.2):
    """ 
    Intializes the vLLM service object. 

    :param vllm_url: vLLM service URL 
    :param max_tokens: Max number of tokens to get generated by the LLM 
    :param temperature: Temperature of the generation process 
    :return: The vLLM service object 
    """
    chat = VLLMOpenAI(
        openai_api_key="EMPTY",
        openai_api_base=vllm_url,
        model_name="meta-llama/Llama-2-7b-chat-hf",
        temperature=temperature,
        max_tokens=max_tokens,
    )
    return chat
#%% 
# Initialize LLM service
llm = setup_chat_llm(vllm_url=INFERENCE_SRV_URL)
#%% 
%%time
# Set up the embedding encoder (Sentence Transformers) and vector store 
model_name = "all-mpnet-base-v2"
model_kwargs = {'device': 'cuda' if torch.cuda.is_available() else 'cpu'}
encode_kwargs = {'normalize_embeddings': False}
embeddings = SentenceTransformerEmbeddings(
    model_name=model_name,
    model_kwargs=model_kwargs,
    encode_kwargs=encode_kwargs
)

# Set up the document splitter 
text_splitter = SentenceTransformersTokenTextSplitter(chunk_size=500, chunk_overlap=0)

# Load PDF documents
loader = PyPDFLoader(file_path="../data/AI_RMF_Playbook.pdf")
rmf_doc = loader.load()
rmf_splits = text_splitter.split_documents(rmf_doc)
rmf_retriever = Chroma.from_documents(documents=rmf_splits, embedding=embeddings)

loader = PyPDFLoader(file_path="../data/OWASP-Top-10-for-LLM-Applications-v101.pdf")
owasp_doc = loader.load()
owasp_splits = text_splitter.split_documents(owasp_doc)
owasp_retriever = Chroma.from_documents(documents=owasp_splits, embedding=embeddings)

loader = PyPDFLoader(file_path="../data/Threat Modeling LLM Applications - AI Village.pdf")
ai_village_doc = loader.load()
ai_village_splits = text_splitter.split_documents(ai_village_doc)
ai_village_retriever = Chroma.from_documents(documents=ai_village_splits, embedding=embeddings)
#%% 
retrievers_info = [
    {
        "name": "NIST AI Risk Management Framework",
        "description": "Guidelines for organizations and people to manage risks associated with the use of AI ",
        "retriever": rmf_retriever.as_retriever()
    },
    {
        "name": "OWASP Top 10 for LLM Applications",
        "description": "Provides practical, actionable, and concise security guidance to navigate the complex and evolving terrain of LLM security",
        "retriever": owasp_retriever.as_retriever()
    },
    {
        "name": "Threat Modeling LLM Applications",
        "description": "A high-level example from Gavin Klondike on how to build a threat model for LLM applications",
        "retriever": ai_village_retriever.as_retriever()
    }
]
#%% 
# Import default LLama RAG prompt
prompt = hub.pull("rlm/rag-prompt-llama")
print(prompt.dict()['messages'][0]['prompt']['template'])
#%% 
multi_retriever_chain = MultiRetrievalQAChain.from_retrievers(
    llm=llm,
    retriever_infos=retrievers_info,
    #default_retriever=owasp_retriever.as_retriever(),
    #default_prompt=prompt,
    #chain_type_kwargs={"prompt": prompt},
    verbose=True)
#%% 
question = "What is prompt injection?"
result = multi_retriever

AI_RMF_Playbook.pdf OWASP-Top-10-for-LLM-Applications-v101.pdf Threat Modeling LLM Applications - AI Village.pdf

Expected behavior

I was expecting to get results equivalent to what is shown at LangChain's documentation dynamically select from multiple retrievers

[dosubot[bot]]

🤖

Hello,

Thank you for bringing this to our attention. You're correct that the MultiRetrievalQAChain class in multi_retrieval_qa.py defaults to using ChatOpenAI() as the LLM for the _default_chain when no default_chain or default_retriever is provided. This is intended as a fallback mechanism, but it can cause issues if you're trying to use a different LLM that isn't compatible with ChatOpenAI().

To resolve this, you can provide a default_chain or default_retriever when calling the from_retrievers method. This will override the _default_chain and allow you to use your own LLM. Here's an example of how you can do this:

default_chain = ConversationChain(
    llm=llm,  # Your own LLM
    prompt=prompt,  # Your own prompt
    input_key="query",
    output_key="result"
)

multi_retriever_chain = MultiRetrievalQAChain.from_retrievers(
    llm=llm,
    retriever_infos=retrievers_info,
    default_chain=default_chain,  # Pass your default_chain here
    verbose=True
)

This issue has been discussed in a similar context in the LangChain repository. You can refer to the issue MultiRetrievalQAChain requires ChatModel... but should it? for more details.

Please try this solution and let us know if it resolves your issue. If you're still encountering problems, please provide more details about the error message you're receiving and we'll be happy to assist further.

Best regards, Dosu

Sources

• MultiRetrievalQAChain requires ChatModel... but should it?
• Issue: Dynamically select from multiple retrievers
• libs/langchain/langchain/chains/router/multi_retrieval_qa.py
• libs/langchain/langchain/llms/openai.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[vecorro]

@dosu-bot

Thanks

Your answer helped me run the script without errors, however I got an unintended result as well as some warnings that might indicate I'm not using the MultiRetrievalQAChain properly.

Here the new code

#%% 
import torch
import langchain
langchain.debug = True

from langchain.llms import VLLMOpenAI
from langchain.document_loaders import PyPDFLoader
from langchain.prompts import PromptTemplate   

# Import for retrieval-augmented generation RAG
from langchain import hub
from langchain.chains import ConversationChain, MultiRetrievalQAChain
from langchain.vectorstores import Chroma
from langchain.text_splitter import SentenceTransformersTokenTextSplitter
from langchain.embeddings.sentence_transformer import SentenceTransformerEmbeddings
#%% 
# URL for the vLLM service
INFERENCE_SRV_URL = "http://localhost:8000/v1"

def setup_chat_llm(vllm_url, max_tokens=512, temperature=0.2):
    """ 
    Intializes the vLLM service object. 

    :param vllm_url: vLLM service URL 
    :param max_tokens: Max number of tokens to get generated by the LLM 
    :param temperature: Temperature of the generation process 
    :return: The vLLM service object 
    """
    chat = VLLMOpenAI(
        model_name="meta-llama/Llama-2-7b-chat-hf",
        openai_api_key="EMPTY",
        openai_api_base=vllm_url,
        temperature=temperature,
        max_tokens=max_tokens,
    )
    return chat
#%% 
# Initialize LLM service
llm = setup_chat_llm(vllm_url=INFERENCE_SRV_URL)
#%% 
%%time
# Set up the embedding encoder (Sentence Transformers) and vector store 
model_name = "all-mpnet-base-v2"
model_kwargs = {'device': 'cuda' if torch.cuda.is_available() else 'cpu'}
encode_kwargs = {'normalize_embeddings': False}
embeddings = SentenceTransformerEmbeddings(
    model_name=model_name,
    model_kwargs=model_kwargs,
    encode_kwargs=encode_kwargs
)
# Set up the document splitter 
text_splitter = SentenceTransformersTokenTextSplitter(chunk_size=500, chunk_overlap=0)

# Load PDF documents
loader = PyPDFLoader(file_path="../data/AI_RMF_Playbook.pdf")
rmf_doc = loader.load()
rmf_splits = text_splitter.split_documents(rmf_doc)
rmf_retriever = Chroma.from_documents(documents=rmf_splits, embedding=embeddings)

loader = PyPDFLoader(file_path="../data/OWASP-Top-10-for-LLM-Applications-v101.pdf")
owasp_doc = loader.load()
owasp_splits = text_splitter.split_documents(owasp_doc)
owasp_retriever = Chroma.from_documents(documents=owasp_splits, embedding=embeddings)

loader = PyPDFLoader(file_path="../data/Threat Modeling LLM Applications - AI Village.pdf")
ai_village_doc = loader.load()
ai_village_splits = text_splitter.split_documents(ai_village_doc)
ai_village_retriever = Chroma.from_documents(documents=ai_village_splits, embedding=embeddings)
#%% 
retrievers_info = [
    {
        "name": "NIST AI Risk Management Framework",
        "description": "Guidelines for organizations and people to manage risks associated with the use of AI ",
        "retriever": rmf_retriever.as_retriever()
    },
    {
        "name": "OWASP Top 10 for LLM Applications",
        "description": "Provides practical, actionable, and concise security guidance to navigate the complex and evolving terrain of LLM security",
        "retriever": owasp_retriever.as_retriever()
    },
    {
        "name": "Threat Modeling LLM Applications",
        "description": "A high-level example from Gavin Klondike on how to build a threat model for LLM applications",
        "retriever": ai_village_retriever.as_retriever()
    }
]
#%% 
prompt_template = (
""" [INST]<<SYS>> You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.<</SYS>>  
Question: {query}  
Context: {history}  
Answer: [/INST] 
""")
prompt = PromptTemplate(template=prompt_template, input_variables=['history', 'query'])
#%% 
default_chain = ConversationChain(
    llm=llm,  # Your own LLM
    prompt=prompt,  # Your own prompt
    input_key="query",
    output_key="result",
    verbose=True,
)

multi_retriever_chain = MultiRetrievalQAChain.from_retrievers(
    llm=llm,
    retriever_infos=retrievers_info,
    default_chain=default_chain,  # Pass your default_chain here
    verbose=True
)
#%% 
question = "What is prompt injection?"
result = multi_retriever_chain.run(question)
#%% 
result

The first possible issue is the following warning:

llm.py:280: UserWarning: The predict_and_parse method is deprecated, instead pass an output parser directly to LLMChain.

I have not explicitly defined any LLMChain. From the code examples you provided, that was not needed, unless my case is different because I'm using LLama2 instead of OpenAI LLMs.

The second issue is that the chain is answering a question I never asked. From the code you can see I'm running the MultiRetrievalQAChain chain with question (query) "What is prompt injection?", the chain seems to respond to a question it made up: "What are the potential risks associated with the use of AI in the organization?"

Here the debug messages:

[chain/start] [1:chain:MultiRetrievalQAChain] Entering Chain run with input:
{
  "input": "What is prompt injection?"
}
[chain/start] [1:chain:MultiRetrievalQAChain > 2:chain:LLMRouterChain] Entering Chain run with input:
{
  "input": "What is prompt injection?"
}
[chain/start] [1:chain:MultiRetrievalQAChain > 2:chain:LLMRouterChain > 3:chain:LLMChain] Entering Chain run with input:
{
  "input": "What is prompt injection?"
}
[llm/start] [1:chain:MultiRetrievalQAChain > 2:chain:LLMRouterChain > 3:chain:LLMChain > 4:llm:VLLMOpenAI] Entering LLM run with input:
{
  "prompts": [
    "Given a query to a question answering system select the system best suited for the input. You will be given the names of the available systems and a description of what questions the system is best suited for. You may also revise the original input if you think that revising it will ultimately lead to a better response.\n\n<< FORMATTING >>\nReturn a markdown code snippet with a JSON object formatted to look like:\n```json\n{\n    \"destination\": string \\ name of the question answering system to use or \"DEFAULT\"\n    \"next_inputs\": string \\ a potentially modified version of the original input\n}\n```\n\nREMEMBER: \"destination\" MUST be one of the candidate prompt names specified below OR it can be \"DEFAULT\" if the input is not well suited for any of the candidate prompts.\nREMEMBER: \"next_inputs\" can just be the original input if you don't think any modifications are needed.\n\n<< CANDIDATE PROMPTS >>\nNIST AI Risk Management Framework: Guidelines for organizations and people to manage risks associated with the use of AI \nOWASP Top 10 for LLM Applications: Provides practical, actionable, and concise security guidance to navigate the complex and evolving terrain of LLM security\nThreat Modeling LLM Applications: A high-level example from Gavin Klondike on how to build a threat model for LLM applications\n\n<< INPUT >>\nWhat is prompt injection?\n\n<< OUTPUT >>"
  ]
}

/home/vmuser/miniconda3/envs/llm-env2/lib/python3.10/site-packages/langchain/chains/llm.py:280: UserWarning: The predict_and_parse method is deprecated, instead pass an output parser directly to LLMChain.
  warnings.warn(

[llm/end] [1:chain:MultiRetrievalQAChain > 2:chain:LLMRouterChain > 3:chain:LLMChain > 4:llm:VLLMOpenAI] [7.02s] Exiting LLM run with output:
{
  "generations": [
    [
      {
        "text": "\n{\n    \"destination\": \"NIST AI Risk Management Framework\",\n    \"next_inputs\": \"What are the potential risks associated with the use of AI in the organization?\"\n}\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
        "generation_info": {
          "finish_reason": "length",
          "logprobs": null
        }
      }
    ]
  ],
  "llm_output": {
    "token_usage": {
      "prompt_tokens": 342,
      "completion_tokens": 512,
      "total_tokens": 854
    },
    "model_name": "meta-llama/Llama-2-7b-chat-hf"
  },
  "run": null
}
[chain/end] [1:chain:MultiRetrievalQAChain > 2:chain:LLMRouterChain > 3:chain:LLMChain] [7.02s] Exiting Chain run with output:
{
  "text": "\n{\n    \"destination\": \"NIST AI Risk Management Framework\",\n    \"next_inputs\": \"What are the potential risks associated with the use of AI in the organization?\"\n}\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"
}
[chain/end] [1:chain:MultiRetrievalQAChain > 2:chain:LLMRouterChain] [7.02s] Exiting Chain run with output:
{
  "destination": "NIST AI Risk Management Framework",
  "next_inputs": {
    "query": "What are the potential risks associated with the use of AI in the organization?"
  }
}
[chain/start] [1:chain:MultiRetrievalQAChain > 5:chain:RetrievalQA] Entering Chain run with input:
{
  "query": "What are the potential risks associated with the use of AI in the organization?"
}
[chain/start] [1:chain:MultiRetrievalQAChain > 5:chain:RetrievalQA > 7:chain:StuffDocumentsChain] Entering Chain run with input:
[inputs]
[chain/start] [1:chain:MultiRetrievalQAChain > 5:chain:RetrievalQA > 7:chain:StuffDocumentsChain > 8:chain:LLMChain] Entering Chain run with input:
{
  "question": "What are the potential risks associated with the use of AI in the organization?",
  "context": "Context:\nmanage 1. 2 treatment of documented ai risks is prioritized based on impact, likelihood, or available resources or methods. about risk refers to the composite measure of an event ’ s probability of occurring and the magnitude ( or degree ) of the consequences of the corresponding events. the impacts, or consequences, of ai systems can be positive, negative, or both and can result in opportunities or risks. organizational risk tolerances are often informed by several internal and external factors, including existing industry practices, organizational values, and legal or regulatory requirements. since risk management resources are often limited, organizations usually assign them based on risk tolerance. ai risks that are deemed more serious receive more oversight attention and risk management resources. suggested actions assign risk management resources relative to established risk tolerance. ai systems with lower risk tolerances receive greater oversight, mitigation and management resources. document ai risk tolerance determination practices and resource decisions. regularly review risk tolerances and re - calibrate, as needed, in accordance with information from ai system monitoring and assessment. transparency & documentation did your organization implement a risk management system to address risks involved in deploying the identified ai solution ( e. g., personnel risk or changes to commercial objectives )? what assessments has the entity conducted on data security and privacy impacts associated with the ai system? does your organization have an existing governance structure that can be leveraged to oversee the organization ’ s use of ai? organizations can document the following : 1 7 6\n\nContext:\nmanage 2 strategies to maximize ai benefits and minimize negative impacts are planned, prepared, implemented, and documented, and informed by input from relevant ai actors. manage 2. 1 resources required to manage ai risks are taken into account, along with viable non - ai alternative systems, approaches, or methods – to reduce the magnitude or likelihood of potential impacts. about organizational risk response may entail identifying and analyzing alternative approaches, methods, processes or systems, and balancing tradeoffs between trustworthiness characteristics and how they relate to organizational principles and societal values. analysis of these tradeoffs is informed by consulting with interdisciplinary organizational teams, independent domain experts, and engaging with individuals or community groups. these processes require sufficient resource allocation. suggested actions plan and implement risk management practices in accordance with established organizational risk tolerances. verify risk management teams are resourced to carry out functions, including : establishing processes for considering methods that are not automated ; semi - automated ; or other procedural alternatives for ai functions. enhance ai system transparency mechanisms for ai teams. enable exploration of ai system limitations by ai teams. identify, assess, and catalog past failed designs and negative impacts or outcomes to avoid known failure modes. identify resource allocation approaches for managing risks in systems : deemed high - risk, that self - update ( adaptive, online, reinforcement self - supervised learning or similar ), 1 8 2\n\nContext:\nmap 3. 2 potential costs, including non - monetary costs, which result from expected or realized ai errors or system functionality and trustworthiness - as connected to organizational risk tolerance - are examined and documented. about anticipating negative impacts of ai systems is a difficult task. negative impacts can be due to many factors, such as system non - functionality or use outside of its operational limits, and may range from minor annoyance to serious injury, financial losses, or regulatory enforcement actions. ai actors can work with a broad set of stakeholders to improve their capacity for understanding systems ’ potential impacts – and subsequently – systems ’ risks. suggested actions perform context analysis to map potential negative impacts arising from not integrating trustworthiness characteristics. when negative impacts are not direct or obvious, ai actors can engage with stakeholders external to the team that developed or deployed the ai system, and potentially impacted communities, to examine and document : who could be harmed? what could be harmed? when could harm arise? how could harm arise? identify and implement procedures for regularly evaluating the qualitative and quantitative costs of internal and external ai system failures. develop actions to prevent, detect, and / or correct potential risks and related impacts. regularly evaluate failure costs to inform go / no - go deployment decisions throughout the ai system lifecycle. 83\n\nContext:\ngovern 2. 3 executive leadership of the organization takes responsibility for decisions about risks associated with ai system development and deployment. about senior leadership and members of the c - suite in organizations that maintain an ai portfolio, should maintain awareness of ai risks, affirm the organizational appetite for such risks, and be responsible for managing those risks. accountability ensures that a specific team and individual is responsible for ai risk management efforts. some organizations grant authority and resources ( human and budgetary ) to a designated officer who ensures adequate performance of the institution ’ s ai portfolio ( e. g., predictive modeling, machine learning ). suggested actions declare risk tolerances for developing or using ai systems. support ai risk management efforts and play an active role in such efforts. integrate a risk and harm prevention mindset throughout the ai lifecycle as part of organizational culture. support competent risk management executives. delegate the power, resources, and authorization to perform risk management to each appropriate level throughout the management chain. organizational management can : organizations can establish board committees for ai risk management and oversight functions and integrate those functions within the organization ’ s broader enterprise risk management approaches. 2 3"
}
[llm/start] [1:chain:MultiRetrievalQAChain > 5:chain:RetrievalQA > 7:chain:StuffDocumentsChain > 8:chain:LLMChain > 9:llm:VLLMOpenAI] Entering LLM run with input:
{
  "prompts": [
    "Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.\n\nContext:\nmanage 1. 2 treatment of documented ai risks is prioritized based on impact, likelihood, or available resources or methods. about risk refers to the composite measure of an event ’ s probability of occurring and the magnitude ( or degree ) of the consequences of the corresponding events. the impacts, or consequences, of ai systems can be positive, negative, or both and can result in opportunities or risks. organizational risk tolerances are often informed by several internal and external factors, including existing industry practices, organizational values, and legal or regulatory requirements. since risk management resources are often limited, organizations usually assign them based on risk tolerance. ai risks that are deemed more serious receive more oversight attention and risk management resources. suggested actions assign risk management resources relative to established risk tolerance. ai systems with lower risk tolerances receive greater oversight, mitigation and management resources. document ai risk tolerance determination practices and resource decisions. regularly review risk tolerances and re - calibrate, as needed, in accordance with information from ai system monitoring and assessment. transparency & documentation did your organization implement a risk management system to address risks involved in deploying the identified ai solution ( e. g., personnel risk or changes to commercial objectives )? what assessments has the entity conducted on data security and privacy impacts associated with the ai system? does your organization have an existing governance structure that can be leveraged to oversee the organization ’ s use of ai? organizations can document the following : 1 7 6\n\nContext:\nmanage 2 strategies to maximize ai benefits and minimize negative impacts are planned, prepared, implemented, and documented, and informed by input from relevant ai actors. manage 2. 1 resources required to manage ai risks are taken into account, along with viable non - ai alternative systems, approaches, or methods – to reduce the magnitude or likelihood of potential impacts. about organizational risk response may entail identifying and analyzing alternative approaches, methods, processes or systems, and balancing tradeoffs between trustworthiness characteristics and how they relate to organizational principles and societal values. analysis of these tradeoffs is informed by consulting with interdisciplinary organizational teams, independent domain experts, and engaging with individuals or community groups. these processes require sufficient resource allocation. suggested actions plan and implement risk management practices in accordance with established organizational risk tolerances. verify risk management teams are resourced to carry out functions, including : establishing processes for considering methods that are not automated ; semi - automated ; or other procedural alternatives for ai functions. enhance ai system transparency mechanisms for ai teams. enable exploration of ai system limitations by ai teams. identify, assess, and catalog past failed designs and negative impacts or outcomes to avoid known failure modes. identify resource allocation approaches for managing risks in systems : deemed high - risk, that self - update ( adaptive, online, reinforcement self - supervised learning or similar ), 1 8 2\n\nContext:\nmap 3. 2 potential costs, including non - monetary costs, which result from expected or realized ai errors or system functionality and trustworthiness - as connected to organizational risk tolerance - are examined and documented. about anticipating negative impacts of ai systems is a difficult task. negative impacts can be due to many factors, such as system non - functionality or use outside of its operational limits, and may range from minor annoyance to serious injury, financial losses, or regulatory enforcement actions. ai actors can work with a broad set of stakeholders to improve their capacity for understanding systems ’ potential impacts – and subsequently – systems ’ risks. suggested actions perform context analysis to map potential negative impacts arising from not integrating trustworthiness characteristics. when negative impacts are not direct or obvious, ai actors can engage with stakeholders external to the team that developed or deployed the ai system, and potentially impacted communities, to examine and document : who could be harmed? what could be harmed? when could harm arise? how could harm arise? identify and implement procedures for regularly evaluating the qualitative and quantitative costs of internal and external ai system failures. develop actions to prevent, detect, and / or correct potential risks and related impacts. regularly evaluate failure costs to inform go / no - go deployment decisions throughout the ai system lifecycle. 83\n\nContext:\ngovern 2. 3 executive leadership of the organization takes responsibility for decisions about risks associated with ai system development and deployment. about senior leadership and members of the c - suite in organizations that maintain an ai portfolio, should maintain awareness of ai risks, affirm the organizational appetite for such risks, and be responsible for managing those risks. accountability ensures that a specific team and individual is responsible for ai risk management efforts. some organizations grant authority and resources ( human and budgetary ) to a designated officer who ensures adequate performance of the institution ’ s ai portfolio ( e. g., predictive modeling, machine learning ). suggested actions declare risk tolerances for developing or using ai systems. support ai risk management efforts and play an active role in such efforts. integrate a risk and harm prevention mindset throughout the ai lifecycle as part of organizational culture. support competent risk management executives. delegate the power, resources, and authorization to perform risk management to each appropriate level throughout the management chain. organizational management can : organizations can establish board committees for ai risk management and oversight functions and integrate those functions within the organization ’ s broader enterprise risk management approaches. 2 3\n\nQuestion: What are the potential risks associated with the use of AI in the organization?\nHelpful Answer:"
  ]
}
[llm/end] [1:chain:MultiRetrievalQAChain > 5:chain:RetrievalQA > 7:chain:StuffDocumentsChain > 8:chain:LLMChain > 9:llm:VLLMOpenAI] [5.56s] Exiting LLM run with output:
{
  "generations": [
    [
      {
        "text": " Based on the provided context, the potential risks associated with the use of AI in the organization include:\n1. Data security and privacy risks: The organization may face risks related to data security and privacy due to the use of AI systems.\n2. Impact on personnel: The use of AI may lead to changes in personnel roles and responsibilities, which could result in negative impacts on employees.\n3. Changes to commercial objectives: AI may lead to changes in commercial objectives, which could result in negative impacts on the organization.\n4. Liability and regulatory risks: The organization may face liability and regulatory risks due to the use of AI systems.\n5. Ethical risks: The use of AI may raise ethical concerns, such as bias, discrimination, and transparency, which could result in negative impacts on the organization.\n6. Reputation risks: The organization may face reputation risks due to the use of AI systems, such as negative media coverage or public perception of the technology.\n7. Technical risks: The organization may face technical risks related to the development and deployment of AI systems, such as system failures or errors.\n8. Integration risks: The organization may face integration risks due to the use of AI systems, such as compatibility issues with existing systems or processes.\n9. Dependence risks: The organization may face dependence risks due to the use of AI systems, such as over-reliance on the technology.\n10. Unknown risks: There may be other risks associated with the use of AI in the organization that are not yet known or anticipated.\n\n\n\n\n\n",
        "generation_info": {
          "finish_reason": "stop",
          "logprobs": null
        }
      }
    ]
  ],
  "llm_output": {
    "token_usage": {
      "prompt_tokens": 1293,
      "completion_tokens": 377,
      "total_tokens": 1670
    },
    "model_name": "meta-llama/Llama-2-7b-chat-hf"
  },
  "run": null
}
[chain/end] [1:chain:MultiRetrievalQAChain > 5:chain:RetrievalQA > 7:chain:StuffDocumentsChain > 8:chain:LLMChain] [5.56s] Exiting Chain run with output:
{
  "text": " Based on the provided context, the potential risks associated with the use of AI in the organization include:\n1. Data security and privacy risks: The organization may face risks related to data security and privacy due to the use of AI systems.\n2. Impact on personnel: The use of AI may lead to changes in personnel roles and responsibilities, which could result in negative impacts on employees.\n3. Changes to commercial objectives: AI may lead to changes in commercial objectives, which could result in negative impacts on the organization.\n4. Liability and regulatory risks: The organization may face liability and regulatory risks due to the use of AI systems.\n5. Ethical risks: The use of AI may raise ethical concerns, such as bias, discrimination, and transparency, which could result in negative impacts on the organization.\n6. Reputation risks: The organization may face reputation risks due to the use of AI systems, such as negative media coverage or public perception of the technology.\n7. Technical risks: The organization may face technical risks related to the development and deployment of AI systems, such as system failures or errors.\n8. Integration risks: The organization may face integration risks due to the use of AI systems, such as compatibility issues with existing systems or processes.\n9. Dependence risks: The organization may face dependence risks due to the use of AI systems, such as over-reliance on the technology.\n10. Unknown risks: There may be other risks associated with the use of AI in the organization that are not yet known or anticipated.\n\n\n\n\n\n"
}
[chain/end] [1:chain:MultiRetrievalQAChain > 5:chain:RetrievalQA > 7:chain:StuffDocumentsChain] [5.56s] Exiting Chain run with output:
{
  "output_text": " Based on the provided context, the potential risks associated with the use of AI in the organization include:\n1. Data security and privacy risks: The organization may face risks related to data security and privacy due to the use of AI systems.\n2. Impact on personnel: The use of AI may lead to changes in personnel roles and responsibilities, which could result in negative impacts on employees.\n3. Changes to commercial objectives: AI may lead to changes in commercial objectives, which could result in negative impacts on the organization.\n4. Liability and regulatory risks: The organization may face liability and regulatory risks due to the use of AI systems.\n5. Ethical risks: The use of AI may raise ethical concerns, such as bias, discrimination, and transparency, which could result in negative impacts on the organization.\n6. Reputation risks: The organization may face reputation risks due to the use of AI systems, such as negative media coverage or public perception of the technology.\n7. Technical risks: The organization may face technical risks related to the development and deployment of AI systems, such as system failures or errors.\n8. Integration risks: The organization may face integration risks due to the use of AI systems, such as compatibility issues with existing systems or processes.\n9. Dependence risks: The organization may face dependence risks due to the use of AI systems, such as over-reliance on the technology.\n10. Unknown risks: There may be other risks associated with the use of AI in the organization that are not yet known or anticipated.\n\n\n\n\n\n"
}
[chain/end] [1:chain:MultiRetrievalQAChain > 5:chain:RetrievalQA] [5.58s] Exiting Chain run with output:
{
  "result": " Based on the provided context, the potential risks associated with the use of AI in the organization include:\n1. Data security and privacy risks: The organization may face risks related to data security and privacy due to the use of AI systems.\n2. Impact on personnel: The use of AI may lead to changes in personnel roles and responsibilities, which could result in negative impacts on employees.\n3. Changes to commercial objectives: AI may lead to changes in commercial objectives, which could result in negative impacts on the organization.\n4. Liability and regulatory risks: The organization may face liability and regulatory risks due to the use of AI systems.\n5. Ethical risks: The use of AI may raise ethical concerns, such as bias, discrimination, and transparency, which could result in negative impacts on the organization.\n6. Reputation risks: The organization may face reputation risks due to the use of AI systems, such as negative media coverage or public perception of the technology.\n7. Technical risks: The organization may face technical risks related to the development and deployment of AI systems, such as system failures or errors.\n8. Integration risks: The organization may face integration risks due to the use of AI systems, such as compatibility issues with existing systems or processes.\n9. Dependence risks: The organization may face dependence risks due to the use of AI systems, such as over-reliance on the technology.\n10. Unknown risks: There may be other risks associated with the use of AI in the organization that are not yet known or anticipated.\n\n\n\n\n\n"
}
[chain/end] [1:chain:MultiRetrievalQAChain] [12.61s] Exiting Chain run with output:
{
  "query": "What are the potential risks associated with the use of AI in the organization?",
  "result": " Based on the provided context, the potential risks associated with the use of AI in the organization include:\n1. Data security and privacy risks: The organization may face risks related to data security and privacy due to the use of AI systems.\n2. Impact on personnel: The use of AI may lead to changes in personnel roles and responsibilities, which could result in negative impacts on employees.\n3. Changes to commercial objectives: AI may lead to changes in commercial objectives, which could result in negative impacts on the organization.\n4. Liability and regulatory risks: The organization may face liability and regulatory risks due to the use of AI systems.\n5. Ethical risks: The use of AI may raise ethical concerns, such as bias, discrimination, and transparency, which could result in negative impacts on the organization.\n6. Reputation risks: The organization may face reputation risks due to the use of AI systems, such as negative media coverage or public perception of the technology.\n7. Technical risks: The organization may face technical risks related to the development and deployment of AI systems, such as system failures or errors.\n8. Integration risks: The organization may face integration risks due to the use of AI systems, such as compatibility issues with existing systems or processes.\n9. Dependence risks: The organization may face dependence risks due to the use of AI systems, such as over-reliance on the technology.\n10. Unknown risks: There may be other risks associated with the use of AI in the organization that are not yet known or anticipated.\n\n\n\n\n\n"
}

[dosubot[bot]]

🤖

Hello,

Thank you for reaching out with your questions. Let's address them one by one.

1. The UserWarning about the deprecated predict_and_parse method in LLMChain is likely due to the use of an outdated method in your custom LLM. The predict_and_parse method might have been replaced or removed in a newer version of the LangChain framework. To resolve this, you should update your custom LLM to use the current methods provided by the LangChain framework. Here is the current version of the predict_and_parse method in the LLMChain class:

def predict_and_parse(
    self, callbacks: Callbacks = None, **kwargs: Any
) -> Union[str, List[str], Dict[str, Any]]:
    """Call predict and then parse the results."""
    warnings.warn(
        "The predict_and_parse method is deprecated, "
        "instead pass an output parser directly to LLMChain."
    )
    result = self.predict(callbacks=callbacks, **kwargs)
    if self.prompt.output_parser is not None:
        return self.prompt.output_parser.parse(result)
    else:
        return result

As the warning message indicates, this method is deprecated. The recommended alternative is to pass an output parser directly to the LLMChain. This can be done when initializing the LLMChain instance by setting the output_parser attribute to an instance of a class that inherits from BaseLLMOutputParser. The output_parser will then be used to parse the output of the language model.

For example, if you have a custom output parser named MyOutputParser, you can use it like this:

llm_chain = LLMChain(llm=my_llm, prompt=my_prompt, output_parser=MyOutputParser())

Then, you can simply use the predict method to get the parsed output:

parsed_output = llm_chain.predict(adjective="funny")

This will return the output of the language model parsed by MyOutputParser.

2. The MultiRetrievalQAChain class in the LangChain framework is a multi-route chain that uses an LLM router chain to choose amongst retrieval QA chains. If the MultiRetrievalQAChain class is answering an unasked question about AI risks instead of the intended query "What is prompt injection?", it could be due to the self-verification process. The LLMChain class uses a series of prompts to create a draft answer, list assertions, check assertions, and revise the answer. If the prompts or the language model are not correctly configured, it could lead to unexpected results.

You should check the implementation of your LLMChain class, especially the prompts and the language model used. Make sure the prompts are relevant to the intended query and the language model is trained with appropriate data. In the context shared, the create_qa_with_structure_chain function is used to create the LLMChain. Ensure that the prompt argument passed to this function is correctly configured for your intended query.

I hope this helps! If you have any further questions, please don't hesitate to ask.

Best, Dosu

Sources

• libs/langchain/langchain/chains/retrieval_qa/base.py
• libs/langchain/langchain/chains/llm_checker/base.py
• libs/langchain/langchain/chains/openai_functions/qa_with_structure.py
• libs/langchain/langchain/chains/llm.py
• libs/langchain/langchain/chains/router/multi_retrieval_qa.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[vecorro]

@dosu-bot

Thanks, now I'm confused about what the llm parameter from the call ConversationChain actually means. Should it be an LLM pipeline instead of a chat LLM?

[dosubot[bot]]

🤖

Hello,

The llm parameter in the ConversationChain class from the LangChain library (version 0.0.321) represents an instance of the ChatLiteLLM class. This class is a chat model that uses the LiteLLM API for language model operations. The llm parameter is used in several functions within the ChatLiteLLM class, such as _create_retry_decorator, acompletion_with_retry, and completion_with_retry. These functions use the llm parameter to interact with the LiteLLM API, handle exceptions, and retry operations.

As for whether it should be an LLM pipeline instead of a chat LLM when using Python 3.10, the choice between an LLM pipeline and a chat LLM would depend on the specific requirements of your application, not the Python version being used.

The difference between an LLM pipeline and a chat LLM in the context of the LangChain library lies in the underlying OpenAI models they are built upon and how they process prompts. The PromptLayerOpenAI class uses OpenAI's standard LLMs, which generate responses for each prompt independently. On the other hand, the PromptLayerOpenAIChat class uses OpenAI's chat models, which generate responses in a conversational context, considering the entire conversation history.

I hope this clarifies your question. If you have any more questions or need further clarification, feel free to ask.

Best, Dosu

Sources

• libs/langchain/langchain/chat_models/litellm.py
• libs/langchain/langchain/llms/promptlayer_openai.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[douglarek]

I believe this is a complete typo; the default 'llm' here should be the function passed in.

_default_chain = ConversationChain(
    llm=llm, prompt=prompt, input_key="query", output_key="result"
)

BTW：As a rigorous project, it is not rigorous to use a bot to respond to issues.