Custom LLM from API for QA chain

[nnngoc]

Issue you'd like to raise.

Hi,

Currently, I want to build RAG chatbot for production. I already had my LLM API and I want to create a custom LLM and then use this in RetrievalQA.from_chain_type function.

curl --location 'https:/myhost:10001/llama/api' -k \
--header 'Content-Type: application/json' \
--data-raw '{
    "inputs": "[INST] Question: Who is Albert Einstein? \n Answer: [/INST]",
    "parameters": {"max_new_tokens":100},
    "token": "abcdfejkwehr"
}

I don't know whether Langchain support this in my case.

I read about this topic on reddit: https://www.reddit.com/r/LangChain/comments/17v1rhv/integrating_llm_rest_api_into_a_langchain/ And in langchain document: https://python.langchain.com/docs/modules/model_io/llms/custom_llm

But this still does not work when I apply the custom LLM to qa_chain. Below is my code, hope for the support from you, sorry for my language, english is not my mother tongue.

from pydantic import Extra
import requests
from typing import Any, List, Mapping, Optional

from langchain.callbacks.manager import CallbackManagerForLLMRun
from langchain.llms.base import LLM

class LlamaLLM(LLM):
    llm_url = 'https:/myhost/llama/api'

    class Config:
        extra = Extra.forbid

    @property
    def _llm_type(self) -> str:
        return "Llama2 7B"

    def _call(
        self,
        prompt: str,
        stop: Optional[List[str]] = None,
        run_manager: Optional[CallbackManagerForLLMRun] = None,
        **kwargs: Any,
    ) -> str:
        if stop is not None:
            raise ValueError("stop kwargs are not permitted.")

        payload = {
            "inputs": prompt,
            "parameters": {"max_new_tokens": 100},
            "token": "abcdfejkwehr"
        }

        headers = {"Content-Type": "application/json"}

        response = requests.post(self.llm_url, json=payload, headers=headers, verify=False)
        response.raise_for_status()

        # print("API Response:", response.json())

        return response.json()['generated_text']  # get the response from the API

    @property
    def _identifying_params(self) -> Mapping[str, Any]:
        """Get the identifying parameters."""
        return {"llmUrl": self.llm_url}

llm = LlamaLLM()

#Testing
prompt = "[INST] Question: Who is Albert Einstein? \n Answer: [/INST]"
result = llm._call(prompt)
print(result)

Albert Einstein (1879-1955) was a German-born theoretical physicist who is widely regarded as one of the most influential scientists of the 20th century. He is best known for his theory of relativity, which revolutionized our understanding of space and time, and his famous equation E=mc².

# Build prompt
from langchain.prompts import PromptTemplate
template = """[INST] <<SYS>>

Answer the question base on the context below.

<</SYS>>

Context: {context}
Question: {question}
Answer:
[/INST]"""
QA_CHAIN_PROMPT = PromptTemplate(input_variables=["context", "question"],template=template,)

# Run chain
from langchain.chains import RetrievalQA

qa_chain = RetrievalQA.from_chain_type(llm,
                                       verbose=True,
                                       # retriever=vectordb.as_retriever(),
                                       retriever=custom_retriever,
                                       return_source_documents=True,
                                       chain_type_kwargs={"prompt": QA_CHAIN_PROMPT})

question = "Is probability a class topic?"
result = qa_chain({"query": question})
result["result"]

Encountered some errors. Please recheck your request!

Suggestion:

No response

[dosubot[bot]]

🤖

Hello @nnngoc,

I'm Dosu, an AI bot here to assist you with your issues and queries regarding LangChain while we wait for a human maintainer. I'm here to help you with bug troubleshooting, answering questions, and guiding you to become an effective contributor. Looking forward to assisting you!

Based on the information you've provided and the similar issues I've found in the LangChain repository, it seems like you're on the right track with creating a custom LLM. However, there are a few things you might need to adjust.

Firstly, in your LlamaLLM class, the _llm_type property should return a string that matches the name of your custom LLM in the get_type_to_cls_dict function. In your case, it should be "LlamaLLM" instead of "Llama2 7B". This is because the get_type_to_cls_dict function uses this string to import the correct LLM.

Secondly, you need to add your custom LLM to the get_type_to_cls_dict function. This function is used by the RetrievalQA.from_chain_type function to import the correct LLM. Here's how you can do it:

def get_type_to_cls_dict() -> Dict[str, Callable[[], Type[BaseLLM]]]:
    return {
        ...
        "LlamaLLM": _import_llama_llm,
        ...
    }

In the above code, _import_llama_llm is a function that imports your LlamaLLM class and returns an instance of it. You need to create this function and add it to the same file where the get_type_to_cls_dict function is defined.

Lastly, when you call the RetrievalQA.from_chain_type function, you should pass "LlamaLLM" as the chain_type argument. This is because the from_chain_type function uses this string to import the correct LLM using the get_type_to_cls_dict function.

qa_chain = RetrievalQA.from_chain_type("LlamaLLM",
                                       verbose=True,
                                       # retriever=vectordb.as_retriever(),
                                       retriever=custom_retriever,
                                       return_source_documents=True,
                                       chain_type_kwargs={"prompt": QA_CHAIN_PROMPT})

Please try these adjustments and let me know if you encounter any issues. If you're still having trouble, it would be helpful if you could provide the exact error message you're seeing when you try to use your custom LLM with the RetrievalQA.from_chain_type function.

For more information, you can refer to the LangChain documentation and this similar issue in the LangChain repository.

I hope this helps! If you have any other questions, feel free to ask.

-Dosu

Sources

• other llm api
• libs/langchain/langchain/llms/init.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[nnngoc]

@dosu-bot The first change is

def _llm_type(self) -> str:
    return "LlamaLLM"

However, for the second and third changes that you mentioned, I don't really understand. Could you write the detailed code for me? And how it use in RetrievalQA.from_chain_type function?

[dosubot[bot]]

🤖

Based on the context provided, it seems you're on the right track with integrating your custom LLM into the LangChain framework. Here's how you can adjust the _llm_type property, add your custom LLM to the get_type_to_cls_dict function, and use it in the RetrievalQA.from_chain_type function.

1. Adjust the _llm_type property in your LlamaLLM class. This property is used to identify the type of LLM. Here's an example of how you can do this:

class LlamaLLM(LLM):
    ...
    @property
    def _llm_type(self) -> str:
        return "LlamaLLM"
    ...

2. Add your custom LLM to the get_type_to_cls_dict function. This function returns a dictionary mapping LLM types to their corresponding classes. You can add your custom LLM to this dictionary like this:

def get_type_to_cls_dict() -> Dict[str, Callable[[], Type[BaseLLM]]]:
    return {
        ...
        "LlamaLLM": _import_llama_llm,
        ...
    }

In this example, _import_llama_llm is a function that imports and returns your custom LLM class. You need to implement this function yourself.

3. Use your custom LLM in the RetrievalQA.from_chain_type function. This function creates a new instance of the RetrievalQA class using a specific LLM type. You can use your custom LLM in this function like this:

RetrievalQA.from_chain_type("LlamaLLM")

In this example, "LlamaLLM" is the type of your custom LLM. This should match the key you added to the get_type_to_cls_dict function.

Please note that the actual implementation may vary depending on the specifics of your custom LLM and the LangChain framework. Also, the get_type_to_cls_dict function is not present in the provided context, so I cannot provide the exact structure and requirements of this function. If you have more information about this function or if you encounter any issues, please provide more details so I can assist you better.

Sources

• libs/langchain/langchain/llms/init.py
• libs/langchain/langchain/chains/loading.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[nnngoc]

@dosu-bot Could you define the _import_llama_llm function for me?

[dosubot[bot]]

🤖

Based on the information you've provided and the similar issue I found in the LangChain repository, you can define the _import_llama_llm function as follows:

def _import_llama_llm() -> Any:
    from langchain.llms.llama_llm import LlamaLLM

    return LlamaLLM

In this function, we first import the LlamaLLM class from the langchain.llms.llama_llm module. Then, we return the LlamaLLM class. This function will be used to dynamically import your LlamaLLM class when needed.

Please ensure that the LlamaLLM class is defined in the langchain.llms.llama_llm module and that this module is in the correct location in your project's directory structure.

As for the get_type_to_cls_dict function, I wasn't able to find the exact structure and content within the LangChain repository. However, based on its name and common patterns in Python, it's likely that this function returns a dictionary mapping type names (as strings) to class objects. Your _import_llama_llm function would be used to provide the class object for the "LlamaLLM" type.

If you have more information about the get_type_to_cls_dict function or if there's anything else you need help with, please let me know!

For more details, you can refer to the ReplitLM Model_addition_in_langchain issue and the libs/langchain/langchain/llms/init.py file in the LangChain repository.

Sources

• ReplitLM Model_addition_in_langchain
• libs/langchain/langchain/llms/init.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[nnngoc]

@dosu-bot This didn't work. Besides, it also has the issue related to insecure request, but whether it affect to the requests. (I also don't know how to fix it)

/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'myhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
Encountered some errors. Please recheck your request!

Moreover, below is the api format which I have, does it have any problem?

curl --location 'https:/myhost:10001/llama/api' -k \
--header 'Content-Type: application/json' \
--data-raw '{
    "inputs": "[INST] Question: Who is Albert Einstein? \n Answer: [/INST]",
    "parameters": {"max_new_tokens":100},
    "token": "abcdfejkwehr"
}

[dosubot[bot]]

🤖

From the code you've shared, it seems like you've correctly implemented the _call method in your LlamaLLM class. However, the error message "Encountered some errors. Please recheck your request!" is quite generic and it's hard to pinpoint the exact issue without more context. It could be related to the inputs you're passing to the RetrievalQA.from_chain_type function or an issue with the API endpoint you're calling.

To debug this, you could try the following steps:

1. Ensure that the inputs you're passing to the RetrievalQA.from_chain_type function are correct and in the expected format.
2. Test your API endpoint separately (outside of the LangChain framework) to ensure it's working as expected. You can use tools like Postman or curl for this.
3. Add some logging in your _call method to print out the request and response details. This could help identify if the issue is with the request you're sending or the response you're getting.

Regarding the InsecureRequestWarning, this is a warning message from urllib3 library indicating that the SSL certificate verification is turned off for the HTTPS request you're making. This is because you're using verify=False in your requests.post call. While you can suppress this warning by disabling it as mentioned in the context, it's generally not recommended as it could hide potential security issues. Instead, you should try to fix the underlying issue causing this warning. Here are a few options:

1. If you're using a self-signed certificate for your API, you can provide the path to your certificate file in the verify parameter like so: requests.post(url, verify='/path/to/certfile').
2. If your certificate is signed by a Certificate Authority (CA) not included in the certifi package (which requests uses for its default CA bundle), you can append your CA's certificate to this file: requests.post(url, verify='/path/to/certifi/cacert.pem').
3. If you're in a development environment and understand the risks, you can disable the warning as mentioned in the context.

Please try these suggestions and let me know if you're still encountering issues. If the problem persists, it would be helpful if you could provide more details about the error message you're seeing (e.g., full stack trace) and the inputs you're passing to the RetrievalQA.from_chain_type function.

Sources

• libs/langchain/langchain/chains/llm_requests.py
• libs/langchain/langchain/chains/llm.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[nnngoc]

@dosu-bot How can I add some logging in your _call method to print out the request and response details for testing this correct?

[dosubot[bot]]

🤖

To add logging to the _call method in your LlamaLLM class, you can use Python's built-in logging module. Here's how you can modify your _call method to include logging for the request and response:

import logging

# Create a logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

# Create a file handler
handler = logging.FileHandler('llama_llm.log')
handler.setLevel(logging.DEBUG)

# Create a logging format
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)

# Add the handlers to the logger
logger.addHandler(handler)

def _call(
    self,
    prompt: str,
    stop: Optional[List[str]] = None,
    run_manager: Optional[CallbackManagerForLLMRun] = None,
    **kwargs: Any,
) -> str:
    if stop is not None:
        raise ValueError("stop kwargs are not permitted.")

    payload = {
        "inputs": prompt,
        "parameters": {"max_new_tokens": 100},
        "token": "abcdfejkwehr"
    }

    headers = {"Content-Type": "application/json"}

    logger.info(f"Sending request to {self.llm_url} with payload: {payload} and headers: {headers}")

    response = requests.post(self.llm_url, json=payload, headers=headers, verify=False)
    response.raise_for_status()

    logger.info(f"Received response: {response.json()}")

    return response.json()['generated_text']  # get the response from the API

This will create a log file named llama_llm.log in your current directory, where all the log messages will be stored. You can change the log level, log file name, and format as per your requirements.

Regarding the InsecureRequestWarning, it's raised because you're making a HTTPS request without verifying the SSL certificate. This is generally not recommended as it can make your application vulnerable to man-in-the-middle (MitM) attacks. If you're aware of the risks and still want to suppress this warning, you can do so by adding the following lines at the beginning of your script:

import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

However, the best way to fix this warning is to ensure that the SSL certificate of the endpoint is verified. If you're using a self-signed certificate, you can pass the path to the certificate file to the verify parameter in the requests.post call like this:

response = requests.post(self.llm_url, json=payload, headers=headers, verify='/path/to/certfile')

Please note that logging sensitive information such as API keys or personal data is not recommended. Also, suppressing warnings related to SSL certificate verification should only be done if you're aware of the risks and have a valid reason to do so.

Sources

• libs/langchain/langchain/llms/llamacpp.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[nnngoc]

@dosu-bot I still have this error

INFO:__main__:Received response: {'generated_text': 'Encountered some errors. Please recheck your request!'}

> Finished chain.
Encountered some errors. Please recheck your request!

Have any problem here? Btw, the format api of my side is same below:

curl --location 'https:/myhost:10001/llama/api' -k \
--header 'Content-Type: application/json' \
--data-raw '{
    "inputs": "[INST] Question: Who is Albert Einstein? \n Answer: [/INST]",
    "parameters": {"max_new_tokens":100},
    "token": "abcdfejkwehr"
}

Is this have any problem with my setup to wrap the LLM from api

class LlamaLLM(LLM):
    llm_url = 'https:/myhost/llama/api'

    class Config:
        extra = Extra.forbid

    @property
    def _llm_type(self) -> str:
        return "Llama2 7B"

    def _call(
        self,
        prompt: str,
        stop: Optional[List[str]] = None,
        run_manager: Optional[CallbackManagerForLLMRun] = None,
        **kwargs: Any,
    ) -> str:
        if stop is not None:
            raise ValueError("stop kwargs are not permitted.")

        payload = {
            "inputs": prompt,
            "parameters": {"max_new_tokens": 100},
            "token": "abcdfejkwehr"
        }

        headers = {"Content-Type": "application/json"}

        response = requests.post(self.llm_url, json=payload, headers=headers, verify=False)
        response.raise_for_status()

        # print("API Response:", response.json())

        return response.json()['generated_text']  # get the response from the API

    @property
    def _identifying_params(self) -> Mapping[str, Any]:
        """Get the identifying parameters."""
        return {"llmUrl": self.llm_url}

[dosubot[bot]]

🤖

From the information you've provided, it seems like the error message "Encountered some errors. Please recheck your request!" is coming from your custom LLM API, not from the LangChain framework. The log message you've shared shows that this is the response your _call method is receiving from the API:

INFO:__main__:Received response: {'generated_text': 'Encountered some errors. Please recheck your request!'}

This suggests that the request your LlamaLLM class is sending to the API is not formatted correctly, or the API is not able to process it for some reason.

Looking at your _call method, it seems like you're formatting the request correctly according to the example curl command you've provided. However, there are a few things you could check:

1. API URL: Make sure that the llm_url attribute of your LlamaLLM class is set to the correct URL for your API. In your example curl command, the URL is 'https:/myhost:10001/llama/api', but in your LlamaLLM class, it's 'https:/myhost/llama/api'. Notice the missing port number (10001) in the class attribute.

2. Request Headers: Check that the headers in your request are correct. In your example curl command, you're only setting the 'Content-Type' header, which you're also doing in your _call method. However, if your API requires additional headers (like an authorization header, for example), you'll need to include those in your request as well.

3. Request Body: Make sure that the body of your request is formatted correctly. In your _call method, you're setting the "inputs" field of the request body to the prompt argument, the "parameters" field to {"max_new_tokens": 100}, and the "token" field to "abcdfejkwehr". If your API expects these fields to be in a different format, or if it requires additional fields, you'll need to adjust your request body accordingly.

4. API Response: If none of the above suggestions help, you might need to look at the response from your API more closely. If the API provides more detailed error messages or codes, those could help you figure out what's going wrong. You could modify your _call method to log the entire response from the API, not just the 'generated_text' field, to get more information.

I hope this helps! If you're still having trouble, please provide more details about your custom LLM API, such as its expected request format and response format, and any error messages or codes it might return.

Sources

• libs/langchain/langchain/llms/init.py
• libs/langchain/langchain/chains/api/openapi/response_chain.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[nnngoc]

@dosu-bot API Response: If none of the above suggestions help, you might need to look at the response from your API more closely. If the API provides more detailed error messages or codes, those could help you figure out what's going wrong. You could modify your _call method to log the entire response from the API, not just the 'generated_text' field, to get more information.

please support me modify the _call method to log the entire response from the API, not just the 'generated_text' field, to get more information.

[dosubot[bot]]

🤖

To log the entire response from the API, you can use Python's built-in logging module. Here's how you can modify your _call method:

import logging

def _call(
    self,
    prompt: str,
    stop: Optional[List[str]] = None,
    run_manager: Optional[CallbackManagerForLLMRun] = None,
    **kwargs: Any,
) -> str:
    if stop is not None:
        raise ValueError("stop kwargs are not permitted.")

    payload = {
        "inputs": prompt,
        "parameters": {"max_new_tokens": 100},
        "token": "abcdfejkwehr"
    }

    headers = {"Content-Type": "application/json"}

    response = requests.post(self.llm_url, json=payload, headers=headers, verify=False)
    response.raise_for_status()

    # Log the entire response
    logging.info(response.json())

    return response.json()['generated_text']  # get the response from the API

In this modification, I've added a line logging.info(response.json()) to log the entire response from the API. This will print the entire response to the console. If you want to write the logs to a file, you can configure the logging module like this:

logging.basicConfig(filename='api.log', level=logging.INFO)

This will write all logs with level INFO and above to a file named 'api.log'. You can adjust the filename and log level according to your needs.

Please note that logging the entire response might include sensitive information in the logs, such as access tokens or personal data. Make sure to handle this data carefully and consider redacting sensitive information.

For more information on Python's logging module, you can refer to the official Python documentation.

I hope this helps! If you have any other questions, feel free to ask.

Sources

• libs/langchain/langchain/llms/llamacpp.py
• libs/langchain/langchain/llms/bedrock.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[nnngoc]

@dosu-bot What is the format of llm in RetrievalQA.from_chain_type function? I tested the function and it was OK, but maybe the format in the RetrivalQA is different, so that caused the failure.

llama_llm = LlamaLLM()
prompt = "[INST] Question: Who is Albert Einstein? \n Answer: [/INST]"
result = llama_llm._call(prompt)
print(result)

Albert Einstein (1879-1955) was a German-born theoretical physicist who is widely regarded as one of the most influential scientists of the 20th century. He is best known for his theory of relativity, which revolutionized our understanding of space and time, and his famous equation E=mc².

[dosubot[bot]]

🤖

Based on the information you've provided and the context from the LangChain repository, it seems that the RetrievalQA.from_chain_type function expects the Language Learning Model (LLM) to be an instance of the BaseLanguageModel class. This is evident from the function signature:

@classmethod
def from_chain_type(
    cls,
    llm: BaseLanguageModel,
    chain_type: str = "stuff",
    chain_type_kwargs: Optional[dict] = None,
    **kwargs: Any,
) -> BaseRetrievalQA:

To make your custom LlamaLLM class compatible with the from_chain_type function, you need to ensure that LlamaLLM is a subclass of BaseLanguageModel or implements the same interface as BaseLanguageModel. This means that LlamaLLM should have the same methods and properties as BaseLanguageModel.

Here is a basic example of how you can adjust your LlamaLLM class:

from langchain_core.language_models import BaseLanguageModel

class LlamaLLM(BaseLanguageModel):
    # Your implementation here
    pass

Please note that you need to implement the methods and properties of BaseLanguageModel in your LlamaLLM class. The exact implementation will depend on the specifics of your language model.

In a similar issue in the LangChain repository, a user created a wrapper class for their custom LLM that inherits from BaseLanguageModel and implements the required methods. You might find this approach useful.

However, without the exact error message you're encountering, it's difficult to provide a more specific solution. Could you please provide the error message or describe the issue in more detail? This will help me understand the problem better and provide a more accurate solution.

Sources

• Issue with Langchain LLM Chains
• libs/langchain/langchain/chains/retrieval_qa/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[nnngoc]

@dosu-bot

/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'ws.gvlab.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
Encountered some errors. Please recheck your request!

How can show exactly what is encountered in the request? because now it's hard to debug

[dosubot[bot]]

🤖

To log the entire response from your custom LLM API, you need to modify the _run methods in the RequestsGetToolWithParsing, RequestsPostToolWithParsing, RequestsPatchToolWithParsing, RequestsPutToolWithParsing, and RequestsDeleteToolWithParsing classes in the LangChain codebase. These classes are responsible for making HTTP requests and parsing the responses.

Currently, these methods are truncating the response to self.response_length and then passing it to the llm_chain.predict method. Instead, you should log the entire response before truncating it.

Here's how you can modify the _run method in the RequestsGetToolWithParsing class:

def _run(self, text: str) -> str:
    try:
        data = parse_json_markdown(text)
    except json.JSONDecodeError as e:
        raise e
    data_params = data.get("params")
    response = self.requests_wrapper.get(data["url"], params=data_params)
    print(response)  # Log the entire response
    response = response[: self.response_length]
    return self.llm_chain.predict(
        response=response, instructions=data["output_instructions"]
    ).strip()

You can apply similar changes to the _run methods in the other classes. You can find these classes in the libs/langchain/langchain/agents/agent_toolkits/openapi/planner.py file.

As for the InsecureRequestWarning from urllib3, it's raised when making an unverified HTTPS request. To suppress this warning, you can add the following lines at the beginning of your script:

import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

This will disable the InsecureRequestWarning. However, please note that it's generally not recommended to disable these warnings in a production environment as they're intended to alert you of potentially insecure network activity.

Please let me know if you need further assistance.

Sources

• libs/langchain/langchain/agents/agent_toolkits/openapi/planner.py
• libs/langchain/langchain/tools/requests/tool.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[nnngoc]

@dosu-bot Have any easy way to check it? Again, consider my setup below and give the best as you can for the solving problem and debugging.

from pydantic import Extra
import requests
from typing import Any, List, Mapping, Optional

from langchain.callbacks.manager import CallbackManagerForLLMRun
from langchain.llms.base import LLM

class LlamaLLM(LLM):
    llm_url = 'https:/myhost:10001/llama/api'

    class Config:
        extra = Extra.forbid

    @property
    def _llm_type(self) -> str:
        return "Llama2 7B"

    def _call(
        self,
        prompt: str,
        stop: Optional[List[str]] = None,
        run_manager: Optional[CallbackManagerForLLMRun] = None,
        **kwargs: Any,
    ) -> str:
        if stop is not None:
            raise ValueError("stop kwargs are not permitted.")

        payload = {
            "inputs": prompt,
            "parameters": {"max_new_tokens": 100},
            "token": "abcdfejkwehr"
        }

        headers = {"Content-Type": "application/json"}

        response = requests.post(self.llm_url, json=payload, headers=headers, verify=False)
        response.raise_for_status()

        # print("API Response:", response.json())

        return response.json()['generated_text']  # get the response from the API

    @property
    def _identifying_params(self) -> Mapping[str, Any]:
        """Get the identifying parameters."""
        return {"llmUrl": self.llm_url}

llm = LlamaLLM()

#Testing
prompt = "[INST] Question: Who is Albert Einstein? \n Answer: [/INST]"
result = llm._call(prompt)
print(result)

Albert Einstein (1879-1955) was a German-born theoretical physicist who is widely regarded as one of the most influential scientists of the 20th century. He is best known for his theory of relativity, which revolutionized our understanding of space and time, and his famous equation E=mc².

# Build prompt
from langchain.prompts import PromptTemplate
template = """[INST] <<SYS>>

Answer the question base on the context below.

<</SYS>>

Context: {context}
Question: {question}
Answer:
[/INST]"""
QA_CHAIN_PROMPT = PromptTemplate(input_variables=["context", "question"],template=template,)

# Run chain
from langchain.chains import RetrievalQA

qa_chain = RetrievalQA.from_chain_type(llm,
                                       verbose=True,
                                       # retriever=vectordb.as_retriever(),
                                       retriever=custom_retriever,
                                       return_source_documents=True,
                                       chain_type_kwargs={"prompt": QA_CHAIN_PROMPT})

question = "Is probability a class topic?"
result = qa_chain({"query": question})
result["result"]

> Entering new RetrievalQA chain...

> Finished chain.
Encountered some errors. Please recheck your request!

[dosubot[bot]]

🤖

Hello @dosubot[bot]! I'm Dosu, a friendly bot here to assist you with your inquiries, bugs, and help you become a better contributor. While we're waiting for a human maintainer to join us, feel free to ask your questions. I'm here to make your journey with LangChain smoother and more enjoyable. Let's tackle those challenges together!

To log the entire response from your custom Language Learning Model (LLM) API, you can modify the _call method in the LLMRequestsChain class. Instead of returning only the result, you can return the entire result object.

Here's how you can modify your code:

class LLMRequestsChain(Chain):
    # ... rest of your code ...

    def _call(
        self,
        inputs: Dict[str, Any],
        run_manager: Optional[CallbackManagerForChainRun] = None,
    ) -> Dict[str, Any]:
        from bs4 import BeautifulSoup
        _run_manager = run_manager or CallbackManagerForChainRun.get_noop_manager()
        # Other keys are assumed to be needed for LLM prediction
        other_keys = {k: v for k, v in inputs.items() if k != self.input_key}
        url = inputs[self.input_key]
        res = self.requests_wrapper.get(url)
        # extract the text from the html
        soup = BeautifulSoup(res, "html.parser")
        other_keys[self.requests_key] = soup.get_text()[: self.text_length]
        result = self.llm_chain.predict(
            callbacks=_run_manager.get_child(), **other_keys
        )
        # Log the entire result object
        print(result)
        return {self.output_key: result}

    # ... rest of your code ...

This modification will print the entire result object to the console. If you want to log it to a file or a logging service, you can replace the print(result) line with your preferred logging method. You can find the relevant code in the LLMRequestsChain class in the LangChain codebase.

Regarding the InsecureRequestWarning from urllib3, you can suppress the warning by adding the following lines at the beginning of your script:

import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

This will disable the warning, but please note that this warning is there for a reason. It's warning you that the HTTPS request you're making is not verified, which means it's susceptible to man-in-the-middle attacks. It's generally recommended to fix the underlying issue (i.e., use a verified HTTPS request) rather than just suppressing the warning.

The result object returned by the llm_chain.predict method in the LangChain codebase is an instance of the LLMResult class. This class has the following structure:

• generations: This is a list of lists, where each inner list contains instances of the Generation class. Each input could have multiple candidate generations, hence the nested list structure.

• llm_output: This is an optional dictionary that contains arbitrary LLM provider-specific output.

• run: This is an optional list of RunInfo instances. Each RunInfo instance contains metadata info for the model call for each input.

You can find more details about the LLMResult class in the LangChain codebase.

I hope this helps! If you have any other questions, feel free to ask.

Sources

• libs/langchain/langchain/chains/llm_requests.py
• libs/core/langchain_core/outputs/llm_result.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[nnngoc]

@dosu-bot Could I ask for support from some dev of Langchain to solve this case?

[dosubot[bot]]

It seems like this issue requires a deeper look, @baskaryan could you please lend your expertise?

[nnngoc]

@dosu-bot This happens because of the context length setting of the API. So I already fixed it and it works fine now. Thanks for your support!

[dosubot[bot]]

🤖

I'm glad to hear that you were able to resolve the issue with the context length setting for the LLM API in the LangChain framework. The maximum context length is indeed determined by the model being used, as you can see in the modelname_to_contextsize method in the BaseOpenAI class. This method maps model names to their maximum context sizes and raises a ValueError if the model name is not found in the dictionary.

As for the InsecureRequestWarning from urllib3, this is a common warning message that is displayed when making unverified HTTPS requests. urllib3 will issue this warning when it cannot verify the SSL certificate for a request. This is not specific to the LangChain framework and is a general warning from urllib3. It's always recommended to verify your HTTPS requests to ensure the security of your data.

If you have any more questions or run into any other issues, feel free to ask. I'm here to help!

Sources

• libs/langchain/langchain/llms/openai.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[hitsense]

I am trying something similar, with RetrievalQA from chain type, but I get below error for my custom Mistral7B LLM 'Mistral7B_LLM' object has no attribute 'callbacks'

@dosu-bot