Closed Antriksh29071989 closed 6 days ago
🤖
Hello @Antriksh29071989! I'm Dosu, a friendly bot here to assist you with any bugs, questions, and help you contribute to the LangChain repository. While we wait for a human maintainer, I'm here to give you a hand. Let's tackle this together!
Thank you for providing detailed information about your issue. Based on the information you've provided and the context from similar issues, I have a few suggestions that might help.
Firstly, it seems like you're using the LlamaCpp
class from the LangChain library to load the model and perform inference. In your code, you've set the f16_kv
parameter to True
. This parameter is used to determine whether to use half-precision for the key/value cache. Using half-precision can help reduce the memory footprint and potentially speed up computations, but at the cost of a slight decrease in precision. This might be causing the unexpected output you're seeing.
In a similar issue titled Output using llamacpp is garbage, the problem was resolved by changing the default value passed for f16_kv
. You might want to try setting f16_kv
to False
in your code and see if that resolves the issue.
Secondly, you've set the n_gpu_layers
parameter to 1
. This parameter determines the number of layers to be loaded into GPU memory. Loading more layers into the GPU memory can potentially speed up the inference process, but it will also consume more GPU memory. If your GPU memory is not sufficient, it might cause issues during inference. You might want to try adjusting this parameter based on your GPU's VRAM pool.
Here's how you can modify your code:
llm = LlamaCpp(
model_path="../models/deepcoder-gguf/deepseek-coder-6.7b-instruct.Q2_K.gguf",
n_gpu_layers=n_gpu_layers,
max_tokens=2000,
top_p=1,
n_batch=n_batch,
callback_manager=callback_manager,
f16_kv=False, # Change this to False
verbose=True, # Verbose is required to pass to the callback manager
)
Please try these suggestions and let me know if they help. If the issue persists, it would be helpful if you could provide more information about your environment, such as the version of Python you're using, the version of the llama_cpp
library, and the specifications of your GPU.
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
If you want to continue the conversation, start your reply with @dosu-bot.
Changed it to f16_kv=False
. Still the same issue.
llm = LlamaCpp(
model_path="..models/deepcoder-gguf/deepseek-coder-6.7b-instruct.Q2_K.gguf",
n_gpu_layers=n_gpu_layers,
max_tokens=2000,
top_p=1,
n_batch=n_batch,
callback_manager=callback_manager,
f16_kv=False,
verbose=True, # Verbose is required to pass to the callback manager
)
Output: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
@dosu-bot
same problem
Same issue. Unable to run Deepseek-6.7 and TheBloke/CodeLlama-7B-Python-GGUF successfully. The models get loaded but output is garbage when called with from langchain_community.llms import LlamaCpp
WizardCoder-7b is working fine though.
The reason is LangChain cover classes do not suppor instruction tuned models, which require special tokens.
All the prompts that you have in chain are transformed into simple string. And this string does not contain proper special tokens for chat.
LlamaCPP class of LangChain calls basic llama_cpp.Llama's call method. Not the "chat" Version of it
Solution: One have to create ChatLlamaCpp class, that invokes chat generation method
As well as to create possibility to pass messages objects through pipeline. Not their parsed strings
To address the issue with using instruction-tuned models like deepseek-coder-6.7b-instruct.Q5_K_M.gguf
with LangChain's LlamaCpp
class, you can implement a ChatLlamaCpp
class that invokes the chat generation method. Here is an example implementation:
import json
import logging
from typing import Any, Dict, List, Mapping, Optional, Tuple
from langchain.schema import ChatGeneration, ChatResult
from langchain_core.callbacks.manager import CallbackManagerForLLMRun
from langchain_core.language_models import BaseChatModel
from langchain_core.messages import AIMessage, BaseMessage, ChatMessage, FunctionMessage, HumanMessage, SystemMessage
logger = logging.getLogger(__name__)
def _convert_dict_to_message(_dict: Mapping[str, Any]) -> BaseMessage:
role = _dict["role"]
if role == "user":
return HumanMessage(content=_dict["content"])
elif role == "assistant":
content = _dict.get("content") or ""
if _dict.get("function_call"):
_dict["function_call"]["arguments"] = json.dumps(_dict["function_call"]["arguments"])
additional_kwargs = {"function_call": dict(_dict["function_call"])}
else:
additional_kwargs = {}
return AIMessage(content=content, additional_kwargs=additional_kwargs)
elif role == "system":
return SystemMessage(content=_dict["content"])
elif role == "function":
return FunctionMessage(content=_dict["content"], name=_dict["name"])
else:
return ChatMessage(content=_dict["content"], role=role)
def _convert_message_to_dict(message: BaseMessage) -> dict:
if isinstance(message, ChatMessage):
message_dict = {"role": message.role, "content": message.content}
elif isinstance(message, HumanMessage):
message_dict = {"role": "user", "content": message.content}
elif isinstance(message, AIMessage):
message_dict = {"role": "assistant", "content": message.content}
if "function_call" in message.additional_kwargs:
message_dict["function_call"] = message.additional_kwargs["function_call"]
elif isinstance(message, SystemMessage):
message_dict = {"role": "system", "content": message.content}
elif isinstance(message, FunctionMessage):
message_dict = {"role": "function", "content": message.content, "name": message.name}
else:
raise ValueError(f"Got unknown type {message}")
if "name" in message.additional_kwargs:
message_dict["name"] = message.additional_kwargs["name"]
return message_dict
class ChatLlamaCpp(BaseChatModel):
"""Chat model using the Llama C++ API."""
client: Any #: :meta private:
def _generate(
self,
messages: List[BaseMessage],
stop: Optional[List[str]] = None,
run_manager: Optional[CallbackManagerForLLMRun] = None,
**kwargs: Any,
) -> ChatResult:
message_dicts, params = self._create_message_dicts(messages, stop)
_params = {"messages": message_dicts}
final_params = {**params, **kwargs, **_params}
response = self.client.run(final_params).json()
return self._create_chat_result(response)
def _create_message_dicts(
self, messages: List[BaseMessage], stop: Optional[List[str]]
) -> Tuple[List[Dict[str, Any]], Dict[str, Any]]:
params = dict(self._client_params)
if stop is not None:
if "stop" in params:
raise ValueError("`stop` found in both the input and default params.")
params["stop"] = stop
message_dicts = [_convert_message_to_dict(m) for m in messages]
return message_dicts, params
def _create_chat_result(self, response: Mapping[str, Any]) -> ChatResult:
generations = []
for res in response["choices"]:
message = _convert_dict_to_message(res["message"])
gen = ChatGeneration(
message=message,
generation_info=dict(finish_reason=res.get("finish_reason")),
)
generations.append(gen)
return ChatResult(generations=generations)
@property
def _client_params(self) -> Mapping[str, Any]:
"""Get the parameters used for the client."""
return {}
@property
def _llm_type(self) -> str:
"""Return type of chat model."""
return "llama-cpp"
This ChatLlamaCpp
class is designed to handle chat-specific interactions by converting messages to and from the required format and invoking the chat generation method of the Llama C++ API. Ensure that the client
attribute is properly initialized to interact with the Llama C++ API [1][2].
To continue the conversation, mention @dosu.
System Info
OS: Apple M1 Max
Name: langchain Version: 0.0.349 Summary: Building applications with LLMs through composability Home-page: https://github.com/langchain-ai/langchain Author: Author-email: License: MIT Requires: aiohttp, async-timeout, dataclasses-json, jsonpatch, langchain-community, langchain-core, langsmith, numpy, pydantic, PyYAML, requests, SQLAlchemy, tenacity Required-by:
Who can help?
@hwchase17 @agola11
Information
Related Components
Reproduction
Steps to reproduce:
I have followed the instructions provided here : https://python.langchain.com/docs/integrations/llms/llamacpp. Though not able inference it correctly.
Model path : https://huggingface.co/TheBloke/deepseek-coder-6.7B-instruct-GGUF
Result:
< """"""""""""""""""""""/"
Requesting you to look into it. Please let me know in case you need more information. Thank you.
I have tried the same model file with llama-cpp-python package and it works as expected. Please find below the code that I have tried:
Output :
Expected behavior
It should inference the model just like the native llama-cpp-python package.