Open dolanp83 opened 1 month ago
+1
+1
A very ugly, non-production hack I've had some success with is
@tool
def multiply(a: str, b: str) -> int:
"""Multiply two numbers."""
print(f"multiply() invoked with a={a}, b={b}")
return int(a) * int(b)
# tools: List[StructuredTool] = [get_entity, invoke_home_assistant_service, brave_search, multiply]
tools: List[StructuredTool] = [multiply]
model = ChatOpenAI(
model="hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4",
# temperature=0.75,
# num_ctx=16 * 1024,
base_url="http://localhost:8000/v1",
openai_api_key="sk-1234567890",
verbose=True,
)
llm_with_tools = model.bind_tools(tools)
def parse_tool_call(x: Any):
if isinstance(x, AIMessage):
if x.content.startswith("<|python_tag|>"):
x.content = x.content.replace("<|python_tag|>", "")
try:
parsed_call = json.loads(x.content)
if parsed_call.get("name") is not None and parsed_call.get("parameters") is not None:
x.tool_calls = [ToolCall(name=parsed_call["name"], args=parsed_call["parameters"], id=x.id)]
except json.JSONDecodeError:
pass
return x
chain = llm_with_tools | RunnableLambda(parse_tool_call)
and then when running
You: what is 6 times 8
debug: {"name": "multiply", "parameters": {"a": "6", "b": "8"}}
debug: Function call: {'name': 'multiply', 'args': {'a': '6', 'b': '8'}, 'id': 'run-11debe90-b2bf-4067-a006-e59b43da8606-0'}
debug: multiply() invoked with a=6, b=8
debug: Function result: 48
AI: 48, I hope that is correct, sir. Shall I be of further assistance?
Your current environment
Using vllm/vllm-openai:v0.5.3.post1 Docker image. Executed within the container:
🐛 Describe the bug
I have been having a lot of trouble getting vLLMs Open AI-compatible API to work with various LangChain/LangGraph tools, so I distilled this down to a contrived example. When a tool call is supposed to happen, the response from vLLM just has the
<python_tag>...
metadata from Llama3.1, but it is unable to call the function.Here's the output when this is run with OpenAI:
Here is the output when this is run with Ollama:
And here is the output when used with vLLM:
Does something else need to be added to the workflow to get this to work with vLLM and Llama 3.1? Thanks!