aviadjo commented 10 months ago

Version Information

Python 3.11
Package Version: 0.5.1
Operating System: MacOS

What is the expected behavior?

I expect the code to work with ibm-generative-ai LLM as it works with OpenAI LLM.

Please provide a unit test that demonstrates the bug.

import os from genai import Credentials from genai.schemas import GenerateParams, ReturnOptions from genai.extensions.langchain import LangChainInterface

BAM LLM

llm = LangChainInterface( model="meta-llama/llama-2-70b-chat", params=GenerateParams( min_new_tokens=1, max_new_tokens=200, temperature=0.15, top_k=20, top_p=1.0, # decoding_method='sample', # stop_sequences=None, repetition_penalty=1.15, return_options=ReturnOptions( generated_tokens=True, token_logprobs=True,

input_tokens=True,

        # top_n_tokens=5
    )
),
credentials=Credentials(
    api_key=os.getenv("GENAI_KEY", None),
    api_endpoint=os.getenv("GENAI_API", None)
)

)

Agent Tools

from langchain.agents import load_tools, initialize_agent, AgentType, AgentExecutor from langchain.tools import StructuredTool, tool

tools = load_tools( tool_names=[ 'ddg-search', # pip install duckduckgo-search 'llm-math', # pip install numexpr ], llm=llm )

Agent

agent = initialize_agent( tools=tools, llm=llm, agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION, verbose=True, handle_parsing_errors=True )

user_query = 'What is CVSS score of CVE-2021-44228? What is the severity of the score?'

Debug

from langchain_core.tracers import langchain langchain.debug = True

Run Agent

agent.run(user_query)

What is the actual behavior?

Entering new AgentExecutor chain... Could not parse LLM output: Answer:

(Note: The current year is 2023.) Observation: Invalid or incomplete response

Other notes on how to reproduce the issue?

Any possible solutions?

Can you identify the location in the GENAI source code where the problem exists?

If the bug is confirmed, would you be willing to submit a PR?

Yes / No (Help can be provided if you need assistance submitting a PR)

David-Kristek commented 10 months ago

Hey @aviadjo, The issue you mentioned isn't directly tied to the ibm-generative-ai library but rather to the LLMs you're using. The prompt the LLM receives is quite complex and the LLM has to respond in the exact format described in the prompt generated by Lanchain depending on what tools you are using.

Sometimes, less well-trained LLMs might not provide responses in the exact format required. In such instances, Langchain cannot parse them, resulting in the message: Could not parse LLM output. You can observe how the response is parsed here.

To make it work I would try to change the params or try another model like meta-llama/llama-2-70b and give it a few tries.