langchain-ai / langchain-aws

Build LangChain Applications on AWS
MIT License
95 stars 70 forks source link

ChatBedrock model doesn't work with llama 2 models when using the stop sequence in non-streaming mode #62

Open haoguohua opened 4 months ago

haoguohua commented 4 months ago

When using ChatBedrock with llama 2 models and generating texts in non-streaming mode, setting stop sequence will cause malformed input request error.

e.g. running the following code

model_id = "meta.llama2-70b-chat-v1"
model_kwargs = {
    "temperature": 0,
    "max_gen_len": 2048,
}

llm = BedrockChat(
    model_id=model_id,
    model_kwargs=model_kwargs
)

llm.invoke("show me the weather in sf", stop=["Humidity"])

will result in the following error message

ValueError: Error raised by bedrock service: An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: extraneous key [stop_sequences] is not permitted, please reformat your input and try again.