Support Mistral through Bedrock

skehlet commented 4 months ago

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

from langchain_community.llms import Bedrock
from langchain.prompts import PromptTemplate
from langchain.chains.combine_documents.stuff import StuffDocumentsChain
from langchain.chains.llm import LLMChain
from langchain.schema import Document

import langchain
langchain.verbose = True

llm = Bedrock(
    model_id="mistral.mixtral-8x7b-instruct-v0:1",
    model_kwargs={"max_tokens": 250},
)
prompt_template = """Summarize the following text in no more than 3 to 4 sentences: 

{text}

CONCISE SUMMARY:"""
prompt = PromptTemplate.from_template(prompt_template)
llm_chain = LLMChain(llm=llm, prompt=prompt)
stuff_chain = StuffDocumentsChain(llm_chain=llm_chain, document_variable_name="text")
doc = Document(
    page_content="""Today, we’re excited to announce the availability of two
                 high-performing Mistral AI models, Mistral 7B and Mixtral 8x7B,
                 on Amazon Bedrock."""
)
results = stuff_chain.run([doc])
print(results)

Error Message and Stack Trace (if applicable)

/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The function run was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use invoke instead. warn_deprecated( Traceback (most recent call last): File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain_community/llms/bedrock.py", line 444, in _prepare_input_and_invoke response = self.client.invoke_model(**request_options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/botocore/client.py", line 553, in _api_call return self._make_api_call(operation_name, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/botocore/client.py", line 1009, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: #: required key [prompt] not found#: extraneous key [inputText] is not permitted, please reformat your input and try again.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/skehlet/Documents/workspace/email-genai-summarizer/test2.py", line 20, in results = stuff_chain.run([doc]) ^^^^^^^^^^^^^^^^^^^^^^ File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain_core/_api/deprecation.py", line 145, in warning_emitting_wrapper return wrapped(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain/chains/base.py", line 545, in run return self(args[0], callbacks=callbacks, tags=tags, metadata=metadata)[ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain_core/_api/deprecation.py", line 145, in warning_emitting_wrapper return wrapped(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain/chains/base.py", line 378, in call return self.invoke( ^^^^^^^^^^^^ File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain/chains/base.py", line 163, in invoke raise e File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain/chains/base.py", line 153, in invoke self._call(inputs, run_manager=run_manager) File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain/chains/combine_documents/base.py", line 137, in _call output, extra_return_dict = self.combine_docs( ^^^^^^^^^^^^^^^^^^ File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain/chains/combine_documents/stuff.py", line 244, in combine_docs return self.llm_chain.predict(callbacks=callbacks, inputs), {} ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain/chains/llm.py", line 293, in predict return self(kwargs, callbacks=callbacks)[self.output_key] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain_core/_api/deprecation.py", line 145, in warning_emitting_wrapper return wrapped(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain/chains/base.py", line 378, in call return self.invoke( ^^^^^^^^^^^^ File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain/chains/base.py", line 163, in invoke raise e File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain/chains/base.py", line 153, in invoke self._call(inputs, run_manager=run_manager) File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain/chains/llm.py", line 103, in _call response = self.generate([inputs], run_manager=run_manager) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain/chains/llm.py", line 115, in generate return self.llm.generate_prompt( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 568, in generate_prompt return self.generate(prompt_strings, stop=stop, callbacks=callbacks, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 741, in generate output = self._generate_helper( ^^^^^^^^^^^^^^^^^^^^^^ File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 605, in _generate_helper raise e File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 592, in _generate_helper self._generate( File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 1177, in _generate self._call(prompt, stop=stop, run_manager=run_manager, kwargs) File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain_community/llms/bedrock.py", line 718, in _call return self._prepare_input_and_invoke( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/skehlet/Documents/workspace/email-genai-summarizer/venv/lib/python3.11/site-packages/langchain_community/llms/bedrock.py", line 451, in _prepare_input_and_invoke raise ValueError(f"Error raised by bedrock service: {e}") ValueError: Error raised by bedrock service: An error occurred (ValidationException) when calling the InvokeModel operation: Malformed input request: #: required key [prompt] not found#: extraneous key [inputText] is not permitted, please reformat your input and try again.

Description

I'm trying to use langchain to use Mistral through AWS Bedrock. It just came out so understandably it's not yet supported.

I found the following quick patch to bedrock.py worked for me, hopefully this is helpful:

--- venv/lib/python3.11/site-packages/langchain_community/llms/bedrock.py.orig  2024-03-03 12:44:35
+++ venv/lib/python3.11/site-packages/langchain_community/llms/bedrock.py       2024-03-03 12:44:58
@@ -104,6 +104,9 @@
             input_body = dict()
             input_body["inputText"] = prompt
             input_body["textGenerationConfig"] = {**model_kwargs}
+        elif provider == "mistral":
+            input_body = dict()
+            input_body["prompt"] = prompt
         else:
             input_body["inputText"] = prompt

@@ -126,6 +129,8 @@
                 text = response_body.get("generations")[0].get("text")
             elif provider == "meta":
                 text = response_body.get("generation")
+            elif provider == "mistral":
+                text = response_body.get("outputs")[0].get("text")
             else:
                 text = response_body.get("results")[0].get("outputText")

System Info

System Information

OS: Darwin OS Version: Darwin Kernel Version 23.3.0: Wed Dec 20 21:30:44 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6000 Python Version: 3.11.7 (main, Dec 4 2023, 18:10:11) [Clang 15.0.0 (clang-1500.1.0.2.5)]

Package Information

langchain_core: 0.1.28 langchain: 0.1.10 langchain_community: 0.0.25 langsmith: 0.1.13 langchain_text_splitters: 0.0.1

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langgraph langserve

rhubarb commented 4 months ago

is there a pr for this?

ritwickb commented 4 months ago

I am having the same issue in using Mistral in Bedrock. And can confirm that the workaround that skehlet mentioned solves the issue. But this would need a proper fix. Please prioritize this fix.

skehlet commented 3 months ago

Finally had a chance to test, I grabbed the latest version, and Mistal through Bedrock is working. Thank you to the LangChain community.

langchain-ai / langchain