DOC: Under Modules/LLM/Integrations, the Sagemaker Endpoint integration example the "ContentHandler()" is giving an error

[loveneetsingh7]

Issue with current documentation:

The Example provided in the Documentation for the Integration of the Sagemaker Endpoint is giving the error

An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
  "code": 400,
  "type": "InternalServerException",
  "message": "Input payload must contain text_inputs key."
}
" 

The error comes from this block of code in the example:

`class ContentHandler(LLMContentHandler): content_type = "application/json" accepts = "application/json"

def transform_input(self, prompt: str, model_kwargs: Dict) -> bytes:
    input_str = json.dumps({prompt: prompt, **model_kwargs})
    return input_str.encode('utf-8')

def transform_output(self, output: bytes) -> str:
    response_json = json.loads(output.read().decode("utf-8"))
    return response_json[0]["generated_text"]`

specifically, because it does not contain the "text_inputs" key in the "transform_input" function of the "ContentHandler" class.

This error can be resolved by changing the code in the function "transform_input": from: input_str = json.dumps({prompt: prompt, **model_kwargs}) to: input_str = json.dumps({"text_inputs": prompt, **model_kwargs})

But still, another error comes:

"in transform_output
    return response_json[0]["generated_text"]
KeyError: 0"

It is because of the function "transform_output" in the class "ContentHandler" class.

This error can be resolved by changing the code in the function "transform_output": from: return response_json[0]["generated_text"] to: return response_json['generated_texts'][0]

This makes the example code run properly without any errors. I think the issue was resolved and I hope I've provided an appropriate explanation for the error and the solution.

Idea or request for content:

The Example of the Integration of the LLMs with the Sagemaker Endpoint that is present in the current documentation does not provide a working example because of the problem in the "ContentHandler" class, which generates errors :

1. An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{ "code": 400, "type": "InternalServerException", "message": "Input payload must contain text_inputs key." } "

2. "in transform_output return response_json[0]["generated_text"] KeyError: 0"

These errors pose a problem that can be solved with the code and recommendations provided above. Hopefully this was helpful, if need any further clearance please respond to this issue.

[MultiTask70]

Resolved my issue ! Thank you

[dpranav12]

I am getting same error - any ideas how to fix ?

$ cat test.py import boto3 import json import os import io import csv

runtime = boto3.client('sagemaker-runtime', region_name='us-east-1')

input_data = {"department_id": 5, "job": "Editor", "rowid": 874778072614862849}

response = runtime.invoke_endpoint(EndpointName=‘xxxx-endpoint-xxxx', ContentType='application/json', Accept='application/json', Body=json.dumps(input_data))

output_data = json.loads(response['Body'].read().decode()) print(output_data)

——————————————————————————————— $ python3 test.py Traceback (most recent call last): File "/home/ec2-user/scripts/test.py", line 14, in response = runtime.invoke_endpoint(EndpointName='xxxx-endpoint-xxxx', File "/home/ec2-user/.local/lib/python3.9/site-packages/botocore/client.py", line 530, in _api_call return self._make_api_call(operation_name, kwargs) File "/home/ec2-user/.local/lib/python3.9/site-packages/botocore/client.py", line 964, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{ "code": 400, "type": "InternalServerException", "message": "Input payload must contain text_inputs key." }

[loveneetsingh7]

The error that you are referring to is the same error I encountered while using the AWS Sagemaker endpoint with Langchain Sagemaker Integration.

It is due to the absence of "text_inputs" as a parameter in the input data you are passing in the endpoint.

The input_data = {"department_id": 5, "job": "Editor", "rowid": 874778072614862849}, needs to have a text_inputs parameter as this might be a parameter that your Sagemaker model expects.

I myself used these methods as the part of content handler for the Sagemaker Endpoint : `content_handler:

def transform_input(self, prompt: str, model_kwargs: Dict) -> bytes:
    input_str = json.dumps({"text_inputs": prompt, **model_kwargs})
    return input_str.encode('utf-8')

def transform_output(self, output: bytes) -> str:
    response_json = json.loads(output.read().decode("utf-8"))
    return response_json['generated_texts'][0]

`

here, in the transform_input function:

the code: input_str = json.dumps({"text_inputs": prompt, **model_kwargs}), transforms the input with the addition of text_inputs as a parameter.

You might want to look into the Sagemaker notebook that you are using to check the expected format of the input it expects and change the data format accordingly. I was using an LLM from hugging face that expected the same - text_inputs as a parameter.

Please respond, if need any further assistance or if this doesn't work.

[dpranav12]

Thanks a lot for responding. But I do not understand how to change my code or where I do I pass the "input_data" (which has the data to be passed to sagemaker) i.e. How do I send the json data to sagemaker endpoint ? I am not using a sagemaker notebook - I am using a python script.

[loveneetsingh7]

Hi, I looked at a code example of using the Sagemaker endpoint, that is on this link: aws-samples/generative-ai-sagemaker

This is the blog post that might help you also: deploy-generative-ai-models-from-amazon-sagemaker-jumpstart-using-the-aws-cdk

This git repo has a lambda function in Python, at code/lambda/lambda_txt2nlu/txt2nlu.py this uses a way very similar to your code in the format of payload and has the text_inputs.

But it is because the model is an LLM, or is a model for text generation. You should look into it and see if it works. Also if this not works or you are not using a model that is for text generation then please provide that information.

[dosubot[bot]]

Hi, @loveneetsingh7! I'm Dosu, and I'm helping the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, the issue you raised was regarding errors in the documentation for integrating LLMs with the Sagemaker Endpoint. The issue has been resolved by making changes to the "ContentHandler" class. One user confirmed that the issue was resolved, while another user is still experiencing the same error and is seeking assistance on how to fix it. The original author provided guidance on how to fix the error by adding the "text_inputs" parameter to the input data and shared code examples and resources for further assistance.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your contribution to the LangChain repository!