Unable to run flan t xxl

[ysekiy]

I got a following error when I run streamlit run app.py flanxxl and python3 kendra_retriever_flan_xxl.py

Error

ValueError: Error raised by inference endpoint: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (500) from primary with message "Input payload must contain text_inputs key. Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/sagemaker_inference/transformer.py", line 128, in transform result = self._transform_fn(self._model, input_data, content_type, accept) File "/opt/ml/model/code/inference.py", line 389, in transform_fn payload = _update_payload(payload) File "/opt/ml/model/code/inference.py", line 267, in _update_payload _validate_payload(payload, batch_params) File "/opt/ml/model/code/inference.py", line 189, in _validate_payload assert constants.TEXT_INPUTS in payload, f"Input payload must contain {constants.TEXT_INPUTS} key." AssertionError: Input payload must contain text_inputs key. ". See https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/jumpstart-dft-hf-text2text-flan-t5-xxl in account <Account ID> for more information.

A reproducible test case or series of steps

git clone https://github.com/aws-samples/amazon-kendra-langchain-extensions.git
cd amazon-kendra-langchain-extensions
python -m venv venv
source ./venv/bin/activate
pip install .
pip install ".[samples]"
export AWS_REGION="<YOUR-AWS-REGION>"
export KENDRA_INDEX_ID="<YOUR-KENDRA-INDEX-ID>"
export FLAN_XXL_ENDPOINT="<YOUR-SAGEMAKER-ENDPOINT-FOR-FLAN-T-XXL>"
cd samples
streamlit run app.py flanxxl

The version of our code being used

https://github.com/aws-samples/amazon-kendra-langchain-extensions/commit/9bab4fafab1d460b2d06ec1e84514a805aca1f31

Any modifications you've made relevant to the bug

n/a

Anything unusual about your environment or deployment

n/a

[ysekiy]

I made pull request. https://github.com/aws-samples/amazon-kendra-langchain-extensions/pull/6

[dimwael]

Amazing work ! It worked for me !

[mrglover]

@ysekiy thank you so much for this fix! I was searching where that input param was but stumbled on your fix...

One quick thing... I believe you switched the region_name param for the SagemakerEndpoint from the region variable to the "us-east-1" string literal... I am hosting mine in us-west-2 so this broke when I tried to run your fix. I also dropped a comment on the pull request.

THANKS AGAIN!

[ysekiy]

https://github.com/aws-samples/amazon-kendra-langchain-extensions/pull/6#issuecomment-1584023047

[MithilShah]

Fixed in latest release