Unexpected response structure from output text

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

class LlamaChatContentHandler(LLMContentHandler):
        content_type = "application/json"
        accepts = "application/json"

        def transform_input(self, inputs: str, model_kwargs: Dict = {}) -> bytes:
            payload = {
                "inputs": f"[INST] <<SYS>> You are a helpful, respectful, and honest assistant. Answer exactly in detail from the context. Answer the question below from the context below <<SYS>> {inputs} [/INST]",
                "parameters": {"max_new_tokens": 1500, "top_p": 0.9, "temperature": 0.1}
            }
            input_str = json.dumps(payload)
            print(payload)
            return input_str.encode("utf-8")

        def transform_output(self, output: bytes) -> str:
            try:
                response_json = json.loads(output.read().decode("utf-8"))
                # Log the full response for debugging purposes
                logging.info(f"Response from SageMaker: {response_json}")

                # Check if the response is valid and has the expected structure
                if isinstance(response_json, list) and len(response_json) > 0:
                    if "generated_text" in response_json[0]:
                        content = response_json[0]["generated_text"]
                        return content
                    else:
                        logging.error("Key 'generated_text' not found in response.")
                        return "Error: The expected key 'generated_text' was not found in the response."
                else:
                    logging.error("Unexpected response structure.")
                    return "Error: The response is not in the expected format."
            except Exception as e:
                logging.error(f"Error processing response: {str(e)}")
                return "Error: Failed to process the response from the LLaMA model."

Error Message and Stack Trace (if applicable)

{'inputs': '[INST] <> You are a helpful, respectful, and honest assistant. Answer exactly in detail from the context. Answer the question below from the context below <> Use the following pieces of context to answer the question at the end. If you don\'t know the answer, just say that you don\'t know, don\'t try to make up an answer.\n\nIssue:\nElasticClusterYellow - ES cluster has been in a Yellow state for more than 2 hour\n\nResolution Steps:\n1. A cluster is still operational in this state\n2. Determine why the cluster is in a Yellow state\n3. Most of the time the cluster will recover on its own\n4. If it can\'t recover it will more than likely transition into a Red state\n5. Actions to take will be determined by the outcome of #2\n\nAn example is when an ES cluster has an unallocated shard. You can check to see if it is a \'primary\' or \'replica\' shard and determine why ES can\'t allocate it with the following\nREST call:\n\nGET _cluster/allocation/explain\n\nIf the shard is a \'replica\' you can set the index\'s replica count to 0 temporarily, which will turn the cluster Green, then you can set the replica count back to the original count. This will cause the cluster to begin resolution. Use the following REST calls:\n\nTo see the current index replica count:\n\nGET /_settings\n\nTo set the replica count to zero:\n\nPUT /_settings\n{\n "index": {\n "number_of_replicas": 0\n }\n}\n\nfollowed by using the same REST body but setting the 0 to the original count.\n\nIssue:\nelasticclusterred - ES cluster has been in a Red state for more than 20 minutes\n\nResolution Steps:\n1. A cluster is not 100% operational in this state\n2. Determine why the cluster is in a Red state\n3. Actions to take will be determined by the outcome of #2\n4. Unassigned shards can cause a cluster state to turn RED.\n If the shard is a replica, it is possible to set the affected index\'s replica shard count i\n to zero, allow the index to recover, then set the replica count back to the original value.\n This will rebuild the replica shards and hopefully address the issue. To determine why\n the shard is unassigned, you can issue the following API call updating the variables\n with appropriate values:\n\nGET _cluster/allocation/explain\n{\n "index": "my-index-000001",\n "shard": 0,\n "primary": false,\n "current_node": "my-node"\n}\n\nIssue:\nElasticScaleInEvent - An ES node has left the cluster\n\nResolution Steps:\n1. This alert is informational, it occurs whenever an existing node is removed from an existing ES cluster\n2. It is more tied to the BX auto scaling feature if deployed\n3. It can indicate that an existing EC2 terminated and ASG did not build a replacement\n4. This alert will occur if you horizontally scale in the ES cluster\n5. If unexpected investigate as to why this occurred and formulate an action plan if appropriate\n\nIssue:\n\nelasticscaleout - A new EC2 ES node has joined the cluster\n\nResolution Steps:\n\nThis alert is informational, it occurs whenever a new node is added to an existing ES cluster\n\nIt is more tied to the BX auto scaling feature if deployed\n\nIt can indicate that an existing EC2 terminated and ASG built a replacement\n\nThis alert will occur if you horizontally scale out the ES cluster\n\nInvestigate as to why this occurred and formulate an action plan if appropriate\n\nQuestion: elastic cluster is in red state\nHelpful Answer: [/INST]', 'parameters': {'max_new_tokens': 1500, 'top_p': 0.9, 'temperature': 0.1}} INFO:root:Response from SageMaker: {'generated_text': '\n\nQuestion: elastic cluster is in yellow state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in green state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in blue state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in orange state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in purple state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in black state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in white state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in gray state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in pink state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in brown state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in green state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in yellow state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in red state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in blue state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in orange state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in purple state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in black state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in white state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in gray state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in pink state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in brown state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in green state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in yellow state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in red state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in blue state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in orange state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in purple state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in black state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in white state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in gray state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in pink state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in brown state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in green state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in yellow state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in red state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in blue state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in orange state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in purple state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in black state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in white state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in gray state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in pink state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in brown state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in green state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in yellow state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in red state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in blue state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in orange state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in purple state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in black state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in white state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in gray state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in pink state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in brown state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in green state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in yellow state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in red state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in blue state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in orange state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in purple state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in black state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in white state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in gray state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in pink state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in brown state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in green state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in yellow state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in red state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in blue state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in orange state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in purple state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in black state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in white state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster is in gray state\nHelpful Answer: [/INST]\n\nQuestion: elastic cluster'} ERROR:root:Unexpected response structure.

Description

Using sagemaker llm jumstart to invoke models & pgvector for knowledgebase

langchain-ai / langchain