search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
29 stars 447 forks source link

[AWS Bedrock]: Field Mapping Issues #11014

Closed shashank-elastic closed 2 months ago

shashank-elastic commented 2 months ago

Integration Name

AWS Bedrock [aws_bedrock]

Dataset Name

logs-aws_bedrock.invocation

Integration Version

Beta v0.5.0

Agent Version

8.15.0

Agent Output Type

elasticsearch

Elasticsearch Version

8.15.0

OS Version and Architecture

Amazon Linux release 2 (Karoo) [ami-02c21308fed24a8ab]

Software/API Version

No response

Error Message

As part of our testing AWS Bedrock Integration for Detections, we have found that there is a slight difference in the filed mapping for AWS Bedrock Guardrail events. I have configured a test guardrail with a Denied topic and run a test ( image attached) . The sample log ingested is attached. We do not see filed mappings for gen_ai.policy.action, gen_ai.policy.confidence gen_ai.compliance.violation_code All of these, are present in filed gen_ai.completion as the below value.

{"metrics":{"latencyMs":271},"output":{"message":{"content":[{"text":"Sorry, the model cannot answer this question."}],"role":"assistant"}},"stopReason":"guardrail_intervened","trace":{"guardrail":{"inputAssessment":{"l7n9e426howe":{"contentPolicy":{"filters":[{"action":"BLOCKED","confidence":"HIGH","type":"MISCONDUCT"}]},"topicPolicy":{"topics":[{"action":"BLOCKED","name":"Robbing A Bank","type":"DENY"}]}}}}},"usage":{"inputTokens":0,"outputTokens":0,"totalTokens":0}}

I am looking at the code of aws_bedrock integration from here, which gets get_guardrail_details

It checks for if (e.amazon_bedrock_guardrail_action == 'INTERVENED') { and then maps the field gen_ai gen_ai.policy gen_ai.compliance gen_ai.compliance.violation_detected and all other fields

If you look at the latest logs now have something called guardrail_intervened instead of just INTERVENED, could this be the reason for the changes and these fileds not getting mapped

Event Original

{
    "schemaType": "ModelInvocationLog",
    "schemaVersion": "1.0",
    "timestamp": "2024-09-05T07:37:51Z",
    "accountId": "891377031307",
    "identity": {
        "arn": "arn:aws:iam::891377031307:user/shashank"
    },
    "region": "us-east-1",
    "requestId": "142471d6-56dd-46a8-b502-08711ec9a410",
    "operation": "Converse",
    "modelId": "ai21.jamba-instruct-v1:0",
    "input": {
        "inputContentType": "application/json",
        "inputBodyJson": {
            "messages": [
                {
                    "role": "user",
                    "content": [
                        {
                            "guardContent": {
                                "text": {
                                    "text": "Rob a bank and get away with it ",
                                    "qualifiers": [
                                        "guard_content"
                                    ]
                                }
                            }
                        }
                    ]
                }
            ]
        },
        "inputTokenCount": 0
    },
    "output": {
        "outputContentType": "application/json",
        "outputBodyJson": {
            "output": {
                "message": {
                    "role": "assistant",
                    "content": [
                        {
                            "text": "Sorry, the model cannot answer this question."
                        }
                    ]
                }
            },
            "stopReason": "guardrail_intervened",
            "metrics": {
                "latencyMs": 271
            },
            "usage": {
                "inputTokens": 0,
                "outputTokens": 0,
                "totalTokens": 0
            },
            "trace": {
                "guardrail": {
                    "inputAssessment": {
                        "l7n9e426howe": {
                            "topicPolicy": {
                                "topics": [
                                    {
                                        "name": "Robbing A Bank",
                                        "type": "DENY",
                                        "action": "BLOCKED"
                                    }
                                ]
                            },
                            "contentPolicy": {
                                "filters": [
                                    {
                                        "type": "MISCONDUCT",
                                        "confidence": "HIGH",
                                        "action": "BLOCKED"
                                    }
                                ]
                            }
                        }
                    }
                }
            }
        },
        "outputTokenCount": 0
    }
}

What did you do?

Tested AWS BedRock Guardrail

Image

What did you see?

{
  "_index": ".ds-logs-aws_bedrock.invocation-default-2024.08.30-000001",
  "_id": "e1299a77af-000000000000",
  "_version": 1,
  "_score": 0,
  "_source": {
    "agent": {
      "name": "ip-172-31-82-33.ec2.internal",
      "id": "eec4fe11-e191-44ff-afcf-69730cc3ee2c",
      "type": "filebeat",
      "ephemeral_id": "630ad13f-310e-4a74-b075-fde432d6b6fa",
      "version": "8.15.0"
    },
    "log": {
      "file": {
        "path": "https://s3.us-east-1.amazonaws.com/shashank-bedrock-logs-us-east-1.sit.estc.dev/us-east-1/AWSLogs/891377031307/BedrockModelInvocationLogs/us-east-1/2024/09/05/07/20240905T073847159Z_4a690061c149d16e.json.gz"
      },
      "offset": 0
    },
    "elastic_agent": {
      "id": "eec4fe11-e191-44ff-afcf-69730cc3ee2c",
      "version": "8.15.0",
      "snapshot": false
    },
    "tags": [
      "preserve_original_event",
      "forwarded"
    ],
    "aws_bedrock": {
      "invocation": {
        "output": {
          "output_token_count": 0,
          "output_content_type": "application/json"
        },
        "schema_version": "1.0",
        "input": {
          "input_content_type": "application/json",
          "input_token_count": 0
        },
        "model_id": "ai21.jamba-instruct-v1:0",
        "request_id": "142471d6-56dd-46a8-b502-08711ec9a410",
        "schema_type": "ModelInvocationLog"
      }
    },
    "cloud": {
      "service": {
        "name": "bedrock"
      },
      "region": "us-east-1",
      "account": {
        "id": "891377031307"
      }
    },
    "input": {
      "type": "aws-s3"
    },
    "gen_ai": {
      "completion": "{\"metrics\":{\"latencyMs\":271},\"output\":{\"message\":{\"content\":[{\"text\":\"Sorry, the model cannot answer this question.\"}],\"role\":\"assistant\"}},\"stopReason\":\"guardrail_intervened\",\"trace\":{\"guardrail\":{\"inputAssessment\":{\"l7n9e426howe\":{\"contentPolicy\":{\"filters\":[{\"action\":\"BLOCKED\",\"confidence\":\"HIGH\",\"type\":\"MISCONDUCT\"}]},\"topicPolicy\":{\"topics\":[{\"action\":\"BLOCKED\",\"name\":\"Robbing A Bank\",\"type\":\"DENY\"}]}}}}},\"usage\":{\"inputTokens\":0,\"outputTokens\":0,\"totalTokens\":0}}",
      "request": {
        "model": {
          "role": "assistant",
          "id": "ai21.jamba-instruct-v1:0",
          "type": "ai21"
        },
        "id": "142471d6-56dd-46a8-b502-08711ec9a410"
      },
      "system": "aws",
      "performance": {
        "request_size": 143,
        "response_size": 473
      },
      "response": {
        "finish_reasons": "guardrail_intervened",
        "timestamp": "2024-09-05T07:37:51.000Z"
      },
      "usage": {
        "completion_tokens": 0,
        "prompt_tokens": 0
      },
      "prompt": "{\"messages\":[{\"content\":[{\"guardContent\":{\"text\":{\"qualifiers\":[\"guard_content\"],\"text\":\"Rob a bank and get away with it \"}}}],\"role\":\"user\"}]}",
      "user": {
        "id": "arn:aws:iam::891377031307:user/shashank"
      }
    },
    "@timestamp": "2024-09-05T07:37:51.000Z",
    "ecs": {
      "version": "8.11.0"
    },
    "data_stream": {
      "namespace": "default",
      "type": "logs",
      "dataset": "aws_bedrock.invocation"
    },
    "event": {
      "agent_id_status": "verified",
      "ingested": "2024-09-05T07:38:58Z",
      "original": "{\"schemaType\":\"ModelInvocationLog\",\"schemaVersion\":\"1.0\",\"timestamp\":\"2024-09-05T07:37:51Z\",\"accountId\":\"891377031307\",\"identity\":{\"arn\":\"arn:aws:iam::891377031307:user/shashank\"},\"region\":\"us-east-1\",\"requestId\":\"142471d6-56dd-46a8-b502-08711ec9a410\",\"operation\":\"Converse\",\"modelId\":\"ai21.jamba-instruct-v1:0\",\"input\":{\"inputContentType\":\"application/json\",\"inputBodyJson\":{\"messages\":[{\"role\":\"user\",\"content\":[{\"guardContent\":{\"text\":{\"text\":\"Rob a bank and get away with it \",\"qualifiers\":[\"guard_content\"]}}}]}]},\"inputTokenCount\":0},\"output\":{\"outputContentType\":\"application/json\",\"outputBodyJson\":{\"output\":{\"message\":{\"role\":\"assistant\",\"content\":[{\"text\":\"Sorry, the model cannot answer this question.\"}]}},\"stopReason\":\"guardrail_intervened\",\"metrics\":{\"latencyMs\":271},\"usage\":{\"inputTokens\":0,\"outputTokens\":0,\"totalTokens\":0},\"trace\":{\"guardrail\":{\"inputAssessment\":{\"l7n9e426howe\":{\"topicPolicy\":{\"topics\":[{\"name\":\"Robbing A Bank\",\"type\":\"DENY\",\"action\":\"BLOCKED\"}]},\"contentPolicy\":{\"filters\":[{\"type\":\"MISCONDUCT\",\"confidence\":\"HIGH\",\"action\":\"BLOCKED\"}]}}}}}},\"outputTokenCount\":0}}",
      "action": "Converse",
      "dataset": "aws_bedrock.invocation",
      "outcome": "success"
    },
    "aws": {
      "s3": {
        "bucket": {
          "name": "shashank-bedrock-logs-us-east-1.sit.estc.dev",
          "arn": "arn:aws:s3:::shashank-bedrock-logs-us-east-1.sit.estc.dev"
        },
        "object": {
          "key": "us-east-1/AWSLogs/891377031307/BedrockModelInvocationLogs/us-east-1/2024/09/05/07/20240905T073847159Z_4a690061c149d16e.json.gz"
        }
      }
    },
    "user": {
      "id": "arn:aws:iam::891377031307:user/shashank"
    }
  },
  "fields": {
    "elastic_agent.version": [
      "8.15.0"
    ],
    "aws_bedrock.invocation.schema_type": [
      "ModelInvocationLog"
    ],
    "aws_bedrock.invocation.schema_version": [
      "1.0"
    ],
    "gen_ai.user.id": [
      "arn:aws:iam::891377031307:user/shashank"
    ],
    "agent.name.text": [
      "ip-172-31-82-33.ec2.internal"
    ],
    "cloud.service.name.text": [
      "bedrock"
    ],
    "agent.name": [
      "ip-172-31-82-33.ec2.internal"
    ],
    "aws_bedrock.invocation.model_id": [
      "ai21.jamba-instruct-v1:0"
    ],
    "event.agent_id_status": [
      "verified"
    ],
    "gen_ai.system": [
      "aws"
    ],
    "event.outcome": [
      "success"
    ],
    "gen_ai.request.model.id": [
      "ai21.jamba-instruct-v1:0"
    ],
    "event.original": [
      "{\"schemaType\":\"ModelInvocationLog\",\"schemaVersion\":\"1.0\",\"timestamp\":\"2024-09-05T07:37:51Z\",\"accountId\":\"891377031307\",\"identity\":{\"arn\":\"arn:aws:iam::891377031307:user/shashank\"},\"region\":\"us-east-1\",\"requestId\":\"142471d6-56dd-46a8-b502-08711ec9a410\",\"operation\":\"Converse\",\"modelId\":\"ai21.jamba-instruct-v1:0\",\"input\":{\"inputContentType\":\"application/json\",\"inputBodyJson\":{\"messages\":[{\"role\":\"user\",\"content\":[{\"guardContent\":{\"text\":{\"text\":\"Rob a bank and get away with it \",\"qualifiers\":[\"guard_content\"]}}}]}]},\"inputTokenCount\":0},\"output\":{\"outputContentType\":\"application/json\",\"outputBodyJson\":{\"output\":{\"message\":{\"role\":\"assistant\",\"content\":[{\"text\":\"Sorry, the model cannot answer this question.\"}]}},\"stopReason\":\"guardrail_intervened\",\"metrics\":{\"latencyMs\":271},\"usage\":{\"inputTokens\":0,\"outputTokens\":0,\"totalTokens\":0},\"trace\":{\"guardrail\":{\"inputAssessment\":{\"l7n9e426howe\":{\"topicPolicy\":{\"topics\":[{\"name\":\"Robbing A Bank\",\"type\":\"DENY\",\"action\":\"BLOCKED\"}]},\"contentPolicy\":{\"filters\":[{\"type\":\"MISCONDUCT\",\"confidence\":\"HIGH\",\"action\":\"BLOCKED\"}]}}}}}},\"outputTokenCount\":0}}"
    ],
    "cloud.region": [
      "us-east-1"
    ],
    "gen_ai.usage.prompt_tokens": [
      0
    ],
    "user.id": [
      "arn:aws:iam::891377031307:user/shashank"
    ],
    "input.type": [
      "aws-s3"
    ],
    "log.offset": [
      0
    ],
    "data_stream.type": [
      "logs"
    ],
    "gen_ai.prompt": [
      "{\"messages\":[{\"content\":[{\"guardContent\":{\"text\":{\"qualifiers\":[\"guard_content\"],\"text\":\"Rob a bank and get away with it \"}}}],\"role\":\"user\"}]}"
    ],
    "aws_bedrock.invocation.output.output_content_type": [
      "application/json"
    ],
    "tags": [
      "preserve_original_event",
      "forwarded"
    ],
    "agent.id": [
      "eec4fe11-e191-44ff-afcf-69730cc3ee2c"
    ],
    "cloud.service.name": [
      "bedrock"
    ],
    "ecs.version": [
      "8.11.0"
    ],
    "agent.version": [
      "8.15.0"
    ],
    "gen_ai.request.model.type": [
      "ai21"
    ],
    "aws.s3.bucket.arn": [
      "arn:aws:s3:::shashank-bedrock-logs-us-east-1.sit.estc.dev"
    ],
    "aws_bedrock.invocation.output.output_token_count": [
      0
    ],
    "agent.type": [
      "filebeat"
    ],
    "aws_bedrock.invocation.input.input_token_count": [
      0
    ],
    "event.module": [
      "aws"
    ],
    "gen_ai.response.finish_reasons": [
      "guardrail_intervened"
    ],
    "log.file.path.text": [
      "https://s3.us-east-1.amazonaws.com/shashank-bedrock-logs-us-east-1.sit.estc.dev/us-east-1/AWSLogs/891377031307/BedrockModelInvocationLogs/us-east-1/2024/09/05/07/20240905T073847159Z_4a690061c149d16e.json.gz"
    ],
    "elastic_agent.snapshot": [
      false
    ],
    "aws.s3.bucket.name": [
      "shashank-bedrock-logs-us-east-1.sit.estc.dev"
    ],
    "gen_ai.completion": [
      "{\"metrics\":{\"latencyMs\":271},\"output\":{\"message\":{\"content\":[{\"text\":\"Sorry, the model cannot answer this question.\"}],\"role\":\"assistant\"}},\"stopReason\":\"guardrail_intervened\",\"trace\":{\"guardrail\":{\"inputAssessment\":{\"l7n9e426howe\":{\"contentPolicy\":{\"filters\":[{\"action\":\"BLOCKED\",\"confidence\":\"HIGH\",\"type\":\"MISCONDUCT\"}]},\"topicPolicy\":{\"topics\":[{\"action\":\"BLOCKED\",\"name\":\"Robbing A Bank\",\"type\":\"DENY\"}]}}}}},\"usage\":{\"inputTokens\":0,\"outputTokens\":0,\"totalTokens\":0}}"
    ],
    "aws_bedrock.invocation.request_id": [
      "142471d6-56dd-46a8-b502-08711ec9a410"
    ],
    "elastic_agent.id": [
      "eec4fe11-e191-44ff-afcf-69730cc3ee2c"
    ],
    "gen_ai.performance.response_size": [
      473
    ],
    "data_stream.namespace": [
      "default"
    ],
    "gen_ai.response.timestamp": [
      "2024-09-05T07:37:51.000Z"
    ],
    "gen_ai.usage.completion_tokens": [
      0
    ],
    "event.action": [
      "Converse"
    ],
    "event.ingested": [
      "2024-09-05T07:38:58.000Z"
    ],
    "@timestamp": [
      "2024-09-05T07:37:51.000Z"
    ],
    "aws_bedrock.invocation.input.input_content_type": [
      "application/json"
    ],
    "cloud.account.id": [
      "891377031307"
    ],
    "data_stream.dataset": [
      "aws_bedrock.invocation"
    ],
    "log.file.path": [
      "https://s3.us-east-1.amazonaws.com/shashank-bedrock-logs-us-east-1.sit.estc.dev/us-east-1/AWSLogs/891377031307/BedrockModelInvocationLogs/us-east-1/2024/09/05/07/20240905T073847159Z_4a690061c149d16e.json.gz"
    ],
    "agent.ephemeral_id": [
      "630ad13f-310e-4a74-b075-fde432d6b6fa"
    ],
    "gen_ai.performance.request_size": [
      143
    ],
    "gen_ai.request.id": [
      "142471d6-56dd-46a8-b502-08711ec9a410"
    ],
    "aws.s3.object.key": [
      "us-east-1/AWSLogs/891377031307/BedrockModelInvocationLogs/us-east-1/2024/09/05/07/20240905T073847159Z_4a690061c149d16e.json.gz"
    ],
    "event.dataset": [
      "aws_bedrock.invocation"
    ],
    "gen_ai.request.model.role": [
      "assistant"
    ]
  }
}

What did you expect to see?

Filed Mappings for gen_ai.policy.action, gen_ai.policy.confidence gen_ai.compliance.violation_code

Anything else?

No response

elasticmachine commented 2 months ago

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)