[Bug]: All prompts and responses are blocked if a bedrock guardrail is hit.

[ravi-katiyar]

What happened?

I am using LiteLLM to integrate bedrock with Libre chat. I have configured some guardrails in bedrock and they are working as expected. However once the guardrail is hit all responses are denied. Even valid prompts. This my litellm config

model_list:
  # https://litellm.vercel.app/docs/proxy/quick_start

  # AWS Bedrock - Anthropic
  - model_name: llama3
    litellm_params:
      model: bedrock/meta.llama3-1-405b-instruct-v1:0
      aws_region_name: us-west-2
      aws_access_key_id: 'os.environ/AWS_ACCESS_KEY_ID'
      aws_secret_access_key: 'os.environ/AWS_SECRET_ACCESS_KEY'
      guardrailConfig: {
        "guardrailIdentifier": "abcdahhaduo", # The identifier (ID) for the guardrail.
        "guardrailVersion": "DRAFT",           # The version of the guardrail.
        "trace": "enabled",                   # The trace behavior for the guardrail. Can either be "disabled" or "enabled"
      }

# NOTE: It may be a good idea to comment out "success_callback", "cache", "cache_params" (both lines under) when you first start until this works!
litellm_settings:
  drop_params: true

#general_settings:
#  master_key: sk_live_SetToRandomValue

In the screenshot below the first prompt hits the guardrail and I see the expected response. But the subsequent prompt should not have been replied with guardrail default prompts.

Relevant log output

No response

Twitter / LinkedIn details

No response

[krrishdholakia]

Hi @ravi-katiyar isn't this bedrock rejecting each request?

• model_name: llama3 litellm_params: model: bedrock/meta.llama3-1-405b-instruct-v1:0 aws_region_name: us-west-2 aws_access_key_id: 'os.environ/AWS_ACCESS_KEY_ID' aws_secret_access_key: 'os.environ/AWS_SECRET_ACCESS_KEY' guardrailConfig: { "guardrailIdentifier": "abcdahhaduo", # The identifier (ID) for the guardrail. "guardrailVersion": "DRAFT", # The version of the guardrail. "trace": "enabled", # The trace behavior for the guardrail. Can either be "disabled" or "enabled" }

What do your logs show?

[ravi-katiyar]

yes, looks like that's bedrock issue. Let me check on that.

[ravi-katiyar]

here are model invocation logs

{
  "schemaType": "ModelInvocationLog",
  "schemaVersion": "1.0",
  "timestamp": "2024-08-27T10:11:19Z",
  "accountId": ".....",
  "identity": {
    "arn": "arn:aws:iam::.....:user/....."
  },
  "region": "us-west-2",
  "requestId": "e57c758c-f8d0-4031-91c2-765a1238dba5",
  "operation": "ConverseStream",
  "modelId": "meta.llama3-1-405b-instruct-v1:0",
  "input": {
    "inputContentType": "application/json",
    "inputBodyJson": {
      "messages": [
        {
          "role": "user",
          "content": [
            {
              "text": "what does C2FO do ?"
            }
          ]
        },
        {
          "role": "assistant",
          "content": [
            {
              "text": "C2FO is a financial technology company that provides a platform for businesses to optimize their cash flow by offering early payment options to their suppliers. Here's a breakdown of what they do:\n\n**The Problem:** Many businesses face cash flow challenges due to slow payment terms, which can lead to delayed payments to suppliers, strained relationships, and missed opportunities.\n\n**C2FO's Solution:** C2FO's platform allows businesses (buyers) to offer their suppliers (sellers) the option to receive early payment on outstanding invoices. This is done through a marketplace where suppliers can choose to accelerate payment on specific invoices at a discount.\n\n**How it works:**\n\n1. **Buyer-Supplier Relationship:** A business (buyer) partners with C2FO and invites their suppliers to join the platform.\n2. **Invoice Upload:** Suppliers upload their outstanding invoices to the C2FO platform.\n3. **Early Payment Options:** Buyers offer early payment options to suppliers through the platform, specifyingSorry, the model cannot answer this question."
            }
          ]
        },
        {
          "role": "user",
          "content": [
            {
              "text": "what is the capital france ?"
            }
          ]
        },
        {
          "role": "assistant",
          "content": [
            {
              "text": "The capital of France is Paris."
            }
          ]
        },
        {
          "role": "user",
          "content": [
            {
              "text": "how does pricing work ?"
            }
          ]
        },
        {
          "role": "assistant",
          "content": [
            {
              "text": "Pricing is a complex topic, and it can vary depending on the context, industry, and specific product or service. However, I'll provide a general overview of pricing strategies and how they work:\n\n**Pricing Objectives:**\n\n1. **Revenue Maximization:** Set prices to maximize revenue, taking into account demand, competition, and costs.\n2. **Profit Maximization:** Set prices to maximize profit, considering revenue, costs, and target profit margins.\n3. **Market Penetration:** Set low prices to attract new customers, gain market share, and build brand awareness.\n4. **Value-Based Pricing:** Set prices based on the perceived value of the product or service to the customer.\n\n**Pricing Strategies:**\n\n1. **Cost-Plus Pricing:** Add a markup to the cost of production or procurement to determine the selling price.\n2. **Competitive Pricing:** Set prices relative to competitors, taking into account market conditions and target market share.\n3. **Value-Based Pricing"
            }
          ]
        },
        {
          "role": "user",
          "content": [
            {
              "text": "how does C2FO pricing work ?"
            }
          ]
        },
        {
          "role": "assistant",
          "content": [
            {
              "text": "C2FO's pricing model is designed to facilitate early payments between buyers and suppliers. Here's an overview of how their pricing works:\n\n**Discounts:** Suppliers offer discounts to buyers in exchange for early payment. The discount is a percentage of the invoice amount and is typically set by the supplier.\n\n**Dynamic Discounting:** C2FO's platform uses a dynamic discounting model, which means that the discount rate changes based on the number of days remaining until the invoice due date. The earlier the payment, the larger the discount.\n\n**Pricing Structure:** C2FO's pricing structure is typically based on a percentage of the invoice amount, and it's usually a tiered structure. For example:\n\n* 1-10 days early payment: 1.5% discount\n* 11-20 days early payment: 1.2% discount\n* 21-30 days early payment: 1.0% discount\n\n**Revenue Streams:** C2FO generates revenue through a"
            }
          ]
        },
        {
          "role": "user",
          "content": [
            {
              "text": "what is the capital of france ?"
            }
          ]
        }
      ],
      "system": [],
      "inferenceConfig": {},
      "additionalModelRequestFields": {}
    },
    "inputTokenCount": 0
  },
  "output": {
    "outputContentType": "application/json",
    "outputTokenCount": 0
  }
}

[ishaan-jaff]

@ravi-katiyar - so to confirm this issue is not on LiteLLM side correct ? It looks like it's triggered because you're sending the message history in subsequent requests

[krrishdholakia]

closing as this is not a litellm issue.