Retry on Sonnet error "Output blocked by content filtering policy"

[kgilpin]

Error example:

Handling exception: Error: Failed to complete: SSE Error: {"type":"error","error":{"details":null,"type":"invalid_request_error","message":"Output blocked by content filtering policy"}             }
    at AnthropicCompletionService.complete (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/packages/navie/dist/services/anthropic-completion-service.js:167:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async ExplainCommand.execute (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/packages/navie/dist/commands/explain-command.js:108:26)
    at async Navie.execute (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/packages/navie/dist/navie.js:161:30)
    at async LocalNavie.ask (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/packages/cli/built/rpc/explain/navie/navie-local.js:135:85)
    at async Explain.explain (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/packages/cli/built/rpc/explain/explain.js:78:9) {
  [cause]: APIConnectionError: {"type":"error","error":{"details":null,"type":"invalid_request_error","message":"Output blocked by content filtering policy"}             }
      at APIError.generate (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/node_modules/@anthropic-ai/sdk/error.js:37:20)
      at Stream.iterator (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/node_modules/@anthropic-ai/sdk/streaming.js:52:48)
      ... 3 lines matching cause stack trace ...
      at async Object.pull (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/node_modules/@langchain/core/dist/utils/stream.cjs:98:41) {
    status: undefined,
    headers: undefined,
    request_id: undefined,
    error: undefined,
    cause: Error: SSE Error: {"type":"error","error":{"details":null,"type":"invalid_request_error","message":"Output blocked by content filtering policy"}             }
        at castToError (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/node_modules/@anthropic-ai/sdk/core.js:709:12)
        at APIError.generate (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/node_modules/@anthropic-ai/sdk/error.js:37:84)
        at Stream.iterator (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/node_modules/@anthropic-ai/sdk/streaming.js:52:48)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async ChatAnthropic._streamResponseChunks (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/node_modules/@langchain/anthropic/dist/chat_models.cjs:677:26)
        at async ChatAnthropic._streamIterator (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/node_modules/@langchain/core/dist/language_models/chat_models.cjs:92:34)
        at async Object.pull (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/node_modules/@langchain/core/dist/utils/stream.cjs:98:41)
  }
}
Stack trace: Error: Failed to complete: SSE Error: {"type":"error","error":{"details":null,"type":"invalid_request_error","message":"Output blocked by content filtering policy"}             }
    at AnthropicCompletionService.complete (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/packages/navie/dist/services/anthropic-completion-service.js:167:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async ExplainCommand.execute (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/packages/navie/dist/commands/explain-command.js:108:26)
    at async Navie.execute (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/packages/navie/dist/navie.js:161:30)
    at async LocalNavie.ask (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/packages/cli/built/rpc/explain/navie/navie-local.js:135:85)
    at async Explain.explain (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/packages/cli/built/rpc/explain/explain.js:78:9)
Cause: Error: {"type":"error","error":{"details":null,"type":"invalid_request_error","message":"Output blocked by content filtering policy"}             }

https://support.anthropic.com/en/articles/9205721-why-am-i-receiving-an-output-blocked-by-content-filtering-policy-error

--

Plan changes for the navie package only. Do not propose changes to the client package.

[github-actions[bot]]

Title

Implement Retry Mechanism for Sonnet Error "Output blocked by content filtering policy"

Problem

When using the AnthropicCompletionService, an error "Output blocked by content filtering policy" is sometimes encountered, causing failures in running commands without retries.

Analysis

The error "Output blocked by content filtering policy" indicates that the content has been flagged by a filtering policy, halting the operation. To make the system more robust, a retry mechanism should be implemented when this specific error occurs. The retry mechanism should attempt to re-execute the operation up to a certain limit before finally failing, ensuring resilience against transient filtering errors.

Proposed Changes

1. packages/navie/src/services/anthropic-completion-service.ts:
   Introduce a retry mechanism when this specific error is encountered inside the complete function.

   • Modify the complete method to catch the specific "Output blocked by content filtering policy" error.
   • Implement a retry strategy, possibly utilizing an exponential backoff or a fixed number of retries before failing permanently.

2. packages/navie/src/lib/errors.ts:
   Define a new custom error class for filtering policy errors to encapsulate the specific error condition.

3. packages/navie/src/commands/explain-command.ts:
   Ensure that the execute function in ExplainCommand is compatible with the retry mechanism introduced in the AnthropicCompletionService.

4. packages/navie/src/navie.ts:
   Update the logic for the execute function to handle retries properly when the AnthropicCompletionService throws the specific custom error.

Here are the detailed changes:

1. packages/navie/src/services/anthropic-completion-service.ts:

   • Check for the specific error message within the complete method.
   • Wrap the completion logic in a retry loop using a configurable retry policy.

2. packages/navie/src/lib/errors.ts:

   • Add a new custom error class FilteringPolicyError which will be used to identify and handle the filtering policy-related errors.

3. packages/navie/src/commands/explain-command.ts:

   • Modify the execute function to retry invoking the AnthropicCompletionService.complete method when the new FilteringPolicyError is raised.

4. packages/navie/src/navie.ts:

   • Ensure seamless integration with the new retry logic in AnthropicCompletionService by updating the execute function to properly capture and respond to retried operations.

By implementing these changes, the codebase will effectively handle the "Output blocked by content filtering policy" errors by retrying the intended operation, providing a more resilient and reliable system.

[kgilpin]

I can't consider this a high priority since these errors tend to recur when we re-run instances.