search

getappmap / appmap-js

Client libraries for AppMap
48 stars 17 forks source link

Token overflow can still occur #2091

Open kgilpin opened 4 weeks ago

kgilpin commented 4 weeks ago

An example: https://github.com/getappmap/navie-benchmark/actions/runs/11522951430/job/32080178265#step:7:2114

Another example: https://github.com/getappmap/navie-benchmark/actions/runs/11522951430/job/32080178265

Search for Context length exceeded. Reducing token count and retrying (direct linking is not working for some reason).

Error: Failed to complete after 5 attempt(s): 400 {"type":"error","error":{"type":"invalid_request_error","message":"prompt is too long: 200679 tokens > 199999 maximum"}}
    at AnthropicCompletionService.complete (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/packages/navie/dist/services/anthropic-completion-service.js:186:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async ExplainCommand.execute (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/packages/navie/dist/commands/explain-command.js:110:26)
    at async Navie.execute (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/packages/navie/dist/navie.js:162:30)
    at async LocalNavie.ask (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/packages/cli/built/rpc/explain/navie/navie-local.js:135:85)
    at async Explain.explain (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/packages/cli/built/rpc/explain/explain.js:77:9) {
  [cause]: BadRequestError: 400 {"type":"error","error":{"type":"invalid_request_error","message":"prompt is too long: 200679 tokens > 199999 maximum"}}
      at APIError.generate (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/node_modules/@anthropic-ai/sdk/error.js:41:20)
      at Anthropic.makeStatusError (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/node_modules/@anthropic-ai/sdk/core.js:275:33)
      at Anthropic.makeRequest (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/node_modules/@anthropic-ai/sdk/core.js:318:30)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async RetryOperation._fn (/home/runner/work/navie-benchmark/navie-benchmark/submodules/appmap-js/node_modules/p-retry/index.js:50:12) {
    status: 400,
    headers: {
      'cf-cache-status': 'DYNAMIC',
      'cf-ray': '8d84554d4bae2958-ORD',
      connection: 'keep-alive',
      'content-length': '120',
      'content-type': 'application/json',
      date: 'Fri, 25 Oct 2024 18:40:07 GMT',
      'request-id': 'req_01XnSSx5uYx6kPHh4FUrMXVw',
      server: 'cloudflare',
      via: '1.1 google',
      'x-robots-tag': 'none',
      'x-should-retry': 'false'
    },
    request_id: 'req_01XnSSx5uYx6kPHh4FUrMXVw',
    error: { type: 'error', error: [Object] },
    attemptNumber: 1,
    retriesLeft: 6
  }
}
dustinbyrne commented 3 weeks ago

In one of your examples, the API is failing because it's overloaded. I don't believe there's any way around this other than retrying, but that didn't seem to work in this case.

{"type":"error","error":{"details":null,"type":"overloaded_error","message":"Overloaded"}   }
dustinbyrne commented 3 weeks ago

In other cases, it seems likely that it's the system prompt which is overflowing the token limit. The current implementation will only truncate user messages.