sourcegraph / cody

AI that knows your entire codebase
https://cody.dev
Apache License 2.0
2.24k stars 213 forks source link

Chat: handle transcript length exceeding context window #4674

Closed abeatrix closed 6 days ago

abeatrix commented 1 week ago

CLOSE https://linear.app/sourcegraph/issue/CODY-1608/chat-token-usage-must-be-updated-before-context CLOSE https://github.com/sourcegraph/cody/issues/4195

This ensures the user is aware when the chat input has exceeded the allowed token limit, and the error stops the prompt building process from processing so that it would move to the step where we add context without transcript.

Test plan

  1. Build Cody from this branch
  2. Pick a model with a smaller context window, e.g. GPT-3.5 Turbo
  3. Copy the content from SimpleChatPanelProvider.ts which exceed the context window for the smaller models
  4. Submit the question and expect the error about input exceed the context window
image
abeatrix commented 6 days ago

Anything we can do here?

I was thinking we could truncate the input, but we might truncate the important part of the input without users knowing. I will create a linear issue and get the product for input on what's the best way to handle this case?

Edit: Updated the error message and set it as a transcript message to provide steps for users to unblock themselves:

image