search

microsoft / vscode-copilot-release

Feedback on GitHub Copilot Chat UX in Visual Studio Code.
https://marketplace.visualstudio.com/items?itemName=GitHub.copilot-chat
Creative Commons Attribution 4.0 International
338 stars 36 forks source link

Copilot Struggles to See Context #719

Closed sdcanny closed 7 months ago

sdcanny commented 10 months ago

I've searched the bug reports and found a few similar issues, but I have not seen any of them resolved and they typically don't provide much information. I did not see anything pertaining to this issue in the Github documentation, either.

I was successfully using Copilot for about a month and thoroughly enjoying its features, but within last few days it's been incredibly hit or miss with actually being able to see Workspace data or highlighted code snippets. The Chat window almost never works and the Inline Chat has been degrading in reliability. I have tried opening individual documents, opening my workspace files, and more. It may have something to do with the length of the code in the file, but we're talking <500 lines and I figure that shouldn't be an issue.

Symptoms typically start with "As an AI," followed by some excuse about how I failed to provide the code. Using @ workspace will occasionally vary the output, but it seems to get worse as the project grows. Sometimes it will make remarks about the first 30 lines or so of my code, if I ask it to describe something or to evaluate my code.

Things I've tried:

I've tested both the regular and the Insider's Build, both have the same issues.

Steps to Reproduce:

  1. Launch either build of Visual Studio Code.
  2. Try to use Copilot.
datanetai commented 10 months ago

Yes, I noticed the same thing. I was highlighting my file today and asking it questions, and it was giving me random answers. In VSCode Insiders, it says 400 Bad Request. My code is around 300 lines and around 3.5k tokens. So they must have reduced the token limit recently. We need at least 8k, since I know their GPT-3.5 Turbo model is up to 8k.

hjain-spcg commented 9 months ago

they could not have tested this product update -- they pushed update without testing it one bit.

digitarald commented 9 months ago

@datanetai @sdcanny I assume both your issues are with chat panel, not inline chat. Correct?

datanetai commented 9 months ago

@datanetai @sdcanny I assume both your issues are with chat panel, not inline chat. Correct?

Yes, it used to happen when I highlighted a large file. However, now it's no longer happening in files with about 200-300 lines of code. It appears that the context size has increased.

digitarald commented 9 months ago

New releases do usually not decrease context window; but we iterate how we rank and slice the various available context parts. So detailed bug reports of cases that seemed to have worked before but stopped working would help most.

@sdcanny your logs don't show chat invocations but ghost-text completions. Can you clarify and reproduce on 0.12?

JamesAllerlei commented 7 months ago

I’m having the same issue. I’ve even tried a full clean reinstall if vs code, python and copilot but cant get it to read my code snippet in context.

digitarald commented 7 months ago

Most issues here seem large file related.

digitarald commented 7 months ago

Duplicate of #1083

JamesAllerlei commented 7 months ago

For me it happens in any script with over 80 lines. Also, relatedly, Copilot misreads the line numbers so is often looking at the wrong thing. Eg if i ask what is on line 41 it says the wrong thing. I assume that is also why it keeps getting the context of the selected code wrong.