☂️ Copilot Chat gets "confused" - which context to follow while responding -> Short-term memory loss?

[Pranav-yadav]

• VS Code Version: 1.79.0
• OS Version: Win 11 x64
• Logs: NA

Summary

Although, we know that Copilot Chat is still under development, there are critical bugs (or design issues of the underlying model) that need to be taken care of, the prominent one being: "Copilot Chat gets confused - which context to follow while responding -> Short-term memory loss?/Lack of Conversational Skills?"

This is an _umbrella issue_ consisting of all such bugs/issues as follows _along with suggestions :)_ > **NOTE**: I'll keep updating the below table whenever I encounter any bugs while interacting with Copilot Chat :)

#	Bug/Issue	Description	Suggestion
1.	Model gets confused with which "context" to follow/use while responding	See screenshots 1.x below	Well this looks like more of a design issue but, I'm sure that; more fine-tuning can help improve this critical bug/issue with the underlying model :)
...n.	To be discovered (TBD)	TBA	TBA

Steps to Reproduce/Chat history {Click to see the screenshots}:

1. Creating and developing the context 1.1 (about Kotlin programming language):  1.2 Continue developing the context with relevant questions  1.3 More context development relevant questions, here as we can see the model has theoretically started to get confused; losing the hold of context. Although it answers the questions generally, it fails to mention anything about "Kotlin" ;) _GPT-4 will mention Kotlin while answering this question, even though it's a general-purpose LLM and _not specifically_ trained/fine-tuned to answer only the questions related to code.  1.4 With the below question, _it is clear that the model has **failed to consider/completely lost the current context**, by responding with a **"context unaware"** answer_ -> It has lost the memories :( -> Short-term memory loss :)  1.5 Now **I try to provide a hint** (ask the same question with a little bit more context, remember not to give it full context but, _to see if it is able to recover and use the context that was built_) _Even though, **it fails to provide "context-aware" answer**_   1.6 **Providing the previous "full context", which it should have built in the first place by itself"**   1.7 **Let's test again if it remembers the context**: _+1 here_  1.8 One more test for checking the context awareness: _+1 here as well_ It seems I helped it to regain its short-term memories :) 

[jckpn]

I'm noticing this as well, it feels like the conversation is reset after each response.

[thephoeron]

I've also had this issue a few times, mostly when working between two different programming languages (C and Common Lisp, in my case), where Copilot Chat responds according to the C language context even though I've explicitly specified in my prompt that I'm asking about Common Lisp.

Occasionally, Copilot Chat also gets confused about context when I'm only working in one language (Common Lisp), but then it starts giving me answers in Python, when at no point in the conversation did I specify Python and have no Python projects open.

In both cases, it's a bit of a struggle to wrangle Copilot Chat back to the right language. Sometimes it will accept a reply such as "I didn't ask about Python, I asked about Common Lisp" and regenerates its reply with the correct language context. Sometimes it won't accept a conversational correction, but accepts a /fix command. And other times it falls into a loop where no matter what you ask it replies with "Sorry, but I can only assist with programming related questions."

But most of the time, Copilot Chat is surprisingly good with maintaining correct context, generating applicable and helpful suggestions for follow-up questions. It does better with contexts about formally specified models, such as algebraic and logical, rather than subjective and qualitative contexts that can only be analyzed statistically. Which is a bit weird, because with GPT-4 it's the other way around.

[culstrup]

I encountered a related issue today: Copilot refactored an older Python script rather than the most recent one. It seemed to default to this outdated version, as though it was automatically loaded into its context window.

In an attempt to rectify the issue, I highlighted all the code in the newer script. After specifically instructing Copilot to refactor this updated script and to update its context, it then provided an up-to-date response.

So, I have a question: Do messages, including the visible code in an open file, automatically get sent to Copilot Chat? Or is it necessary to manually highlight the code each time we want it to be analyzed?

[AndeYashwanth]

I notice this everytime and it's frustrating. I haven't in my 4 months of copilot chat use(keep in mind that I havent used it much due to this issue) I found it remembering anything that happened in the same chat. So, I'm just using copilot for code auto suggestions and chatgpt 3.5 for conversational style questions, even though I paid for copilot. edit: happens both in intellej and vscode

[MikeImbar]

The issue is still present. It completely loses context.

[tin-soldier]

Same issue. I has the memory of a goldfish