Has copilot suddenly (today) become a lot dumber?

[aaa3334]

I am wondering if other people have noticed this but today I have managed to solve 0 problems with copilot (usually the number is quite high). It is constantly ignoring half the prompt and ignoring most of what is sent into it...

Have other people found this?

Honestly it is almost as though they have changed the model to gpt3 and removed all the extra knowledge (chatgpt is useless with nextjs but copilot is usually not too bad, but today it is just ignoreing the code i put in, giving the most general things and also not recognising a lot of basic things (eg if I try and debug why useFormStatus() is not wokring it will tell me it is a custom component when it is very much not (but before it would tell me it is an expermiental feature).

Is it just my copilot that has got a LOT dumber to the extent it is useless?

[digitarald]

@aaa3334 do you have a specific example for "ignoring half the prompt and ignoring most of what is sent into it". Is it refusing to answer or provides wrong answers. If you have logs, those would be great as well.

[kupietools]

I'm finding the same thing. I'm pretty new to copilot, and I find the majority of the time, in chat, it definitely disregards things I say entirely, goes off on wild tangents, introduces needless breaking changes to the scripts, etc. Very often it will give a bad answer, I will point out the problem, it gives a different totally off-kilter answer ignoring things I've told it, so I point out the problem with this one, and it goes straight back to the first one, and from there only alternates between those two bad answers over and over. Sometimes it just repeats the same incorrect answer over and over, ignoring my corrections, or, saying it will address my concerns but then providing the same bad code verbatim. Or, it will suddenly forget the entire conversation and ask me to provide the code and/or the question from scratch.

Sorry, I'm pretty new, how do I get the logs? Do you mean copy and paste the entire chat, or are there logs somewhere? I'm in VS Code on Mac.

[aaa3334]

Sorry missed this - but it seems as though it is suggesting new code more than it is adjusting code which is highlighted.

So for example if I copy and paste some api docs in and highlight my implementation and ask it to help me figure out where I have made a mistake, it will ignore the code I have highlighted (even though it is saying it is using it as a reference) and write out either by text step by step guide, or it will write new code for me... it still matches the coding language but it is ignoring what I have highlighted half the time (sometimes not but oftentimes it is, especially if the docs are a little longer).

[kupietools]

it will ignore the code I have highlighted (even though it is saying it is using it as a reference) and write out either by text step by step guide, or it will write new code for me

Yes, I get this all the time, it will answer simple questions about code by completely rewriting the entire code block to supposedly "solve" the issue, rather than answering the question or limiting its changes to the small requested change.

For instance, it will give me a solution that nearly works, and I will ask a question about some minor part of it that I think might be causing the remaining problem, and it will reply with a completely different solution using an entirely different technique. Example: yessterday, working on a Selenium script in node, it produced what appeared to be a potential solution, but also completely rewrote function parameters to require passing a variable called "locator", which appeared from context to be perhaps a By.xyz("") function, rather than a WebElement argument assigned by deconstruction as "func({element})", as all calls to that function in the rest of my code pass. I asked it what "locator" is meant to be, and it responded by saying "Apologies for the confusion" and writing completely different code, another new and completely different approach to the original problem, abandoning the solution it just gave me entirely simply because I had a question about one small aspect of it.

(BTW I then asked it to revert this subsequent rewrite, said that I just wanted to try the solution it just gave except without it introducing breaking changes by completely changing the parameters of the function, but by then it had forgot completely about my needs, and with subsequent every question I asked, it tried to talk about what "locators" are. It repeatedly answered my replies with, "Let's get back to the original request. You asked what locators are", it had completely shifted the entire topic of the session away from what I needed onto a random topic that it had hallucinated.)

[PilotGFX]

I'm just confirming that what the OP is assuming is true. I am using it every day for many hours and have a good grasp of its current performance, which basically have been in the dumpster the last week.

1) Has started using very long time to "decide which workspace to use", 2) Preloads the prompt with so much irrelevant content from your whole workspace 3) At times, will refuse to answer extremely simple questions 4) Direct example: I ask can you help me isolate the rows that has nan? > uses 4 minutes to "decide workspace to use", choose to include 5 different jupyter notebooks and two completely unrelated files from "Fooocus", then finally say it cannot complete the request because one of the ipynb files doesn't exist. 5) Yes, its also ignoring what i have highlighted and always do this annoying "gathering from workspace" thing which always result in complete malfunctioning due to appending tons of random code to the prompt. 6) Yes, i have also experienced that it basically ignored my prompt and gave a totally unrelated answer

This week where the product has not been working and i had to disable it have made me realize how peaceful it is to code without any intellisense or completion(irrelevant, but noteworthy)

Seemingly unrelated but possibly related: Intellisense also started suggesting thousands of irrelevant functions from code it had found all across my whole harddrive, as soon as i typed a single character.

Update: copilot has stopped doing the workspace thing now, completion and chat seem to have improved, but the "start in editor", "fix this" etc accessible by right clicking are in a useless state at the moment(replaces large chunks of code with one or two lines that doenst work). I have no interests other than provide insight into the issue..

[digitarald]

Closing for now as the model in the chat panel updated (see https://github.com/microsoft/vscode-copilot-release/issues/560#issuecomment-2187648041) and this has gathered a mix of different scenarios, some @workspace, some general panel questions – making this issue not actionable for the team to fix. A new chat version has come out since that have improved further how context is gathered, how @workspace works and how inline chat applies edits.

Please file new issues with specific prompts/code steps and expected answers.

[whoisArnob]

SAME, It was very intelligent and used to put visually appealing answers than any other LLMs, even ChatGPT itself, Microsoft really fine-tuned it perfectly, but now it feeeels DUMB asf

[PilotGFX]

I can only confirm after clearing possible causes on my end; it is still very dumb. Consistently though. It has gained a very persistent belief that any question must be answered with a full copy of whatever file that you were in. I have to prompt it very delicately that it does NOT have to include the whole script. This is why we build our own sh1t, or still use photoshop from 2019.. it stars like this: some guy with a tie...

On Tue, 6 Aug 2024, 20.38 whoisArnob, @.***> wrote:

SAME, It was very intelligent and used to put visually appealing answers than any other LLMs, even ChatGPT itself, Microsoft really fine-tuned it perfectly, but now it feeeels DUMB asf

— Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-copilot-release/issues/1210#issuecomment-2272010882, or unsubscribe https://github.com/notifications/unsubscribe-auth/A44HXCKQZYYWBMBSYAQKKFLZQEQ2DAVCNFSM6AAAAABHVRUOUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZSGAYTAOBYGI . You are receiving this because you commented.Message ID: @.***>