search

microsoft / vscode-copilot-release

Feedback on GitHub Copilot Chat UX in Visual Studio Code.
https://marketplace.visualstudio.com/items?itemName=GitHub.copilot-chat
Creative Commons Attribution 4.0 International
295 stars 28 forks source link

Sorry, the response was filtered by the Responsible AI Service. Please rephrase your prompt and try again. Learn more. #1209

Open salientknight opened 1 month ago

salientknight commented 1 month ago

This has been happening routinely on a wide range of requests. If I repeat the question enough times it will eventually give an answer. Just with up-arrow enter. This problem is rendering the tool useless.

Steps to Reproduce:

1.Ask a question 2.

salientknight commented 1 month ago

To resolve this problem, I have updated and started a new chat

salientknight commented 1 month ago

Since reporting this problem, I am not getting the same error on nearly every request. It's very frustrating

fisforfaheem commented 1 month ago

i get this everyday

zbeaver4 commented 1 month ago

This just started cropping up for me. it's making chat unusable.

fisforfaheem commented 1 month ago

where is the COpilot teams??? instead of making new features kindly UPDATE us here on issues, also no changelog and daily updates.

ricbwood commented 1 month ago

Same here, it's the answer to every question I post. I used the 'send to new chat' option to start a new session. That seems to have cleared it for now.

digitarald commented 1 month ago

@salientknight and others who keep getting these often; which kind of stacks/languages are you working on when this happens? We are working to update the RAI filters and knowing the specific areas better would massively help.

For others, please post GitHub Copilot Chat logs when this happened, so we can get more details on what happened from diagnostics.

fisforfaheem commented 1 month ago

We are using Google Flutter SDK AND DART SDK LANGUAGE. I get it mostly every few days

salientknight commented 1 month ago

This tool is becoming unusable. I get Sorry, the response was filtered by the Responsible AI Service. Or some other lame error all the time and have to ask my questions over and over. It also cannot keep context at all. I ask a workspace question and in the follow ups it can't keep track of the workspace. If I ask a question on highlighted code the minute it's not highlighted it loses context... and this problem has been here for quite a while with no traction

fisforfaheem commented 1 month ago

cuz they are still using chatgpt 3.5 which is old

On Fri, Jun 7, 2024 at 6:31 PM Daniel Lefebvre @.***> wrote:

This tool is becoming unusable. I get Sorry, the response was filtered by the Responsible AI Service. Or some other lame error all the time and have to ask my questions over and over. It also cannot keep context at all. I ask a workspace question and in the follow ups it can't keep track of the workspace. If I ask a question on highlighted code the minute it's not highlighted it loses context... and this problem has been here for quite a while with no traction

— Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-copilot-release/issues/1209#issuecomment-2154847445, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIRXJSC2VWOSN7RQJW5C7B3ZGGY4JAVCNFSM6AAAAABHU2QKNOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJUHA2DONBUGU . You are receiving this because you commented.Message ID: @.***>

digitarald commented 1 month ago

@salientknight please confirm which dev stacks you are working on when you hit the content filter.

fisforfaheem commented 1 month ago

Flutter and dart sdk, and pub.dev

On Fri, Jun 7, 2024, 11:08 PM Harald Kirschner @.***> wrote:

@salientknight https://github.com/salientknight please confirm which dev stacks you are working on when you hit the content filter.

— Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-copilot-release/issues/1209#issuecomment-2155294579, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIRXJSE45MSZBKYG3H62UHLZGHZI7AVCNFSM6AAAAABHU2QKNOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJVGI4TINJXHE . You are receiving this because you commented.Message ID: @.***>

Silloky commented 3 weeks ago

Same problem here

PierreLambert3 commented 3 weeks ago

Same, shameful coming from a payed service. I'm starting to experiment with alternatives to github copilot because of this BS

fisforfaheem commented 3 weeks ago

They are not improving it and its been months also wrong and old suggestions

csidirop commented 2 weeks ago

Same here with a really easy CSS question. After responding about two paragraphs it stoped with this message.

VSC version: 1.90.1

virtuallyunknown commented 2 weeks ago

I'm starting to get this error almost every day, and the "Learn more" button leads to this particular page but if you pay attention to the URL, you will see a location hash (#enabling-or-disabling-duplication-detection) which does not exist on said page.

So really, I have no idea why Copilot chat will just randomly refuse to respond to certain messages.

image

Silloky commented 1 week ago

@virtuallyunknown oh, that's an important detail !

So I searched a bit, and it turns out OpenAI Codex throws that exact error when it matches public code. And GitHub Copilot is based on Codex. According to this thread on Reddit, and more specifically, all the messages leading to this reply, this problem we are encountering is due to Copilot's auto-protection from copyright infringement. Copilot was trained using all public code on GitHub, and when it answers our questions, it either writes its own code from what it has learnt or simply outputs the code of a repo is learnt during training. As this poses copyright and licence problems, Copilot hasn't been retrained, and now has a filter that checks if the code it's about to output is too close to existing public code. According to this page, step 3, if you head into the Copilot output in the terminal, you should see the link to the repo Copilot was about to copy and paste. I have tried this example, and it does work !

image

We can also try disabling 'Public code suggestions' on this page, maybe it'll help, I don't know...

All this documentation applies to the inline Copilot suggestions, but there's no reason it'd be different for Copilot Chat.

So, if anybody gets the error again in Chat, could you please head into the 'GitHub Copilot Chat' output, and send the output.

Thanks again @virtuallyunknown for finding this !

virtuallyunknown commented 1 week ago

Nice finds @Silloky

According to this page, step 3, if you head into the Copilot output in the terminal, you should see the link to the repo Copilot was about to copy and paste.

This works and I can reproduce the example, but only for in-editor code completions (or whatever the correct term is), but the issue we have is with Copilot chat. With Copilot chat the log stays completely empty.

All this documentation applies to the inline Copilot suggestions, but there's no reason it'd be different for Copilot Chat.

Exactly.

We can also try disabling 'Public code suggestions' on this page, maybe it'll help, I don't know...

The prompt from my screenshot above always gets for some reason (meaning I can reproduce on demand) and I tried it with that option disabled, but it makes no difference.

Maybe it takes a while for the changes to apply, but honestly if we're paying $120/year, we shouldn't be going down rabbit holes and more importantly getting ghosted by Microsoft. Between this issue and #1175, the problems keep on piling and I'm yet to hear any official response.

EDIT: Forgot to include chat logs:

2024-06-24 17:58:23.129 [info] [chat fetch] request.response: [https://api.githubcopilot.com/chat/completions], took 2726 ms
2024-06-24 17:58:23.130 [info] [streamChoices] Request id invalid, should have "completionId" and "created": {"headerRequestId":"d4f888c9-d9f7-44df-9d20-b8a0cf7d6d68","completionId":"","created":0,"serverExperiments":"","deploymentId":""},{"headerRequestId":"d4f888c9-d9f7-44df-9d20-b8a0cf7d6d68","completionId":"","created":0,"serverExperiments":"","deploymentId":""}
2024-06-24 17:58:23.131 [info] [streamMessages] message 0 returned. finish reason: [content_filter]
2024-06-24 17:58:23.131 [info] [streamChoices] request done: requestId: [d4f888c9-d9f7-44df-9d20-b8a0cf7d6d68] model deployment ID: []
virtuallyunknown commented 1 week ago

Also, it looks like the issue has been going on at least since February.

Silloky commented 1 week ago

Thanks @virtuallyunknown ! This was like cool low-level OSINT 🙃 ! (sort of)

Yes, I understand the problem is with Chat, not inline code-completions.

As I understand it, this is only the consequences of GitHub's legal problems concerning public-code use for training. GitHub should integrate the same flow for Chat as for the inline, that is, to show the public code snippets that it's using.

(meaning I can reproduce on demand)

Yes, I can't reproduce it either. I guess as it's AI and probabilistic, the outcome is never exactly the same, and so it's not always flagged.

The prompt from my screenshot above always gets for some reason (meaning I can reproduce on demand) and I tried it with that option disabled, but it makes no difference.

Well that was to be expected really. Turning this off probably flags for inline completions too, if you think about what it means.

As for the 120$/year, well, I can't really complain, as I got Copilot and GitHub Pro for free as a student 😄 .

It would be nice if we had any feedback on this though. Just to know if they're working on it...

Silloky commented 1 week ago

I was going to add, if anyone feels like contributing to the project and solving this nagging problem, but I just realised it wasn't open-source.

Or is it ?

It should be, and they should just keep the cloud service paid. That would be logical for a dev community...

salientknight commented 1 week ago

I want to add that if you use up arrow two or three time to resubmit the question the problem eventually goes away

On Mon, Jun 24, 2024 at 11:13 AM Elias Kirkwood @.***> wrote:

I was going to add, if anyone feels like contributing to the project and solving this nagging problem, but I just realised it wasn't open-source.

Or is it ?

It should be, and they should just keep the cloud service paid. That would be logical for a dev community...

— Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-copilot-release/issues/1209#issuecomment-2186815757, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJS3QDEDD2A5RGFX5ZUULN3ZJAZSZAVCNFSM6AAAAABHU2QKNOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBWHAYTKNZVG4 . You are receiving this because you were mentioned.Message ID: @.***>