search

microsoft / vscode-copilot-release

Feedback on GitHub Copilot Chat UX in Visual Studio Code.
https://marketplace.visualstudio.com/items?itemName=GitHub.copilot-chat
Creative Commons Attribution 4.0 International
301 stars 28 forks source link

Public code being blocked when setting is Public code: Allow. #742

Open bn-l opened 6 months ago

bn-l commented 6 months ago

This is the same as #712, just included proper data and screenshots for debugging. 

[INFO] [ghostText] [2024-01-12T07:34:26.046Z] Found inline suggestions locally
[INFO] [ghostText] [2024-01-12T07:34:26.242Z] Found inline suggestions locally
[INFO] [ghostText] [2024-01-12T07:34:28.273Z] Cancelled during debounce
[INFO] [default] [2024-01-12T07:34:28.428Z] [fetchCompletions] engine https://copilot-proxy.githubusercontent.com/v1/engines/copilot-codex
[INFO] [default] [2024-01-12T07:34:28.920Z] request.response: [https://copilot-proxy.githubusercontent.com/v1/engines/copilot-codex/completions] took 492 ms
[INFO] [streamChoices] [2024-01-12T07:34:29.056Z] request done: headerRequestId: [70f06ca7-192d-4231-82b3-b90557940e25] model deployment ID: [x1c5e8d1294d6]
[INFO] [code-referencing] [2024-01-12T07:39:19.200Z] Public code references are enabled.
[INFO] [ghostText] [2024-01-12T07:40:37.542Z] Cancelled during debounce
[INFO] [ghostText] [2024-01-12T07:40:37.688Z] Cancelled by contextual filter
[INFO] [ghostText] [2024-01-12T07:40:38.265Z] Cancelled by contextual filter
[INFO] [ghostText] [2024-01-12T07:40:38.462Z] Cancelled by contextual filter
[INFO] [ghostText] [2024-01-12T07:40:38.722Z] Cancelled by contextual filter
[INFO] [ghostText] [2024-01-12T07:40:41.860Z] Cancelled by contextual filter
[INFO] [ghostText] [2024-01-12T07:40:42.190Z] Cancelled by contextual filter
[INFO] [ghostText] [2024-01-12T07:40:42.476Z] Cancelled during debounce
[INFO] [ghostText] [2024-01-12T07:40:42.609Z] Cancelled during debounce
[INFO] [ghostText] [2024-01-12T07:40:42.740Z] Cancelled during debounce
[INFO] [default] [2024-01-12T07:40:42.915Z] [fetchCompletions] engine https://copilot-proxy.githubusercontent.com/v1/engines/copilot-codex
[INFO] [ghostText] [2024-01-12T07:40:44.897Z] Cancelled by contextual filter
[INFO] [ghostText] [2024-01-12T07:40:45.694Z] Cancelled during debounce
[INFO] [ghostText] [2024-01-12T07:40:45.832Z] Cancelled during debounce
[INFO] [default] [2024-01-12T07:40:45.999Z] [fetchCompletions] engine https://copilot-proxy.githubusercontent.com/v1/engines/copilot-codex
[INFO] [default] [2024-01-12T07:40:47.428Z] [fetchCompletions] engine https://copilot-proxy.githubusercontent.com/v1/engines/copilot-codex
[INFO] [ghostText] [2024-01-12T07:40:48.013Z] Cancelled during debounce
[INFO] [ghostText] [2024-01-12T07:40:48.163Z] Cancelled during debounce
[INFO] [ghostText] [2024-01-12T07:40:48.266Z] Cancelled during debounce
[INFO] [default] [2024-01-12T07:40:48.497Z] [fetchCompletions] engine https://copilot-proxy.githubusercontent.com/v1/engines/copilot-codex
[INFO] [ghostText] [2024-01-12T07:54:20.237Z] Cancelled during debounce
[INFO] [default] [2024-01-12T07:54:20.359Z] [fetchCompletions] engine https://copilot-proxy.githubusercontent.com/v1/engines/copilot-codex
[INFO] [default] [2024-01-12T07:54:20.566Z] [fetchCompletions] engine https://copilot-proxy.githubusercontent.com/v1/engines/copilot-codex
[INFO] [default] [2024-01-12T07:54:21.801Z] request.response: [https://copilot-proxy.githubusercontent.com/v1/engines/copilot-codex/completions] took 1234 ms
[INFO] [streamChoices] [2024-01-12T07:54:21.804Z] solution 0 returned. finish reason: [stop]
[INFO] [streamChoices] [2024-01-12T07:54:21.807Z] request done: headerRequestId: [82dbdb49-d050-4a71-b36a-b4f3b6ef31f6] model deployment ID: [x878ecfd0739c]
[INFO] [default] [2024-01-12T07:54:44.589Z] [fetchCompletions] engine https://copilot-proxy.githubusercontent.com/v1/engines/copilot-codex
[INFO] [default] [2024-01-12T07:54:45.114Z] request.response: [https://copilot-proxy.githubusercontent.com/v1/engines/copilot-codex/completions] took 524 ms
[INFO] [streamChoices] [2024-01-12T07:54:45.115Z] solution 1 returned. finish reason: [stop]
[INFO] [streamChoices] [2024-01-12T07:54:45.116Z] solution 0 returned. finish reason: [stop]
[INFO] [streamChoices] [2024-01-12T07:54:45.117Z] request done: headerRequestId: [e1638719-7bd8-4de8-abd6-f68314835add] model deployment ID: [x878ecfd0739c]
[INFO] [default] [2024-01-12T07:54:47.660Z] [fetchCompletions] engine https://copilot-proxy.githubusercontent.com/v1/engines/copilot-codex
[INFO] [default] [2024-01-12T07:54:47.894Z] [fetchCompletions] engine https://copilot-proxy.githubusercontent.com/v1/engines/copilot-codex
[INFO] [default] [2024-01-12T07:54:48.209Z] request.response: [https://copilot-proxy.githubusercontent.com/v1/engines/copilot-codex/completions] took 314 ms
[INFO] [streamChoices] [2024-01-12T07:54:48.211Z] solution 0 returned. finish reason: [stop]
[INFO] [streamChoices] [2024-01-12T07:54:48.212Z] solution 2 returned. finish reason: [stop]
[INFO] [streamChoices] [2024-01-12T07:54:48.212Z] solution 1 returned. finish reason: [stop]
[INFO] [streamChoices] [2024-01-12T07:54:48.213Z] request done: headerRequestId: [28536ae2-2eed-4919-9ffe-af0fefd38297] model deployment ID: [x878ecfd0739c]

The weird thing is, in the log itself it says:

[INFO] [code-referencing] [2024-01-12T07:39:19.200Z] Public code references are enabled.

Screenshots:

image

image

Steps to Reproduce:

  1. Go to https://github.com/settings/copilot. Then Policies > Suggestions matching public code > select "allow" > Press save button.
  2. Begin a copilot chat in the sidebar

Comments: I've noticed this for about a month.

wxnderer commented 6 months ago

same issue here

Suamppa commented 6 months ago

Same here

Starvingg commented 6 months ago

Same issue here, it often happens with general questions. It once happened with a specific question

lramos15 commented 6 months ago

Please update to the latest VS Code insiders + latest copilot chat release.

Then include your copilot chat output log

CMD / CTRL + SHIFT + U -> Copilot Chat

gyuray-dev commented 6 months ago

Same here

lramos15 commented 6 months ago

@gyuray-dev Please follow the instructions above and use insiders + pre-release to attach logs

gyuray-dev commented 6 months ago

@lramos15 Sorry, this is what you requested.

VScode insiders Version: 1.86.0-insider (user setup) Commit: 0b571a26e213bf1ec6266e37152ced80c006bd1e Date: 2024-01-18T05:36:10.559Z Electron: 27.2.1 ElectronBuildId: 26149897 Chromium: 118.0.5993.159 Node.js: 18.17.1 V8: 11.8.172.18-electron.0 OS: Windows_NT x64 10.0.19045

I installed pre-release version of copilot and reloaded.

I have updated public code matching policy to Allowed yesterday. image

This is error screenshot. I activated the chat by selecting code and right click -> colpilot -> Explain this: image

Github copilot output:

[INFO] [code-referencing] [2024-01-19T01:44:14.009Z] Public code references are enabled.

Github Copilot chat output:

2024-01-19 10:43:36.710 [info] [FetcherService] Using Helix fetcher
2024-01-19 10:43:36.710 [info] [gitExtensionService] Initializing Git extension service.
2024-01-19 10:43:36.710 [info] [gitExtensionService] Successfully activated the vscode.git extension.
2024-01-19 10:43:36.710 [info] [gitExtensionService] Enablement state of the vscode.git extension: true.
2024-01-19 10:43:36.710 [info] [gitExtensionService] Successfully registered Git commit message provider.
2024-01-19 10:43:36.847 [info] [auth] GitHub login failed
2024-01-19 10:43:36.850 [info] [extension] You are not signed in to GitHub. Please sign in to use Copilot.
2024-01-19 10:43:46.426 [info] [auth] Logged in as gyuray-dev
2024-01-19 10:43:46.838 [info] [chat] copilot token chat_enabled: true
2024-01-19 10:43:46.838 [info] [githubTitleAndDescriptionProvider] Initializing GitHub PR title and description provider provider.
2024-01-19 10:43:46.838 [info] [githubTitleAndDescriptionProvider] GitHub.vscode-pull-request-github extension is not yet activated.
2024-01-19 10:43:46.838 [info] [auth] Got Copilot token for gyuray-dev
2024-01-19 10:43:46.840 [info] [auth] Logged in as gyuray-dev
2024-01-19 10:43:46.841 [info] [chat] copilot token chat_enabled: true
2024-01-19 10:43:46.841 [info] [auth] Got Copilot token for gyuray-dev
2024-01-19 10:44:13.587 [info] [FetcherService] Using Helix fetcher
2024-01-19 10:44:13.587 [info] [gitExtensionService] Initializing Git extension service.
2024-01-19 10:44:13.587 [info] [gitExtensionService] Successfully activated the vscode.git extension.
2024-01-19 10:44:13.587 [info] [gitExtensionService] Enablement state of the vscode.git extension: true.
2024-01-19 10:44:13.587 [info] [gitExtensionService] Successfully registered Git commit message provider.
2024-01-19 10:44:13.759 [info] [auth] Logged in as gyuray-dev
2024-01-19 10:44:14.229 [info] [chat] copilot token chat_enabled: true
2024-01-19 10:44:14.229 [info] [githubTitleAndDescriptionProvider] Initializing GitHub PR title and description provider provider.
2024-01-19 10:44:14.229 [info] [githubTitleAndDescriptionProvider] GitHub.vscode-pull-request-github extension is not yet activated.
2024-01-19 10:44:14.229 [info] [auth] Got Copilot token for gyuray-dev
2024-01-19 10:44:14.232 [info] [auth] Logged in as gyuray-dev
2024-01-19 10:44:14.233 [info] [chat] copilot token chat_enabled: true
2024-01-19 10:44:14.233 [info] [auth] Got Copilot token for gyuray-dev
2024-01-19 10:44:55.848 [info] [chat fetch] url https://api.githubcopilot.com/chat/completions
2024-01-19 10:44:55.848 [info] [chat fetch] modelMaxPromptTokens 3072
2024-01-19 10:44:55.848 [info] [chat fetch] modelMaxResponseTokens 3545
2024-01-19 10:44:55.849 [info] [chat fetch] chat model gpt-4
2024-01-19 10:44:57.110 [info] [chat fetch] request.response: [https://api.githubcopilot.com/chat/completions], took 1261 ms
2024-01-19 10:45:01.728 [info] [streamMessages] message 0 returned. finish reason: [content_filter]
2024-01-19 10:45:01.729 [info] [streamChoices] request done: requestId: [4b4f1715-9d74-4282-af70-f9b2e760fcbe] responseId: [4b4f1715-9d74-4282-af70-f9b2e760fcbe] model deployment ID: [x8d3c95500d67]

I have tried re-installing, signing out/in, /clear, make new sessions etc but nothing have worked.

lramos15 commented 6 months ago

@gyuray-dev That messaging indicates that it is not a code filter but one of our other safety filters which cannot be turned off. What is happening in your case is one of the safety filters is being tripped, likely due to the multiple mentions of slave.

gyuray-dev commented 6 months ago

@lramos15 Sorry, I haven't read the message carefully. However, using the exact same reproduction process, the original VScode app prints Sorry, the response matched public code so it was blocked. image

So it might just be a message misprint from VScode. Anyway, if it is just an issue with content filtering, how can I get help from Copilot with the master/slave structure like EtherCAT having so many slave words?

lramos15 commented 6 months ago

@gyuray-dev It's not incorrect messaging, just we used to call all filters "public code matches". This has been updated in the latest insiders + pre-release to better distinguish to the user when they hit a filter that can be turned off vs one that cannot.

There's not much that can be done at this time, we record cases of filtration and work to improve the models which power them over time. Right now the general filter rate for users is very low. You can try to use alternative naming like client to see if that helps get you the answers you want.

Triskel26 commented 6 months ago

same issue here