microsoft / vscode-copilot-release

Feedback on GitHub Copilot Chat UX in Visual Studio Code.
https://marketplace.visualstudio.com/items?itemName=GitHub.copilot-chat
Creative Commons Attribution 4.0 International
327 stars 30 forks source link

reply claims that code is correct (it wasn't), and then changes the code, anyway #1939

Open macMikey opened 7 hours ago

macMikey commented 7 hours ago

Type: Bug

see prompt + code.

Extension version: 0.22.2024100702 VS Code version: Code 1.94.2 (Universal) (384ff7382de624fb94dbaf6da11977bba1ecd427, 2024-10-09T16:08:44.566Z) OS version: Darwin arm64 22.6.0 Modes:

System Info |Item|Value| |---|---| |CPUs|Apple M1 Max (10 x 2400)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: enabled_on
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_graphite: disabled_off
video_decode: enabled
video_encode: enabled
webgl: enabled
webgl2: enabled
webgpu: enabled
webnn: disabled_off| |Load (avg)|2, 2, 2| |Memory (System)|32.00GB (0.92GB free)| |Process Argv|--crash-reporter-id 7e89f650-4496-4cba-81dc-51018a60c509| |Screen Reader|no| |VM|0%|
A/B Experiments ``` vsliv368:30146709 vspor879:30202332 vspor708:30202333 vspor363:30204092 vscod805:30301674 binariesv615:30325510 vsaa593:30376534 py29gd2263:31024239 c4g48928:30535728 azure-dev_surveyone:30548225 962ge761:30959799 pythongtdpath:30769146 pythonnoceb:30805159 asynctok:30898717 pythonmypyd1:30879173 2e7ec940:31000449 pythontbext0:30879054 accentitlementsc:30995553 cppperfnew:31000557 dsvsc020:30976470 pythonait:31006305 dsvsc021:30996838 724cj586:31013169 dvdeprecation:31068756 dwnewjupytercf:31046870 impr_priority:31102340 nativerepl2:31139839 refactort:31108082 pythonrstrctxt:31112756 wkspc-onlycs-t:31132770 nativeloc1:31134641 wkspc-ranged-t:31151552 cf971741:31144450 autoexpandse:31146404 iacca2:31156134 notype1:31157159 5fd0e150:31155592 dwcopilotcf:31162479 icondisabled:31158250 ```
macMikey commented 7 hours ago

i did not see a "answer is incorrect" in the list of options. here is the problem: copilot both says that the code is correct, which it is not, and then proposes changing the code to make it correct.