microsoft / vscode-copilot-release

Feedback on GitHub Copilot Chat UX in Visual Studio Code.
https://marketplace.visualstudio.com/items?itemName=GitHub.copilot-chat
Creative Commons Attribution 4.0 International
342 stars 37 forks source link

Claude 3.5 Sonnet doesn't write code directly to the editor in Edit mode #2692

Open TheWonderfulTartiflette opened 4 days ago

TheWonderfulTartiflette commented 4 days ago

Type: Bug

When using the Edit mode, Claude 3.5 Sonnet will write its code in the conversattion window rather than in the editor directly, like it would do if I was using the regular Chat mode

Extension version: 0.22.2 VS Code version: Code 1.95.3 (f1a4fb101478ce6ec82fe9627c43efbf9e98c813, 2024-11-13T14:50:04.152Z) OS version: Windows_NT x64 10.0.22631 Modes:

System Info |Item|Value| |---|---| |CPUs|Intel(R) Core(TM) Ultra 9 185H (22 x 3072)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: enabled_on
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_graphite: disabled_off
video_decode: enabled
video_encode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled
webgpu: enabled
webnn: disabled_off| |Load (avg)|undefined| |Memory (System)|31.50GB (13.16GB free)| |Process Argv|--crash-reporter-id 024431e7-4766-4589-a0a4-6877ebe3e958| |Screen Reader|no| |VM|0%|
A/B Experiments ``` vsliv368:30146709 vspor879:30202332 vspor708:30202333 vspor363:30204092 vscod805cf:30301675 binariesv615:30325510 vsaa593cf:30376535 py29gd2263:31024239 c4g48928:30535728 azure-dev_surveyone:30548225 2i9eh265:30646982 962ge761:30959799 pythonnoceb:30805159 asynctok:30898717 pythonmypyd1:30879173 h48ei257:31000450 pythontbext0:30879054 cppperfnew:31000557 dsvsc020:30976470 pythonait:31006305 dsvsc021:30996838 jg8ic977:31013176 dvdeprecation:31068756 dwnewjupyter:31046869 nativerepl1:31139838 pythonrstrctxt:31112756 cf971741:31144450 iacca1:31171482 notype1:31157159 5fd0e150:31155592 dwcopilot:31170013 stablechunks:31184530 ```
aeschli commented 4 days ago

I'm not able to reproduce with the latest pre-release:

Here are my steps.

}


- open chat edits view, make sure 'Hello.ts' is part of the working set
- select Claude 3.5 Sonnet as model
- ask `rename `hello` to `world``
![Image](https://github.com/user-attachments/assets/8deada23-8a49-4df4-a74a-3cc634b56407)

Can you provide similar steps that show the problem?
TheWonderfulTartiflette commented 3 days ago

Yes my bad, I forgot to say that this happens only sometimes. There doesn't seem to be history for the Copilot Edits, unlike Copilot Chat, so I can't give you exactly the prompt/answer it gave, but I think that this mainly happened when he needed to modify long files (mine was like 600 lines in python) btw how did you change the font in the copilot window ? i don't see where you change it

TheWonderfulTartiflette commented 3 days ago

I had this prompt where the issue occurred : Create a Python code, from start to finish, of a sort of “Paint” app, where the user can draw, fill, use shapes that he can resize, save the file, etc etc etc.. Make it as feature rich as possible. And make the UI not too bad

And for the first time, this occurred with o1-preview, whereas before this always occurred with Claude 3.5 Sonnet. Here is the screenshot Image

aeschli commented 3 days ago

Thanks a lot. It's difficult to reproduce, but I saw it yesterday too. Likely the LLM uses a code block syntax that we don't expect (but the UI can rennder)