search

zed-industries / zed

Code at the speed of thought – Zed is a high-performance, multiplayer code editor from the creators of Atom and Tree-sitter.
https://zed.dev
Other
50.12k stars 3.07k forks source link

copilot chat doesn't use chatGPT 4 #16053

Closed karemkassem closed 1 month ago

karemkassem commented 3 months ago

Check for existing issues

Describe the bug / provide steps to reproduce it

the two modes for the new copilot chat ( GPT-4 and GPT-3.5) don't seem to make any difference, I asked both of them which model of gpt they are using and they both replied with GPT-3, even when picking the GPT-4. I tried the same thing with the same account in vscode and it returned that it uses GPT-4o. ( as seen in the pictures below ) is this a bug? or something that I did wrong? anyway I have been using zed ever since its release on linux and it has been amazing. great work.

Environment

Zed: v0.147.2 (Zed) OS: Linux Wayland arch unknown Memory: 15.3 GiB Architecture: x86_64 GPU: Intel(R) Graphics (ADL GT2) || Intel open-source Mesa driver || Mesa 24.1.5-arch1.1

If applicable, add mockups / screenshots to help explain present your vision of the feature

Screenshot_20240810_141735 Screenshot_20240810_141809

If applicable, attach your Zed.log file to this issue.

Zed.log
notpeter commented 3 months ago

I can reproduce this with my GitHub account. I also get the following response:

I am based on OpenAI's GPT-3 model.
linuxmobile commented 3 months ago

image

same here.

aperiogon3141 commented 3 months ago

yes i am experiencing the same issue

Hawkbawk commented 2 months ago

I'm not sure just how much I'd trust the model here, as it very well could be hallucinating or just plain wrong. If you look through the code, you can see that the GitHub API is indeed being called with different models, and there's not anything else that Zed can do beyond that. A better test would be to see how the different models perform on the same task, as GPT-4 should be significantly better. Regardless, I've just created a merge request to enable GPT-4o functionality (#16446 ) which I can confirm from my testing is definitely being used. The 128K context window and much faster response time is absolutely wonderful for asking questions in an unfamiliar codebase and confirms the API is working as intended.

AidfulAI commented 1 month ago

Never trust the output of the model if you ask it about itself. If the system prompt is not updated correctly, you will get wrong information.

To test if you are using GPT-3 or GPT-4 use the following prompt, which is with ~9 out of 10 only correctly solved by GPT-4: Give me 10 sentences that end with the word "apple"

notpeter commented 1 month ago

I'm going to go ahead and close this. In the intervening two months I've become accustomed to models hallucinating other models as their own name, even when the correct name is explicitly defined in the system prompt, so this is no longer as surprising to me that it could get this wrong.