microsoft / vscode-copilot-release

Feedback on GitHub Copilot Chat UX in Visual Studio Code.
https://marketplace.visualstudio.com/items?itemName=GitHub.copilot-chat
Creative Commons Attribution 4.0 International
345 stars 37 forks source link

Confuses gpt-4 with gpt-4o #2000

Open pamelafox opened 1 month ago

pamelafox commented 1 month ago

Type: Bug

@azure what's the pricing for gpt-4o?

Azure (Preview) Here are the pricing details for GPT-4 models provided by the Azure OpenAI Service:

Input: $0.01 per 1,000 tokens Output: $0.03 per 1,000 tokens For further details on special pricing information, you can refer to the GPT-4 pricing page.

Responses are currently scoped to the CA-Python-2411 tenant. Select a different tenant.

Extension version: 0.22.2024102304 VS Code version: Code - Insiders 1.95.0-insider (Universal) (fe997185b5e6db94693ed6ef5456cfa4e8211edf, 2024-10-23T05:06:13.568Z) OS version: Darwin arm64 23.6.0 Modes:

System Info |Item|Value| |---|---| |CPUs|Apple M1 Pro (10 x 2400)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: enabled_on
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_graphite: disabled_off
video_decode: enabled
video_encode: enabled
webgl: enabled
webgl2: enabled
webgpu: enabled
webnn: disabled_off| |Load (avg)|4, 6, 6| |Memory (System)|16.00GB (0.05GB free)| |Process Argv|--crash-reporter-id 29b0fda8-7569-499a-9770-cfbb5ac8c0cf| |Screen Reader|no| |VM|0%|
A/B Experiments ``` vsliv368:30146709 vspor879:30202332 vspor708:30202333 vspor363:30204092 vscod805cf:30301675 vsaa593cf:30376535 py29gd2263:31024238 c4g48928:30535728 a9j8j154:30646983 962ge761:30841072 pythongtdpath:30726887 welcomedialog:30812478 pythonnoceb:30776497 asynctok:30898717 dsvsc014:30777825 dsvsc015:30821418 pythonmypyd1:30859725 2e7ec940:31000449 pythontbext0:30879054 accentitlementst:30870582 dsvsc016:30879898 dsvsc017:30880771 dsvsc018:30880772 cppperfnew:30980852 pythonait:30973460 945dj816:31013170 a69g1124:31018687 dvdeprecation:31040973 dwnewjupytercf:31046870 newcmakeconfigv2:31071590 nb_pri_only:31057983 nativerepl1:31134653 refactort:31084545 pythonrstrctxt:31093868 flighttreat:31119334 wkspc-onlycs-t:31132770 nativeloc1:31118317 wkspc-ranged-c:31125598 cf971741:31144450 e80f6927:31120813 fje88620:31121564 i21gd607:31141543 iacca2:31144504 notype1:31143044 b50ed353:31138333 showbadge:31139796 f8igb616:31140137 ```
pamelafox commented 1 month ago

From Output- it looks like a query rewriting step is responsible:

2024-10-23 12:25:49.171 [info] [retrieve_ms_learn_documentations] Function retrieve_ms_learn_documentations invoking. 2024-10-23 12:25:49.173 [info] [Microsoft.Azure.CopilotForAzure.AIService.Plugins.VSCodeDevLearn.VSCodeDevLearnPlugin] Retrieving MS Learn Documentation as query type 'Hybrid' for 'Query: GPT-4 pricing Context: Visual Studio Code'. 2024-10-23 12:25:49.919 [info] [Microsoft.Azure.CopilotForAzure.AIService.Plugins.VSCodeDevLearn.VSCodeDevLearnPlugin] Get MS Learn doc: {"Url":"https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/gpt-with-vision?wt.mc_id=knwlserAPI_inproduct_sdkextention#special-pricing-information","PageType":"conceptual","Score":0.02049180120229721} 2024-10-23 12:25:49.919 [info] [Microsoft.Azure.CopilotForAzure.AIService.Plugins.VSCodeDevLearn.VSCodeDevLearnPlugin] Get MS Learn doc: {"Url":"https://learn.microsoft.com/en-us/power-platform-release-plan/2022wave1/data-integration/power-query-visual-studio-code-based-sdk?wt.mc_id=knwlserAPI_inproduct_sdkextention#power-query-visual-studio-code-based-sdk","PageType":"conceptual","Score":0.01666666753590107} 2024-10-23 12:25:49.919 [info] [Microsoft.Azure.CopilotForAzure.AIService.Plugins.VSCodeDevLearn.VSCodeDevLearnPlugin] Get MS Learn doc: {"Url":"https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/gpt-with-vision?wt.mc_id=knwlserAPI_inproduct_sdkextention#special-pricing-information","PageType":"conceptual","Score":0.02111111208796501} 2024-10-23 12:25:49.919 [info] [Microsoft.Azure.CopilotForAzure.AIService.Plugins.VSCodeDevLearn.VSCodeDevLearnPlugin] Get MS Learn doc: {"Url":"https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?wt.mc_id=knwlserAPI_inproduct_sdkextention#gpt-4-and-gpt-4-turbo-models","PageType":"conceptual","Score":0.03100961446762085} 2024-10-23 12:25:49.919 [info] [Microsoft.Azure.CopilotForAzure.AIService.Plugins.VSCodeDevLearn.VSCodeDevLearnPlugin] Get MS Learn doc: {"Url":"https://learn.microsoft.com/en-us/azure/ai-services/openai/gpt-v-quickstart?pivots=programming-language-python\u0026wt.mc_id=knwlserAPI_inproduct_sdkextention#gpt-4-turbo-model-upgrade","PageType":"conceptual","Score":0.012820512987673283} 2024-10-23 12:25:49.919 [info] [retrieve_ms_learn_documentations] Function retrieve_ms_learn_documentations succeeded. 2024-10-23 12:25:49.919 [info] [retrieve_ms_learn_documentations] Function completed. Duration: 0.7482871s

pamelafox commented 1 month ago

I think our docs could probably use a refresh to emphasize 4o versus gpt-4-turbo-vision: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/gpt-with-vision?wt.mc_id=knwlserAPI_inproduct_sdkextention#special-pricing-information

pamelafox commented 1 month ago

I even got a confabulated result when I got it to query a page which lists the 4o models. Notice it says "GPT-4 Mini" here- but there is not GPT-4 mini, only gpt-4o-mini. I'm guessing gpt-4o isn't in the weights at all, so it doesn't like outputting it as an option.

@azure /learn what GPT models are available besides gpt-4o?

Azure (Preview)
Azure OpenAI Service provides a variety of GPT models besides GPT-4o:

GPT-4 Mini
GPT-4 Turbo with Vision
GPT-4 (standard and different versions such as preview models)
GPT-3.5-Turbo series
Embeddings model series
For more details, visit the Azure OpenAI Service models page.
https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?tabs=python-secure
pamelafox commented 1 month ago

FYI, Bullwinkle says this about the turbo-vision docs: "Yes all of the vision docs in the AOAI TOC are slated to be updated before Ignite to make them GPT-4o/GPT4o-mini centric and to drop any legacy references to the GPT-4 "

So I'm just going to avoid 4o questions for now.

pamelafox commented 1 month ago

I see this issue landed in the main GitHub repo, this issue is specific to @azure.