Please give Github Copilot chat VISION 👀

[Khalizo]

Please can you give GitHub copilot eyes using OpenAI's GPT4 vision API?

As you can imagine, the use cases are endless and not limited to:

• Referencing matplotlib charts to help recreate new charts
• Analysing screenshots of websites or apps to create code to recreate those screenshots
• Provided suggestions via code to help improve designs
• etc ...

Here is the link to the API: https://platform.openai.com/docs/guides/vision

Thanks, Babs

[dbsxdbsx]

Yes, I want to pop up this feature. With an image uploaded to Copilot, we can do someting like , for example:

1. For AI code: "Look at the red boxed part in the image, could you tell me how to recog this kind of stuff using OpenCV in python/C++"?
2. For UI design："I want to make a nice looking tab bar just like what I show to you, how to do it using flutter(it is ok if need 3rd party package, just tell me what and how to do)"

[hidaris]

It looks like the cursor has already done this https://twitter.com/amanrsanger/status/1729943964000883186 , hoping to see similar functionality in Copilot

[IAMIbrahimmemon]

what woould be awesome is if you could connect with figma and it would make the UI layout

[alexander-kastil]

Parsing and implementing Figma drafts in GitHub workspaces would be 💪

[dbsxdbsx]

Frankly speaking, now I am using Cursor not Vscode anymore, though I've already subscribed 1year for ms copilot 2 month ago before I moved to Cursor.

For the Copilot team, be it tech limitation or not paying enough attention, I feel that the MS Copilot is trying to add some useless features, like voice understanding(besides, this voice understanding is really poor) and GitHub commit-msg auto-generation. For the real user-acquired features, like the vision/image understanding and more project context understanding, acquire updated msg from online search, or the doc and composer features(from the Cursor), I see no roadmap for them.

Even after the MS Copilot updated to OpenAi o1 mini model, I still usually feel that the AI is just giving me superficial guidance like: "You should check this, debug that...", and not point out the core issues---it seems like the model is my boss, not the contrary way.

[alexander-kastil]

Frankly speaking, now I am using Cursor not Vscode anymore, though I've already subscribed 1year for ms copilot 2 month ago before I moved to Cursor.

For the Copilot team, be it tech limitation or not paying enough attention, I feel that the MS Copilot is trying to add some useless features, like voice understanding(besides, this voice understanding is really poor) and GitHub commit-msg auto-generation. For the real user-acquired features, like the vision/image understanding and more project context understanding, acquire updated msg from online search, or the doc and composer features(from the Cursor), I see no roadmap for them.

Even after the MS Copilot updated to OpenAi o1 mini model, I still usually feel that the AI is just giving me superficial guidance like: "You should check this, debug that...", and not point out the core issues---it seems like the model is my boss, not the contrary way.

I do not see commit-msg auto-generation as a useless feature. If you switch to small commits, the generated messages are pretty good. And lots of good commit messages result in marvelous, good pull request descriptions!

[dbsxdbsx]

Frankly speaking, now I am using Cursor not Vscode anymore, though I've already subscribed 1year for ms copilot 2 month ago before I moved to Cursor. For the Copilot team, be it tech limitation or not paying enough attention, I feel that the MS Copilot is trying to add some useless features, like voice understanding(besides, this voice understanding is really poor) and GitHub commit-msg auto-generation. For the real user-acquired features, like the vision/image understanding and more project context understanding, acquire updated msg from online search, or the doc and composer features(from the Cursor), I see no roadmap for them. Even after the MS Copilot updated to OpenAi o1 mini model, I still usually feel that the AI is just giving me superficial guidance like: "You should check this, debug that...", and not point out the core issues---it seems like the model is my boss, not the contrary way.

I do not see commit-msg auto-generation as a useless feature. If you switch to small commits, the generated messages are pretty good. And lots of good commit messages result in marvelous, good pull request descriptions!

Maybe I put it too arbitrary. Anyway, we human have to do the monitor job: CHECK EVERYTHING the AI generated, which led me think the auto-short commit msg generation seems to be not quite useful. But you are right, it does can decorate the commit msg better, AS LONG AS we do know what all modified codes are, and we do at least have a rough idea on what is a good commit msg.

[alexander-kastil]

@Khalizo Found something that I want to share: https://marketplace.visualstudio.com/items?itemName=ms-vscode.vscode-copilot-vision … Somehow I think it closes that issue

[alexander-kastil]

@Khalizo Found something that I want to share: https://marketplace.visualstudio.com/items?itemName=ms-vscode.vscode-copilot-vision … Somehow I think it closes that issue

I now had the chance to test the extension. It has one big disadvantage. You have to bring your own model and key. That would be ok for me, but the main disadvantage is that it does not support models from Azure Open AI ... which is what I am using.

Please support this issue https://github.com/microsoft/vscode-copilot-vision/issues/73

[Khalizo]

Thanks for sharing this! I have moved across to Cursor now for a while now. Really like the Claude 3.5 sonnet model for coding.

It's a shame Copilot can't iterate as fast...

On Sat, 2 Nov 2024, 07:51 Alexander Kastil, @.***> wrote:

@Khalizo https://github.com/Khalizo Found something that I want to share: https://marketplace.visualstudio.com/items?itemName=ms-vscode.vscode-copilot-vision … Somehow I think it closes that issue

I now had the chance to test the extension. It has one big disadvantage. You have to bring your own model and key. That would be ok for me, but the main disadvantage is that it does not support Models from Azure Open AI ... which is what I am using

— Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-copilot-release/issues/554#issuecomment-2452910965, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFMRRLP67DJNTINA3PKFJVLZ6SAA7AVCNFSM6AAAAAA7DB762SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJSHEYTAOJWGU . You are receiving this because you were mentioned.Message ID: @.***>