Context should try harder before saying "Sorry, but I can only assist with programming related questions."

[potatoqualitee]

• VS Code Version: Latest Stable
• OS Version: Mac latest beta

Steps to Reproduce:

1. Write a chapter in a book titled "AI for IT", Chapter 5 "Reports and Documentation" in Asciidoc (.adoc) format
2. Auto complete a few paragraphs with inline Copilot
3. Ask Copilot Chat "what should i write next? give me an outline and the conclusion paragraph for this section"
4. Copilot Chat says "Sorry I can't help you with that. I can only help with programming"
5. Tell it "But im writing an AI book for IT pros"
6. Watch as it outlines the chapter

Thank you 🙇🏼

[sergioisidoro]

Just a note that I think the "Sorry, but I can only assist with programming related questions." should be relaxed a bit, since not everything we do in VS code is programming, sometimes it's ui copy, translation resources, etc.

Sometimes I use co-pilot to help me with suggestions for UI copy (eg. when text is too long for a certain UI element, I ask co-pilot to make it shorter, and give it feedback like "can you make it more friendly"). It's super helpful, especially for people who are not native speakers of a language to write good UI copy

I got the "Sorry I can't help you with that. I can only help with programming" for the first time today.

Using this trick works, I just need to say "yeah, but I'm writing a copy for a UI in react native" and it will gladly output the answer, but I would urge the co-pilot team to broaden the restriction to not only programming, but other programming and product building tasks :)

[potatoqualitee]

@roblourens just saw your off topic label. am I in the wrong repo?

[roblourens]

No, the issue is about our overactive off-topic filter :)

[neo-anderson]

Another example: Q: What are other terms like "tech debt" to describe code that is work in progress, still working on it and needs polish? A: Sorry, but I can only assist with programming related questions. Q: tech debt is programming related A: Sorry, but I can only assist with programming related questions.

[dpk899]

03 May 2024. This is still an Issue.

[Faazlaeeq]

Why copilot?

[fgaletic]

This happens more and more. Are we supposed to pay more?

[Multisaft7]

Same here... and my questions are definetly code related :-D

[frigvid]

I've been suffering with this more and more too. I ask it to literally generate some code, and I get "Sorry, but I can only assist with programming related questions." back. And then, if I only answer with something like "yes you can" it generates what I asked for. I'm practically having an aneurysm trying to make it be useful. It's rather quick to just . . . not do what it was created for, even if the question or prompt is or isn't programming related. Whether it is or isn't should be up to us, not to an arbitrary ruling or weights in the model. Not when it clearly is not up to the task.

[luismvargasg]

Same here, with code related questions...

[fgaletic]

I've been suffering with this more and more too. I ask it to literally generate some code, and I get "Sorry, but I can only assist with programming related questions." back. And then, if I only answer with something like "yes you can" it generates what I asked for. I'm practically having an aneurysm trying to make it be useful. It's rather quick to just . . . not do what it was created for, even if the question or prompt is or isn't programming related. Whether it is or isn't should be up to us, not to an arbitrary ruling or weights in the model. Not when it clearly is not up to the task.

It's crazy how many times I had to say "yes you can" to it and then it does what it's supposed to. Super annoying.

[potatoqualitee]

This does not solve the problem, but it is a workaround: I created a GitHub Copilot Chat Extension that allows you to use the OpenAI (Azure or openai.com) API within GitHub Copilot Chat.

https://marketplace.visualstudio.com/items?itemName=chrissylemaire.assistants-chat-extension

If you don't already have an assistant setup, it'll offer to setup a Beavis and Butthead assistant which demonstrates that there really aren't any restrictions on the chat output.

Note that this does cost API tokens.

[fgaletic]

@potatoqualitee thank you, I got stuck at this step:

"In the search bar, type "Assistants Chat Extension". Couldn't find it.

Anyway, do we know anything about a recent update that made the chat worse? It's incredible how uncooperative and generic it's been lately.

[potatoqualitee]

@fgaletic ah, ive renamed it since and will update the docs, thank you. You can just skip that step, it'll auto-detect the missing key and prompt you to add either openai or azure openai info.

[digitarald]

We updated the off-topic filter and see a significant drop in false-positive filtering in experiment results.

Closing as fixed, but please don't hesitate to file new issues in case you still run into this.

[potatoqualitee]

thank you! 🙏🏼