Suggest correct command when incorrect command name provided

waldekmastykarz commented 11 months ago

When the user provides an incorrect command name, let's use our Mendable integration to try to deduct the correct equivalent.

When the user provided an incorrect command name, and the CLI is configured in interactive mode:

we take the invalid command name without options
we embed it in a prompt
we start a new conversation with Mendable and send the prompt with the invalid message
if we get a response with the correct command name, we show the user the message: Did you mean: m365 correct command --all their --options
if we get a response command_not_found, we show the user a humand-readable message like: `Unfortunately, we couldn't find a matching command. Run 'm365 help' to see the list of available commands, or use the 🌶️ assistant to enter what you'd like to accomplish, eg. 'm365? "How to upgrade an SPFx project?"'

Here's the prompt that we could use which seem to work:

A user incorrectly typed command name: "<command name here>". Which command did they mean? If you find a matching command, return only the correct command's name in plain-text. Do not reply in full sentence, but instead return just the command's name. Ensure that the command's name starts with 'm365'. If you don't find a matching command, return "command_not_found".

waldekmastykarz commented 11 months ago

@pnp/cli-for-microsoft-365-maintainers thoughts?

Jwaegebaert commented 11 months ago

Amazing suggestion! I think something similar would also be awesome if it could validate the option names but that's more complex. One point we should consider is that AI responses are good at giving totally incorrect/made-up suggestions. Would there be a way for us to prevent stuff like this from happening?

Another idea could maybe be to implement this logic using e.g. Fuse.js. This is a lightweight library that is all about typos 😄 Here we could also provide a whole bunch of options that could help us fine-tune the results.

waldekmastykarz commented 11 months ago

Fuzzy search is a great alternative as well indeed. Its benefit is accuracy of the results and we'd need to check how it performs comparing to Mendable. If we choose for fuzzy search, then I suggest that we look for ways to build the index in the background and cache it so that we don't have to rebuild it each time, as it never changes (until you upgrade the CLI, similarly to command completion).

garrytrinder commented 11 months ago

Great idea @waldekmastykarz

One point we should consider is that AI responses are good at giving totally incorrect/made-up suggestions. Would there be a way for us to prevent stuff like this from happening?

I wonder if this is a symptom of using too high a temperature that causes hallucinations to happen. Are we able to configure this in mendable? If so, dropping it to 0 could alleviate these issues.

waldekmastykarz commented 11 months ago

We can choose between creative, normal and precise and we're already on precise.

martinlingstuyl commented 11 months ago

I like the idea, but I think I'd personally not want to use AI for this and take the road of fuzzy search.

Fuzzy search probably is faster, because we save on round trips to a server.

I feel like AI would be an overkill for this scenario and not be the best use of its power. AI deals well with unpredictable input and unpredictable output, but instead the input and the output here is highly predictable in a sense.

So I'm not yet convinced 🙂

waldekmastykarz commented 11 months ago

It all depends on how well fuzzy search can match our command names. We should definitely give it a try as it's a solid option for what we're trying to accomplish. If it's good enough, then I agree that it would be preferable over AI.

Would anyone like to do a PoC with a couple of command names to see how well fuzzy search would work?

Jwaegebaert commented 11 months ago

In the past, I've played around a bit with the fuzzy search. So I'll see if I can craft up a POC for it 😄

waldekmastykarz commented 11 months ago

In the past, I've played around a bit with the fuzzy search. So I'll see if I can craft up a POC for it 😄

Cool! Looking forward to seeing it in action

Jwaegebaert commented 11 months ago

I've made a quick POC on codesandbox to get a glimpse of how Fuse.js might work for us. https://codesandbox.io/s/cli-for-m365-fuzzy-command-search-zk5tr6?file=/src/App.js

milanholemans commented 11 months ago

Nice PoC @Jwaegebaert! Love it!

Adam-it commented 10 months ago

@Jwaegebaert I gave it a check and looks pretty awesome. The suggestion are really good. I was wondering (I know it wasn't initially part of this issue) but is it possible to also give spelling suggestions for option names?

Adam-it commented 10 months ago

BTW, just a stupid idea, but maybe first time we detect someone did a spelling mistake we could suggest setting up command completion https://pnp.github.io/cli-microsoft365/user-guide/completion

I guess in many cases it helps to type/tab the command, option correctly 😅

Jwaegebaert commented 10 months ago

I was wondering (I know it wasn't initially part of this issue) but is it possible to also give spelling suggestions for option names?

I don't think that would be impossible.

garrytrinder commented 10 months ago

Nice work @Jwaegebaert i just gave it a quick test.

One thing I noticed is that you have to match a keyword to get a result, for example using SharePoint returns no results but spo does.

I would argue that folks will more likely use SharePoint over spo in a query, so we might want to think how we can accomplish this.

martinlingstuyl commented 10 months ago

I like it @Jwaegebaert, seems to work nicely!

Jwaegebaert commented 10 months ago

One thing I noticed is that you have to match a keyword to get a result, for example using SharePoint returns no results but spo does.

That's a great point! Most mistakes are probably because users aren't using abbreviations. At first, I thought making this work with Fuse.js would be a real challenge, but I managed to get it somewhat working 😄

I've updated the POC to also function without abbreviations and provide suggestions for the right command. Just a heads up, I've only applied this to commands from m365 docs to m365 aad uservoiceidentityreport set. So, I'd recommend giving it a try with "azure active directory"

garrytrinder commented 10 months ago

@Jwaegebaert thats a great update, significant improvement!

I tried azure, azure ad, active directory and azure active directory. Works great!

Jwaegebaert commented 7 months ago

@pnp/cli-for-microsoft-365-maintainers, any further thoughts regarding the POC that was created using the Fuse.js library?

I think this library could be quite helpful to get this enhancement implemented.

Adam-it commented 7 months ago

I already gave it a check previously and left my comment. I say let's start working on this 🤩👍

leeoniya commented 7 months ago

in case you're open to using something much faster and smaller than Fuse.js, i've done an pretty thorough comparison of a few dozen fuzzy libs here: https://github.com/leeoniya/uFuzzy#a-biased-appraisal-of-similar-work

here's a demo comparing just a few libs, including Fuse.js with a 4MB corpus: https://leeoniya.github.io/uFuzzy/demos/compare.html?libs=uFuzzy,fuzzysort,QuickScore,Fuse&search=super%20ma

you can drag/drop your own list into that UI, too :)

milanholemans commented 7 months ago

Nice, thanks for the info @leeoniya!

waldekmastykarz commented 7 months ago

I think this library could be quite helpful to get this enhancement implemented.

Let's do it!

Jwaegebaert commented 7 months ago

Perfect, I'll open up the issue. Quite busy to pick it up myself but I'll keep tabs.

MathijsVerbeeck commented 2 months ago

I would love to give this a go, however it'll be quite a hard one I think 😁

milanholemans commented 2 months ago

Difficult roads often lead to beautiful destinations 😊 It's not that difficult, just don't reinvent the wheel.

pnp / cli-microsoft365

Suggest correct command when incorrect command name provided #5223