feat: give commands the ability to execute logic

lukealvoeiro commented 2 weeks ago

Current state of commands (before this PR)

Commands currently only serve autocompletion purposes. For example:

As it currently stands, commands do not trigger anything to occur when they are included in the prompt and enter is pressed.

Where does the `/file` command currently fall short?

The /file command is included in the string that we will sent to the LLM, which introduces some uncertainty in regard to the user intent. This is because the LLM does not reliably know what a command such as /file:src/* means.

How to remediate this?

1. ✅ Introduce a way for commands to modify their portion of the prompt that will be sent off

❌ Indicate in the system prompt that commands exist, and that we should ignore the first portion of them (the command name)

Why did I choose 1 above?

Currently, only one command exists - /file. However, we can imagine that folks may want to contribute other commands. Some of ideas of these may include:

/rewind(:times) - rewinds until after the last user message. this would allow the user to remove an incorrect path the LLM went down, and instruct it in a different way.
/quorum:"set up 3 instances of goose that talk to this app and load test it" - this would make quorum more reliably be picked up (currently it is a toolkit that at times is not used despite indicating you want to use it)
/add_toolkit:<toolkit_name> - adds a toolkit midway through a session's execution
/checkpoint - adds a checkpoint to the exchange that you could revert back to

Across these commands, we notice different user intents:

/rewind and /checkpoint take an action on the exchange itself and should be scrubbed from the prompt sent to the LLM
/quorum performs some logic and returns a response string, which we would then add to the prompt sent to the LLM
/add_toolkit also takes an action on the exchange itself, but might also add onto the prompt sent to the LLM (indicating that we just added a new tool).

From these examples, I see there is value in

Being able to execute some action based on the value of the command (/name:value)
Being able to replace the string /name:value with a value determined by the command

This led me to the signature for the Command.execute method:

def execute(self, query: str) -> str:
    return ""

🤔 I don't like the above - what other options are there?

Option 1: instead of commands executing some logic, we could remove the execute method and only allow them to be used for autocompletion. We would need to make a small tweak so that only the value of the command is persisted in the prompt, not the command name itself.

Option 2: completely remove the idea of commands, and either scrap the idea of file completions entirely, or just keep the file completion around and don't have it inherit from Command

michaelneale commented 2 weeks ago

a suggestion - would it be possible to detect and suggest a command a user could explore so they can accidentally discover it? One of the benefits of goose is that you ideally don't need to know how to drive it, commands start to bring more syntax/knowledge needed, so would be nice to be able to prompt people for that so it feels organic.

lukealvoeiro commented 2 weeks ago

a suggestion - would it be possible to detect and suggest a command a user could explore so they can accidentally discover it? One of the benefits of goose is that you ideally don't need to know how to drive it, commands start to bring more syntax/knowledge needed, so would be nice to be able to prompt people for that so it feels organic.

We were probably gonna work on surfacing features to users just in time, so we could work this in (if we decide to move forward with using commands).

lukealvoeiro commented 2 weeks ago

@lifeizhou-ap I know we had some discussion topics ongoing but the rest of the team is onboard - going ahead and merging this PR and we can follow up with any changes.

block-open-source / goose