search

copilot-extensions / gh-debug-cli

CLI tool that lets you chat with your agent locally for faster feedback and debugging
MIT License
10 stars 0 forks source link
cli-tool debugging sse validation-error

readme

GH Debug CLI

This tool allows you to chat with your agent locally in order to create a faster feedback loop for developers developing an extension. Debug mode is enabled by default so that you can see clearer information around what exactly is getting parsed successfully.

The different SSE events in the agent protocol that the CLI gives debug output for are:

  1. errors
  2. references
  3. confirmations

Note: This tool does not handle the payload verification process. To use this tool to validate your events, please temporarily disable payload verification for local testing and re-enable when completed.

Install the debug tool

  1. Authenticate with GitHub CLI OAuth app 
    gh auth login --web -h github.com
  2. Install / upgrade extension 
    gh extension install github.com/copilot-extensions/gh-debug-cli
  3. See more info about the cli tool 
    gh debug-cli -h

Using the debug tool

  1. Run the following command gh debug-cli -h to see the different flags that it takes in. 
    
> gh debug-cli -h
This cli tool allows you to debug your agent by chatting with it locally.

Usage: [flags]

Flags: -h, --help help for this command --log-level DEBUG Log level to help debug events. Supported types are DEBUG, TRACE, NONE. DEBUG returns general logs. TRACE prints the raw http response. (default "DEBUG") --token string GitHub token for chat authentication (optional) --url string url to chat with your agent (default "http://localhost:8080") --username string username to display in chat (default "sparklyunicorn")

> The token noted in the flag above is used to authenticate against the provided LLM. If you are using a different service, then this token is not needed. Generate the user-to-server token by [creating a GitHub Applicatiion](https://docs.github.com/en/apps/creating-github-apps/about-creating-github-apps/about-creating-github-apps) and then following the [using the device flow to generate a user access token](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/generating-a-user-access-token-for-a-github-app#using-the-device-flow-to-generate-a-user-access-token) to generate the token.
2. You can alternatively set these flags as environment variables (in all caps) so you don't need to pass them in every time. The only "required" one to get this up and running is the url for your agent

export URL="http://localhost:8080/agent/blackbeard"

3. When you run the CLI, you will see any flags that were previously set in your environment variables as the output.

gh debug-cli Setting url to http://localhost:8080/agents/blackbeard

Start typing to chat with your assistant... sparklyunicorn: 

4. Type something to simulate chatting with your assistant.

gh debug-cli Setting url to http://localhost:8080/agents/blackbeard

Start typing to chat with your assistant... sparklyunicorn: hello assistant: Ahoy, @monalisa! A jolly good day to ye, me heartie. How can ol' Blackbeard be of service to ye today?

Huzzah! You successfully received a message! ╔═══════════╤════════════════════════════════════════════════════════════════╗ ║ Role │ Content ║ ╟━━━━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢ ║ assistant │ [condensed] Ahoy, @monalisa! A jolly good day to ye, me hearti ║ ╟━━━━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢ ║ Parsed message data ║ ╚═══════════╧════════════════════════════════════════════════════════════════╝ sparklyunicorn: 

5. To debug your SSE events, you can set up a key word that your assistant uses to send you a specific type of event. My blackbeard agent allows me to send a keyword "confirmation", and here I can see the debug output on what is parsed from the SSE event

sparklyunicorn: confirmation assistant: Arrr, @monalisa! I be ready and waitin' for yer confirmation. Be ye ready to set sail on this treacherous journey and receive a custom limerick 'bout petals? Aye or nay, let me know yer decision, and I'll be at yer service.

Huzzah! You successfully received a message! ╔═══════════╤════════════════════════════════════════════════════════════════╗ ║ Role │ Content ║ ╟━━━━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢ ║ assistant │ [condensed] Arrr, @monalisa! I be ready and waitin' for yer co ║ ╟━━━━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢ ║ Parsed message data ║ ╚═══════════╧════════════════════════════════════════════════════════════════╝

Huzzah! You successfully received a confirmation! ╔══════════════╤═════════════════════════════════════════════════════╗ ║ Key │ Value ║ ╟━━━━━━━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢ ║ type │ action ║ ║ title │ Be ye sure ye want a custom limerick 'bout petals ? ║ ║ message │ Arrr, this here action be irreversible, matey! ║ ║ confirmation │ map[id:123] ║ ╟━━━━━━━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢ ║ Parsed confirmation data ║ ╚══════════════╧═════════════════════════════════════════════════════╝

Be ye sure ye want a custom limerick 'bout petals ? Arrr, this here action be irreversible, matey! Reply: [y/N]

6. If I got a bad confirmation, it would look something like this

sparklyunicorn: bad confirmation

Alas...The following is not a valid confirmation: ["conf"]

assistant: Avast, @monalisa! Me apologies if I didn't quite understand yer request. Pray tell, could ye please clarify what be wrong with the confirmation? I be here to assist ye, me matey!

7. And if debug mode was set to false, then I would only see the confirmation prompt itself.

gh debug-cli --log-level none Setting url to http://localhost:8080/agents/blackbeard

Start typing to chat with your assistant... sparklyunicorn: confirmation assistant: Ahoy, @monalisa! Ye be seekin' confirmation, me hearty. Are ye sure ye want a custom limerick 'bout petals? This here action be irreversible, matey!

Be ye sure ye want a custom limerick 'bout petals ? Arrr, this here action be irreversible, matey! Reply: [y/N]


8. Currently, the supported event types for debug mode are references, errors, and confirmations! Have fun chatting with your assistant!

## Copilot Extensions Documentation
- [Using Copilot Extensions](https://docs.github.com/en/copilot/using-github-copilot/using-extensions-to-integrate-external-tools-with-copilot-chat)
- [About building Copilot Extensions](https://docs.github.com/en/copilot/building-copilot-extensions/about-building-copilot-extensions)
- [Set up process](https://docs.github.com/en/copilot/building-copilot-extensions/setting-up-copilot-extensions)
- [Communicating with the Copilot platform](https://docs.github.com/en/copilot/building-copilot-extensions/building-a-copilot-agent-for-your-copilot-extension/configuring-your-copilot-agent-to-communicate-with-the-copilot-platform)
- [Communicating with GitHub](https://docs.github.com/en/copilot/building-copilot-extensions/building-a-copilot-agent-for-your-copilot-extension/configuring-your-copilot-agent-to-communicate-with-github)