Creates an internal CLI tool (and corresponding Worker) for triaging issues in the worker-sdk project

[andyjessop]

What this PR solves / how to test

Triaging issues takes up a lot of our time. It's a very manual process, and can be quite tedious. This tool is an attempt to alleviate some of that tedium by enlisting an enthusiastic AI helper to classify issues on demand, helping us to make faster and better decisions in the triaging process.

The PR has been split into 4 commits for easier reviewing:

• chore: setup issue-triager package
• chore: create shared types
• feat: create CLI for updating embeddings and triaging issues
• feat: create Worker for exposing similar issues with Vectorize

In short, a Worker is created hosting a Vectorize index that stores embeddings of all issues on the repo. A last_updated_at timestamp is also stored in a DO. The Worker doesn't do anything else - it's quite dumb and doesn't go looking for the issues itself, it just embeds what it's given.

The "smart" part of the tool is in the CLI, which has several jobs including:

• updating the embeddings worker with the latest GitHub issues,
• fetching untriaged issues from the GitHub API,
• fetching similar issues from the embeddings worker,
• and finally prompting the AI to classify a selected issue.

Usage

Follow the README.md to setup with the correct environment variables.

The triaging tool is run using pnpm triage - it will then guide you through selecting the issues you want to classify.

GitHub GraphQL API

The GraphQL API was chosen for a few important reasons:

1. You cannot at present retrieve project information via the REST API, so the GraphQL API is the only way to get untriaged issues (as "Untriaged" is not a status available on the issue itself, but is a relationship to a specific project).
2. There is some difficulty in retrieving all issues using the REST API, but this is available using the GraphQL API, and is necessary to ensure that the vector DB has all available embeddings.

Some notes on how this might be improved in the future

Accuracy At present, this tool is limited by the maximum context window allowed on WorkersAI, which looks like 6144 characters. So even if we find 10 good matches for similar issues, we can't include them all in the prompt. We can fit 2 or 3. This is only just better than a one-shot attempt with just the untriaged issue as context.

In the absence of more context, a potential improvement would be to create our own LoRA adapter, by training it on our existing issues and their resolutions. It's worth noting here that we don't know exactly how much better, if at all, this will be because on WorkersAI you can only fine tune the older models with this method. So the question then becomes is Llama 2 + a LoRA adapter better than straight Llama 3? Probably, but there's uncertainty.

The main way we can increase accuracy and efficacy of this tool is by increasing the context window. This opens the door to including not just similar issues, but code snippets, community discussions, etc. Ultimately we could have a number of vector indexes that each provide relevant snippets to the context.

Auto-updating This was a "quick win" so was setup to be as simple as possible, and the interaction with the GitHub API is all via the CLI. We could, however, setup a web hook from the repo straight to the worker, and update the vector DB in real time.

Beyond "classifying" Potential use-cases for this or similar bots are:

• commenting on PRs with suggested fixes
• informing the user of similar issues and their resolutions
• alerting maintainers of critical bugs
• even things like reviewing PRs and suggesting fixes. All depends on the accuracy and capability of the LLMs, of course.

Author has addressed the following

• Tests
  • [ ] TODO (before merge)
  • [ ] Included
  • [x] Not necessary because: This is a quick wins week experiment. Can flesh out with tests if we decide to keep it.
• E2E Tests CI Job required? (Use "e2e" label or ask maintainer to run separately)
  • [ ] I don't know
  • [ ] Required / Maybe required
  • [x] Not required because: Doesn't touch user-facing code, just a QoL update.
• Changeset (Changeset guidelines)
  • [ ] TODO (before merge)
  • [ ] Included
  • [x] Not necessary because: ^ as above.
• Public documentation
  • [ ] TODO (before merge)
  • [ ] Cloudflare docs PR(s):
  • [^] Not necessary because: ^ as above

[changeset-bot[bot]]

⚠️ No Changeset found

Latest commit: dd8da2dc4061832dfce326b7f96b1b0209131b03

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[threepointone]

Wow exciting!