CodeWhisper

AI-Powered End-to-End Task Implementation & blazingly fast Codebase-to-LLM Context Bridge

About • Why CodeWhisper? • Key Features • Quick Start • Installation • Usage • Templates • Configuration • API • Benchmarking • Contributing • Roadmap • FAQ

📖 About

CodeWhisper is a powerful tool that bridges the gap between your codebase and Large Language Models (LLMs). It serves two primary functions:

1. AI-Powered End-to-End Task Implementation: Tackle complex, codebase-spanning tasks with ease. CodeWhisper doesn't just suggest snippets; it plans, generates, and applies comprehensive code changes across your entire project, from backend logic to frontend integration. CodeWhisper's generations are SOTA and outperform other AI-code generation tools in benchmarks. See Benchmarking for more details.

2. Precision-Guided Context Curation for LLMs: Harness the power of human insight to feed AI exactly what it needs. Quickly transform carefully selected parts of your codebase into rich, relevant context for LLMs, ensuring more accurate and project-aligned results.

Whether you're implementing comprehensive features, tackling complex refactoring, conducting thorough code reviews, or seeking AI-driven architectural insights, CodeWhisper equips your AI tools with the comprehensive understanding they need. It's not just about coding assistance – it's about enabling AI to be a true collaborator in your software development process.

💬 Join the Community

Connect with fellow users and developers, share insights, discuss features, and get support for leveraging CodeWhisper in your coding workflow by joining our CodeWhisper Discord.

🤔 Why CodeWhisper?

CodeWhisper was born out of a simple yet powerful idea: to provide AI models with meticulously curated context from your entire codebase in the most comfortable way possible. What started as a tool to generate comprehensive and customizable prompts from your codebase has evolved into a full-fledged AI-assisted development workflow solution.

The Power of Manual Context Curation

Many AI coding assistants and tools fall short when tackling tasks that demand a comprehensive understanding of your project. They often lack the big-picture context necessary for making informed decisions about your codebase as a whole. CodeWhisper addresses this limitation through its unique manually curated context approach, delivering end-to-end task implementation with a git-first workflow:

✨ Key Features

📺 Video

Both videos feature CodeWhisper using Claude 3.5 Sonnet for the plan and code generation steps.

Code Generation workflow

https://github.com/user-attachments/assets/64922cf6-658e-4036-a251-cfc458a14801

Interactive codebase-to-LLM

https://github.com/user-attachments/assets/a3a1ad99-d402-4781-9672-7228c0aa2f93

🚀 Quick Start

Get started with CodeWhisper in just a few steps:

# Install CodeWhisper globally
npm install -g codewhisper

# Navigate to your project directory
cd /path/to/your/project

# Start an AI-assisted coding task
# CodeWhisper will prompt you to select a model from the list of available models
codewhisper task

# The mode of operation for generating code modifications is set automatically based on the model.
# You can override this by using the --diff or --no-diff option.
codewhisper task --diff
codewhisper task --no-diff

# You can also specify a model directly
# Claude-3.5 Sonnet
codewhisper task -m claude-3-5-sonnet-20240620

# GPT-4o
codewhisper task -m gpt-4o-2024-08-06

# DeepSeek Coder
codewhisper task -m deepseek-coder

# Or use a local Ollama model (not recommended as it will be slow and inaccurate for comprehensive feature implementation tasks)
codewhisper task -m ollama:llama3.1:70b --context-window 131072 --max-tokens 8192

# To undo changes made by an AI-assisted task, use the --undo option
codewhisper task --undo

# To redo the last task with the option to change the model, file selection or plan, use the --redo option.
# Note: CodeWhisper saves the plan, instructions, model and selected files from the last task. Other options (such as --dry-run) need to be specified again.
codewhisper task --redo

# Run the task without the AI-generated planning step
codewhisper task --no-plan

# Automatically accept the AI-generated plan and directly proceed to the code generation step
codewhisper task --accept-plan

# Generate an AI-friendly prompt using interactive mode
codewhisper interactive

# List available models
codewhisper list-models

Note: If you are using CodeWhisper's LLM integration with codewhisper task, you will need to set the respective environment variable for the model you want to use (e.g., export ANTHROPIC_API_KEY=your_api_key or export OPENAI_API_KEY=your_api_key or export GROQ_API_KEY=your_api_key or export DEEPSEEK_API_KEY=your_api_key ).

For more detailed instructions, see the Installation and Usage sections.

Supported Models and Current Recommendations

While CodeWhisper supports a variety of providers and models, our current recommendations are based on extensive testing and real-world usage. Here's an overview of the current status:

Model Evaluation

This section is still under development. We are actively testing and evaluating models.

* Whole-file edit mode is generally more precise but may lead to issues with maximum output token length, potentially limiting the ability to process larger files or multiple files simultaneously. It can also result in incomplete outputs for very large files, with the model resorting to placeholders like "// other functions here" instead of providing full implementations.

For more details, see the Benchmarking section.

Experimental Support

• Groq as a provider
  • We're eager to test Llama 3.1 405B on Groq
  • Current rate limits are too restrictive for thorough testing
  • Awaiting access to paid plans and the larger Llama 3.1 model for further evaluation

Local Models (via Ollama)

Currently not recommended for complex tasks. Models tested include:

• Llama 3.1 (8B to 70B variants)
• DeepSeek Coder V2
• Mistral Nemo
• Mistral Large

These models currently struggle to follow instructions accurately for comprehensive task implementation. However, we are actively working on:

1. Improving the workflow for smaller local models
2. Developing an evaluation pipeline for consistent performance measurement
3. Fine-tuning prompts to better suit the capabilities of local models

📦 Installation

You can use CodeWhisper without installation using npx, or install it globally:

# Using npx (no installation required)
npx codewhisper <command>

# Global installation
npm install -g codewhisper

💻 Usage

CodeWhisper offers several commands to cater to different use cases:

For detailed usage instructions and examples, please refer to USAGE.md.

Undoing AI-Assisted Task Changes

To undo changes made by an AI-assisted task, use the --undo option with the task command:

codewhisper task --undo

This command will:

• Discard uncommitted changes if any
• Delete the AI-generated branch if not on the main branch
• Offer to revert the last commit if on the main branch

The command will always ask for confirmation before making any changes. It will show you the exact actions it's about to perform, including the full commit message of any commit it's about to revert.

Always review the proposed changes carefully before confirming, as this operation may result in loss of work.

Redoing AI-Assisted Tasks

CodeWhisper now supports redoing AI-assisted tasks from the review plan stage. This feature allows you to restart your last task with the option to modify the generated plan as well as the model and file selection. To use this feature, use the --redo option with the task command:

codewhisper task --redo

When you use the --redo option:

1. CodeWhisper will retrieve the last task data for the current project directory.
2. It will display the details of the last task, including the task description, instructions, model used, and files included.
3. You'll be given the option to change the AI model for code generation.
4. You'll also have the opportunity to modify the file selection for the task.
5. The task will then continue from the review plan stage, where you can then modify the plan to your liking.

This feature is particularly useful when:

• You want to try a different AI model for the same task
• You need to adjust the file selection for the same task
• You want to quickly tweak the plan without starting from scratch

Note: The redo functionality uses a cache stored in your home directory, so it persists across different sessions and is not affected by git operations like branch switching or resetting.

Example workflow:

1. Run an initial task: codewhisper task
2. Review the code modifications and decide you want to tweak the plan or try a different model
3. Undo the task: codewhisper task --undo
4. Redo the task: codewhisper task --redo
5. Optionally change the model when prompted
6. Optionally adjust the file selection
7. Adjust the plan as needed

This feature enhances the flexibility of CodeWhisper's AI-assisted workflow, allowing for quick iterations and experimentation with different models or scopes for your tasks.

📝 Templates

CodeWhisper uses Handlebars templates to generate output. You can use pre-defined templates or create custom ones. For in-depth information on templating, see TEMPLATES.md.

🔧 Configuration

For more details on custom templates, extending CodeWhisper, and integrating with other tools, check CUSTOMIZATION.md.

Environment Variables

To use CodeWhisper's LLM integration, you need to set the appropriate environment variable for the model you want to use:

GitHub Integration

To use the GitHub issue integration feature, you need to set the GITHUB_TOKEN environment variable with a valid GitHub personal access token.

You can create a fine-grained personal access token in your GitHub account settings. The token needs the following permission:

• "Issues" repository permission with read access

This allows CodeWhisper to list repository issues.

To set up the token:

1. Create a fine-grained personal access token following GitHub's documentation.
2. Set the environment variable:

export GITHUB_TOKEN=your_github_personal_access_token

To use the GitHub integration in your CodeWhisper tasks:

1. Use the --github-issue flag to select from open issues in the current repository:

codewhisper task --github-issue -m claude-3-5-sonnet-20240620

2. Use the --github-issue-filters flag to filter issues by label, assignee, or other criteria:

codewhisper task --github-issue --github-issue-filters assignee:abc,label:p1 -m claude-3-5-sonnet-20240620

The --github-issue-filters option accepts comma-separated key:value pairs. For a full list of available filter options, refer to the GitHub API documentation.

Note: This endpoint can be used without authentication if only public resources are requested. However, using a token is recommended to avoid rate limiting and access private repositories.

For more detailed instructions on using the GitHub integration and other CodeWhisper features, please refer to USAGE.md.

📚 API

CodeWhisper can be used programmatically in your Node.js projects. For detailed API documentation and examples, please refer to USAGE.md.

Benchmarking

CodeWhisper includes a benchmarking tool to evaluate its performance on Exercism Python exercises. This tool allows you to assess the capabilities of different AI models and configurations.

Key Features

• Docker-based execution for consistent environments
• Concurrent worker support for faster benchmarking
• Detailed Markdown reports with performance metrics
• Options to customize test runs (number of tests, planning mode, diff mode)

Usage

1. Build the Docker image:

   ./benchmark/docker_build.sh

2. Set up the appropriate API key as an environment variable.

3. Run the benchmark:

   ./benchmark/run_benchmark.sh --model <model_name> --workers <num_workers> --tests <num_tests> [options]

Output

The benchmark generates a detailed Markdown report including:

• Summary statistics (total time, cost, pass percentage)
• Per-exercise results (time, cost, mode, model, tests passed)

Reports are saved in benchmark/reports/ with timestamped filenames.

Results

CodeWhisper's performance has been evaluated across different models using the Exercism Python exercises. Below is a summary of the benchmark results:

*The cost calculation was not working properly for this benchmark run.

Note: All benchmarks are one-shot only, unlike other benchmarks which use multiple generations that depend on the results of the test run.

The full reports used to generate these results are available in the benchmark/reports/ directory.

These results provide insights into the efficiency and accuracy of different models when used with CodeWhisper. The "Tests Passed" percentage indicates the proportion of Exercism tests successfully completed, while the time and cost metrics offer a view of the resource requirements for each model.

As we continue to run benchmarks with various models and configurations, this table will be updated to provide a comprehensive comparison, helping users make informed decisions about which model might best suit their needs.

For full details on running benchmarks, interpreting results, and available options, please refer to the Benchmark README.

🤝 Contributing

We welcome contributions to CodeWhisper! Please read our CONTRIBUTING.md for details on our code of conduct and the process for submitting pull requests.

🏎️ Roadmap

• [x] Add AI-assisted task creation and code generation
• [x] Add GitHub integration for fetching issues and pull requests
• [ ] Add other integrations for fetching issues and pull requests (GitLab, Jira, Linear, etc.)
• [x] Finish OpenAI and Groq support
• [x] Add support for other LLMs
• [x] Add support for local models via Ollama
• [x] Experiment with partial file modifications
• [ ] Experiment with generateObject with a fixed schema
• [ ] Run evaluations on generated code
• [ ] Possibly add agentic behaviors

❓ FAQ

📄 License

MIT License © 2024-PRESENT Gordon Mickel

Contributors 👨‍💻

👏 Acknowledgments

• Handlebars for templating
• Commander.js for CLI support
• fast-glob for file matching
• Inquirer.js for interactive prompts
• Vercel AI SDK for the great AI SDK

📬 Contact

Gordon Mickel - @gmickel - gordon@mickel.tech

Project Link: https://github.com/gmickel/CodeWhisper

Repo Analytics

Project Growth

⭐ If you find CodeWhisper useful, please consider giving it a star on GitHub to show your support! ⭐

Made with ❤️ by Gordon Mickel.