Ragdaemon

[granawkins]

This aims to be the simplest possible integration of ragdaemon into mentat:

• CodeContext.daemon is added, updated initially (async) in Session._main, and then updated on each call to CodeContext.get_code_message. It replaces everything to do with filters (including embeddings).
• include_files, CodeFeature and code_file_manager are relatively undisturbed for simplicity. In future PRs we might refactor these to take better advantage of ragdaemon.
• By default, I mock out daemon.update for tests. For tests quick require full context, I added the @pytest.mark.ragdaemon flag, which will use (and cache) embeddings with chroma. Longer term this should also be mocked out, or tests of that functionality moved to the ragdaemon repo.
• Currently this doesn't chunk files. Ragdaemon does it with an LLM instead of ctags. I've turned it off for now (annotators doesn't include 'chunker'), but will aim to after some more optimization.

Pull Request Checklist

• [x] Make ragdaemon a pip package
• [x] Remove unused files (filters, embeddings..)
• [ ] Documentation has been updated, or this change doesn't require that

[mentatbot[bot]]

MENTAT CODE REVIEW IN ACTIVE DEVELOPMENT. Only in use on mentat and internal repos. Please Reply with feedback.

This PR introduces ragdaemon into Mentat, significantly altering how code context is managed. The changes are foundational and appear to streamline the process by leveraging ragdaemon's capabilities. Key considerations include ensuring backward compatibility, robustness in .gitignore handling, and the removal of ctags-related functionality. Additionally, the testing strategy should be adapted to cover these new integrations comprehensively. Documentation and examples on using the new Daemon-based approach would be beneficial for maintainers and developers.

[mentatbot[bot]]

MENTAT CODE REVIEW IN ACTIVE DEVELOPMENT. Only in use on mentat and internal repos. Please Reply with feedback.

This PR introduces significant changes by integrating ragdaemon into Mentat. Key areas to focus on include ensuring backward compatibility, precise documentation on the changes, and robust testing to cover the new functionalities. Additionally, consider specifying exact versions for dependencies like ragdaemon to ensure consistent and stable builds.

[mentatbot[bot]]

MENTAT CODE REVIEW IN ACTIVE DEVELOPMENT. Only in use on mentat and internal repos. Please Reply with feedback.

This PR introduces significant changes by integrating ragdaemon into Mentat. Key areas to focus on include ensuring backward compatibility, precise documentation on the changes, and robust testing to cover the new functionalities. Additionally, consider specifying exact versions for dependencies like ragdaemon to ensure consistent and stable builds.

[mentatbot[bot]]

MENTAT CODE REVIEW IN ACTIVE DEVELOPMENT. Only in use on mentat and internal repos. Please Reply with feedback.

This PR introduces significant changes by integrating ragdaemon into Mentat. Key areas to focus on include ensuring backward compatibility, precise documentation on the changes, and robust testing to cover the new functionalities. Additionally, consider specifying exact versions for dependencies like ragdaemon to ensure consistent and stable builds.

[mentatbot[bot]]

MENTAT CODE REVIEW IN ACTIVE DEVELOPMENT. Only in use on mentat and internal repos. Please Reply with feedback.

This PR introduces a foundational change by integrating ragdaemon into Mentat, significantly altering the code context management. It's crucial to ensure backward compatibility, robust testing, and clear documentation to facilitate a smooth transition. Additionally, pinning dependencies like ragdaemon to specific versions can help maintain consistent behavior across different environments.

[granawkins]

Tests passing (but behind main). Todo here:

• [x] redo Spice implementation w updates
• [ ] redirect to pip

Todo on ragdaemon before merging here:

• [x] Redo Spice integration and mentat integration
• [ ] Reimplement daemon.load(), w/ checksum match
• [x] Fix db records overlap/corruption issues
• [ ] Benchmark against Mentat's current auto-context
• [ ] Write documentation
• [x] Add to pip

[mentatbot[bot]]

MENTAT CODE REVIEW IN ACTIVE DEVELOPMENT. Only in use on mentat and internal repos. Please Reply with feedback.

The integration of ragdaemon into Mentat introduces significant changes, particularly in how code context is managed. It's crucial to ensure that these changes are well-documented, backward-compatible, and thoroughly tested. Additionally, consider specifying exact versions for dependencies like ragdaemon to ensure consistent and stable builds. The use of Path operations for file handling and the potential performance implications of the new Daemon-based approach should also be carefully considered.