search

pasky / claude.vim

Claude vim plugin for AI pair programming - a hacker's gateway to LLMs
MIT License
203 stars 10 forks source link

Prompt caching support & Potential improvements #16

Open p4-k4 opened 1 week ago

p4-k4 commented 1 week ago

First of all, thanks for bringing this to life!

Unsure exactly if prompt caching is implemented under the hood however, I'd hope so! There's some huge savings $$ to be made.

https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching

It would also be nice to have feedback on cost, just as Cline does. It summarises cost based on a "Task" and also granually on each prompt and response for that task.

Cline has functionality to detect // Rest of code remains the same and warn you, then you can manually merge these parts.

The ability to be able to @tag files/folders would be great too to have finer control over what's included in a prompt.

Cheers

pasky commented 1 week ago

All nice ideas! I have been already thinking about prompt-caching in the past, it would be totally reasonable. It would also let us rely more on tool use for suggesting edits, rather than the markdown blocks which are actually pretty fragile (especially the associated movements).

I'm also unlikely right now to implement this myself, but I'll be happy to review contributions.

There are cost estimates printed as messages currently - there might be some better UX, of course.

@tagging may make sense, though in the way I use vim personally the context aligns pretty well. (If anything, it would be nice to have support for smart languageserver-driven context makeup, perhaps using https://github.com/codestoryai/sidecar or something similar.) But a feature on my wishlist forever is to be able to switch between a "complex" and "simple" prompt - I'd use claude.vim more if it didn't launch into lengthy problem-solving deliberation for every simple question I ask it. :)

p4-k4 commented 1 week ago

Yeah you're aware of prompt caching in the anthropic API call right? (it's a relatively new thing and in beta I think) but I saved a lot of both time and money with it, I wouldn't run sonnet without it otherwise too expensive.

I almost (very reluctantly) switched to vscode just because of Cline as vim lacks these kind of plugins, except that claude.vim seems to be closest resembling. I'm currently using vscode/cline for time consuming tasks, then switching back to nvim for manual writing although obviously not preferred.

Cline has a very small and simple front-facing feature set and it functions extremely well without being over-complicated, but its core logic is quite powerful - If you get a chance, have a look, might spark some ideas.

Pretty sure there would be some pre-defined instructions/prompts, as it also pulls things such as diagnostics, terminal output etc from vscode, it has look-ahead on mistakes and fixes them (extremely well) etc.

My experience with @tagging filesfolders in Cline is that we can selectively reduce tokens that go out and also restrict Cline to only working on those particular files for that given task. Otherwise it just reads files as it gathers context (adding to the prompt cache).

I'm far from lua adept myself unfortunately but hopefully someone else picks this up or if I get desperate I'll start digging.

pasky commented 1 week ago

It's not hard to get going if you use claude.vim to extend claude.vim, even if you don't know any vimscript or Python! (Though don't expect Claude to do everything on its own either.)

I don't have a lot of time to develop new Claude.vim features at the time being, I'm usually improving it only in short burts when I use it intensely, which isn't really "daily" right now due to my work mix.