search

gr-b / repogather

Easily copy all relevant source files in a repository to clipboard. For use in LLM code understanding and generation workflows
MIT License
149 stars 9 forks source link

Allow the use of other LLM/provider #1

Open jasonm23 opened 2 weeks ago

jasonm23 commented 2 weeks ago

It would be fantastic if repogather was not vendor locked to OpenAI services.

Providing for the use of Ollama, LocalAI etc. would be lovely.

gr-b commented 2 weeks ago

I agree!

For the initial implementation, gpt-4o-mini was the obvious choice -- it's the fastest model capable of computing the relevance reliably, and scales very well to repositories in the 1m token range (medium sized). For most users' hardware, the limited size of local context length will make this take quite a while to return. However, as a matter of principle, I'd like to support it as soon as possible (along with other vendors).

Ideally, I'd like to fine-tune a task specific LLM (for source relevance filtering), and then prune or quantize it to make it as small as possible while still yielding appropriate results. This could enable running repogather locally and costlessly, while preserving the UX.

jasonm23 commented 2 weeks ago

Thank you for the response, glad you agree.

Obviously, when you have time etc.

jtoy commented 2 weeks ago

I agree!