use channels for git commit loader to stream per commit instead of loading all commits in memory

Abraxas-365 / langchain-rust

🦜️🔗LangChain for Rust, the easiest way to write LLM-based programs in Rust

MIT License

491 stars 63 forks source link

Closed prabirshrestha closed 4 months ago

prabirshrestha commented 4 months ago

[x] use flume channels only when git feature is enabled.
[x] stream the commits to channel instead of loading all commits in memory
[x] update example to use while loop to stream so it can work with large git repositories.