Parallel ingest system - Githubissues

Marker-Inc-Korea / RAGchain

Extension of Langchain for RAG. Easy benchmarking, multiple retrievals, reranker, time-aware RAG, and so on...

Apache License 2.0

274 stars 28 forks source link

Parallel ingest system #319

Open vkehfdl1 opened 9 months ago

vkehfdl1 commented 8 months ago

I implement Runnable system at ingest modules at #442 You first try batch and abatch function from LECL, and track how parallel ingest is effective. Plus, I watched langchain parallel ingest loader.

Eastsidegunn commented 8 months ago

me too