[Question]: Upload a 4kb txt document, analyze it overnight, spend 30 million tokens, split over 3000 fragments, stuck at 99.7%.

infiniflow / ragflow

RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding.

https://ragflow.io

Apache License 2.0

22.32k stars 2.18k forks source link

[Question]: Upload a 4kb txt document, analyze it overnight, spend 30 million tokens, split over 3000 fragments, stuck at 99.7%. #2007

Open lanyur opened 2 months ago

lanyur commented 2 months ago

Describe your problem

KevinHuSh commented 2 months ago

Is it locally deployed or could you share a sample of this text file? It's weird 4K text file will generate 3K chunks. I guess it stucked with ES indexing. 3000 chunks means that it will call LLM 3K times. I suggest to enlarge chunk token number in configuration. BTW, You can choose other LLM suppliers.