Feature Request: Add support for custom chunking strategy for all types of document readers

phidatahq / phidata

Build AI Agents with memory, knowledge, tools and reasoning. Chat with them using a beautiful Agent UI.

https://docs.phidata.com

Mozilla Public License 2.0

15.55k stars 2.13k forks source link

Feature Request: Add support for custom chunking strategy for all types of document readers #1373

Open yogin16 opened 3 weeks ago

yogin16 commented 3 weeks ago

Inspiration: 5-levels of chunking

https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/tutorials/LevelsOfTextSplitting/5_Levels_Of_Text_Splitting.ipynb

Reader should support multiple levels of chunking strategy, including a custom chunker

manthanguptaa commented 3 weeks ago

That's a great suggestion @yogin16! I will bring this up to the team and see when we can have this feature incorporated. Thank you for this excellent suggestion