search

vercel / modelfusion

The TypeScript library for building AI applications.
https://modelfusion.dev
MIT License
1.1k stars 78 forks source link

Add chunkOverlap parameter to splitAtToken / splitAtCharacter #108

Open lgrammel opened 11 months ago

lgrammel commented 11 months ago

Ideally there would just be an optional ‘overlap’ parameter in the existing split functions. This might be a good time to add a test framework as well to test that the splitting works correctly. Ideally jest bc GPT-4 can easily write tests for it.