Engineer - Text Chunker - Alternative Chunking Strategies

[stuartjohnpage]

Background

Currently, we only implement one strategy choice: Recursive Split. This was reverse-engineered from LangChain, with plans to add more methods in the future. Open source library written in Elixir for handling recursion-based text-splitting.

Acceptance Criteria

Scenario: Implementing additional text-splitting strategies

Given the Text Chunker library is currently using only the Recursive Split strategy

• [ ] When I implement alternative text-splitting strategies in the Text Chunker library
• [ ] Then the library should support at least two additional strategies alongside Recursive Split
• [ ] And it should allow the user to select the desired text-splitting strategy via an interface option
• [ ] And each strategy should be modular to enable easy updates and addition of new strategies in the future
• [ ] And comprehensive tests should verify that each strategy correctly splits text according to its design

• [ ] And the documentation should be updated to describe each new strategy, including when and how to use it.

  created by ross.garfield+demo@revelry.co using Prodops for demo purposes only