This PR adds a fixed size text splitter component. This is a simple text splitter that splits text into a specified fixed size with an overlap. Unit tests have been added for this. All examples and tests that previously used the LangChain text splitter components have been updated to use this component.
Type of Change
[X] New feature
[ ] Bug fix
[ ] Breaking change
[ ] Documentation update
[ ] Project configuration change
Complexity
Complexity: Medium
How Has This Been Tested?
[X] Unit tests
[X] E2E tests
[X] Manual tests
Checklist
The following requirements should have been met (depending on the changes in the branch):
Introduced a fixed size text splitter component for splitting text into specified fixed size chunks with overlap. Updated examples and tests to utilize this new component.
to commit the new content to the CHANGELOG.md file, please type:
'/update_changelog --pr_update_changelog.push_changelog_changes=true'
Description
This PR adds a fixed size text splitter component. This is a simple text splitter that splits text into a specified fixed size with an overlap. Unit tests have been added for this. All examples and tests that previously used the LangChain text splitter components have been updated to use this component.
Type of Change
Complexity
Complexity: Medium
How Has This Been Tested?
Checklist
The following requirements should have been met (depending on the changes in the branch):