[Subtask] Convert utility functions from splitText.ts to Python

[debuggerone]

Subtask Overview

This subtask focuses on converting utility functions such as those in splitText.ts from JavaScript to Python, ensuring all functionalities are preserved and optimized for the Python environment.

Tasks

• [x] Translate utility functions from splitText.ts to Python.
• [x] Ensure each function is optimized for Python’s standard libraries and idiomatic code practices.
• [x] Test all converted functions to verify they handle text splitting and token limits effectively.

Acceptance Criteria

• All utility functions from splitText.ts are successfully converted to Python.
• The functions are optimized for Python’s capabilities and maintain their original functionality.
• Converted functions are tested and validated to handle text splitting and token management correctly.

[Stevenic]

More of a future thinking component. I have an idea for recursive summarizer that would use this. It's basically a variant of LangChain's recursive text splitter that I've been fine tuning for the last year or so.

[debuggerone]

I have an idea for that too but it would require a combination of some systems like neo4j and a vector db...