Text chunking using recursive text splitting for plain text and markdown

[oxaroky02]

• Use the baran gem which implements character splitters based on Langchain's text splitters
• Define a recursive text splitting chunker with customizable separator rules
• Modify Text parser to use recursive text splitting
• Define CommonMark parser derived from Text with custom separator rules to account for markdown and embedded HTML tables
• Modify Docx parser to subclass CommonMark and tweak pandoc invocation to not word-wrap long lines.
• Modify Parsers#parser_for to select CommonMark parser for .md files
• Modify the default chunking size/overlap to bigger defaults of 1000 and 200 characters respectively, which yield better chunks, especially with the new text splitter

[oxaronick]

These chunks are readable!

[oxaronick]

Before I dive in, is it expected that PDF files would still use simple word splitting?

[oxaroky02]

Before I dive in, is it expected that PDF files would still use simple word splitting?

For this PR, yes. The PDF conversion yields text with ... oddness. I haven't really tested that with the new splitter, and I'd like more time for that.