The regex sentence splitter is not a very accurate sentence splitter but we would like to provide an initial implementation of aggregation and splitting for bidirectional streaming use, in the case of streaming text chunks/tokens needing to be aggregated to sentences for further sentence analysis.
For tracking purposes, output streamed sentences remain directly concatenable.
The regex sentence splitter is not a very accurate sentence splitter but we would like to provide an initial implementation of aggregation and splitting for bidirectional streaming use, in the case of streaming text chunks/tokens needing to be aggregated to sentences for further sentence analysis.
For tracking purposes, output streamed sentences remain directly concatenable.
Closes: https://github.com/caikit/caikit-nlp/issues/345