Separator mode for regex segmentation.

PAIR-code / lit

The Learning Interpretability Tool: Interactively analyze ML models to understand their behavior in an extensible and framework agnostic interface.

https://pair-code.github.io/lit

Apache License 2.0

3.46k stars 352 forks source link

Separator mode for regex segmentation. #1433

Closed copybara-service[bot] closed 5 months ago

copybara-service[bot] commented 5 months ago

Separator mode for regex segmentation.

Instead of matching segment prefixes, this treats the full match as it's own segment. Use this instead of ugly lookbehind assertions for line and paragraph splitting.