Custom levels - Githubissues

segment-any-text / wtpsplit

Toolkit to segment text into sentences or other semantic units in a robust, efficient and adaptable way.

MIT License

677 stars 39 forks source link

Closed bminixhofer closed 3 years ago

bminixhofer commented 3 years ago

This PR makes two changes:

The split instructions are not hardcoded anymore, instead now stored in the ONNX model metadata.
Split levels can be accessed via:

Python: splitter.get_levels() JS: splitter.getLevels() Rust: splitter.logic().split_sequence().get_levels()

In preparation for customized split levels for some models (e. g. compound constituents for German).