Add Jieba and HanLP for Chinese sentence tokenization.

urchade / GLiNER

Generalist and Lightweight Model for Named Entity Recognition (Extract any entity types from texts) @ NAACL 2024

https://arxiv.org/abs/2311.08526

Apache License 2.0

1.48k stars 127 forks source link

Closed simonChoi034 closed 5 months ago

simonChoi034 commented 5 months ago

Add Chinese language tokenizers: Jieba and HanLP

Implement two tokenizers for improved Chinese NER:
1. Jieba: Fast but less accurate
2. HanLP: Slower but significantly more accurate
Enhance NER results by addressing Chinese language tokenization challenges

urchade commented 5 months ago

Nice, thank you