Keyword extraction pipeline fixes

guillaume-be / rust-bert

Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)

https://docs.rs/crate/rust-bert

Apache License 2.0

2.67k stars 216 forks source link

Closed guillaume-be closed 1 year ago

guillaume-be commented 1 year ago

Fix keyword extraction pipeline skipping 1-character tokens
Fix keyword extraction pipeline excluding all n-grams longer than 2
Expose new optional tokenizer_forbidden_ngram_chars to filter forbidden characters in n-grams. Defaults to punctuation (excluding hyphen)