fix: tokenizer when no tokens - Githubissues

artefactory / NLPretext

All the goto functions you need to handle NLP use-cases, integrated in NLPretext

https://artefactory.github.io/NLPretext/

Apache License 2.0

139 stars 13 forks source link

fix: tokenizer when no tokens #296

Closed julesbertrand closed 11 months ago

julesbertrand commented 11 months ago

Description

When no tokens are retrieved for a text (e.g. preprocessing removed all words), then tokenizer is returning a ValueError about spacy lan_model not supported

the check for spacy lan_model should be done beforehand
if text is empty, tokens can be empty and should be returned as an empty list

+ Bump dependencies to latest available using poetry lock

Related Issue

Type of Change

[ ] 📚 Examples / docs / tutorials / dependencies update
[x] 🔧 Bug fix (non-breaking change which fixes an issue)
[ ] 🥂 Improvement (non-breaking change which improves an existing feature)
[ ] 🚀 New feature (non-breaking change which adds functionality)
[ ] 💥 Breaking change (fix or feature that would cause existing functionality to change)
[ ] 🔐 Security fix

Checklist

[x] I've read the CODE_OF_CONDUCT.md document.
[x] I've read the CONTRIBUTING.md guide.
[x] I've updated the code style using make format-code.
[x] I've written tests for all new methods and classes that I created.
[x] I've written the docstring in Google format for all the methods and classes that I used.