Improve Tokenization Efficiency

acmpesuecc / Library-Search-Engine

This library search engine project aims to recommend books to users based on their interests using machine learning and latent similarity indexing.

MIT License

0 stars 0 forks source link

Improve Tokenization Efficiency #2

Open nikhila-kuchimanchi opened 5 hours ago

nikhila-kuchimanchi commented 5 hours ago

Optimize the spacy_tokenizer function to handle edge cases where the input contains special characters or non-ASCII text. Implement checks to handle such cases before tokenizing.

bunsamosa-bot[bot] commented 5 hours ago

Thank you for opening this issue! A Maintainer will review it soon!