Define Function to Analyze Word Validity

[Cpgeragh]

Description: Write a function to split the generated text into words and check each word against the English word list to determine its validity.

Checklist:

• [ ] Research methods for tokenizing or splitting text into words.
• [x] Write a function that splits the generated text into individual words.
• [x] For each word, check if it exists in the English word list.
• [ ] Return the count of valid and invalid words for analysis.

[Cpgeragh]

Research

https://www.analyticsvidhya.com/blog/2019/07/how-get-started-nlp-6-unique-ways-perform-tokenization/#:~:text=In%20Python%2C%20tokenization%20in%20NLP,rules%20and%20language%2Dspecific%20considerations. https://medium.com/@ajay_khanna/tokenization-techniques-in-natural-language-processing-67bb22088c75 https://neptune.ai/blog/tokenization-in-nlp https://www.datacamp.com/blog/what-is-tokenization https://www.geeksforgeeks.org/nlp-how-tokenizing-text-sentence-words-works/