Definintion of tokenizer in 8.3 Working with bigger data

[niwak01]

I think it should be modified as the comment below .

def tokenizer(text):
    text = re.sub('<[^>]*>', '', text)
    #emoticons = re.findall('(?::|;|=)(?:-)?(?:\)|\(|D|P)', text.lower())                 # "text.lower()" should be "text"
    emoticons = re.findall('(?::|;|=)(?:-)?(?:\)|\(|D|P)', text)
    text = re.sub('[\W]+', ' ', text.lower()) +\
        ' '.join(emoticons).replace('-', '')
    tokenized = [w for w in text.split() if w not in stop]
    return tokenized

[rasbt]

Thanks for the suggestion!

Hm, I don't recall why I used text.lower() up there and wonder if there was a specific reason, or whether it's really unnecessary as you point out 🤔.

Doing a quick test, it seems like they both produce the same results. But yeah, maybe there was a special edge case that I had in mind. Hmm...

[niwak01]

Thank you very much for your response.

I think you might have intended to write as follows emoticons = re.findall('(?::|;|=)(?:-)?(?:)|(|d|p)', text.lower()) in order to find not only ':-P', but also ':-p)'

[rasbt]

Oh this makes absolute sense, thanks a lot! Changed it!