Add function to remove tags after processing

cbaziotis / ekphrasis

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

MIT License

661 stars 90 forks source link

Add function to remove tags after processing #14

Closed mobiuscreek closed 5 years ago

mobiuscreek commented 5 years ago

Description

Just added a small function to remove tags after annotation and/or normalization. I wanted to have the sentences return without having the tags for further processing. For example after fixing an elongated word. Not the ideal solution but it gets the job done.

Added function remove_tags in helpers.py
Added function call from the preprocessor

To do

Test function remove_tags
Code needs linting

cbaziotis commented 5 years ago

Thanks!