How to take account the meaning-distortion risk? How detail will the algorithm looks into the text?
Specifically, what if the highly frequent used words are not only used alone but also part of collocations. For instance, in "turn on the light", ''on purpose", "on the one hand... on the other hand", "on" may be removed according to his normalization method. However, this will distort/ remove the original meanings. Maybe when "on purpose" becomes "purpose" is still understandable, but the meaning of "turn_the light", "_the one hand" have been distorted and become ambiguous.
How to take account the meaning-distortion risk? How detail will the algorithm looks into the text? Specifically, what if the highly frequent used words are not only used alone but also part of collocations. For instance, in "turn on the light", ''on purpose", "on the one hand... on the other hand", "on" may be removed according to his normalization method. However, this will distort/ remove the original meanings. Maybe when "on purpose" becomes "purpose" is still understandable, but the meaning of "turn_the light", "_the one hand" have been distorted and become ambiguous.
Jane, 2018.1.19