search

larsga / whazzup

Automatically exported from code.google.com/p/whazzup
0 stars 0 forks source link

Tokenize foo-bar into three tokens #69

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
In many languages, compound terms cause difficulties because they are rare, 
while the terms they're made up from are not. So "foo-bar" should become 
"foo-bar", "foo", and "bar".

Original issue reported on code.google.com by lar...@gmail.com on 16 Jan 2013 at 9:05