Closed GoogleCodeExporter closed 9 years ago
The BreakIteratorSegmenter is more technically motivated that linguistically.
This is just how the underlying Java BreakIterator behaves.
We could consider to to add a parameter to directly merge word and hyphen
tokens if they are directly adjacent and the token following the hypen is not
directly adjacent.
I did a little test with the other segmenters on the following example:
Input: "ihre Negativbei- spiele immer"
Expected: "ihre", "Negativbei-", "spiele", "immer"
BreakIteratorSegmenter: "-" is separate token
OpenNlpSegmenter: as expected
StanfordSegmenter: "-" is separate token
LanguageToolSegmenter: as expected
Original comment by richard.eckart
on 21 Oct 2012 at 11:28
Original comment by richard.eckart
on 16 Feb 2013 at 10:56
Original comment by richard.eckart
on 16 Feb 2013 at 11:04
So I'm not sure what to do here. It's the case that the BreakIterator does
split like this. We have other segmenters, that are smarter. I'd just close
this issue as WontFix...
Original comment by richard.eckart
on 18 Mar 2013 at 4:44
StanfordSegmenter also splits the hyphen into a separate token.
OpenNlpSegmenter and LanguageToolSegmenter do not.
Original comment by richard.eckart
on 19 Mar 2013 at 6:52
Closing this. It's how the BreatIterator works. Use a different segmenter if
this behavior is not good for a certain use case.
Original comment by richard.eckart
on 24 Jun 2013 at 10:45
Original issue reported on code.google.com by
eckle.kohler
on 18 Oct 2012 at 6:22Attachments: