Potentially incorrect tokenization

[GoogleCodeExporter]

When writing a tokenization unit test for the ClearTK wrappers for ClearNLP, I 
found an inconsistency between OpenNLP's tokenization and ClearNLP's.

Consider the string:
String s = "\"John & Mary's dog,\" Jane thought (to herself).\n"
                + "\"What a #$%!\n" 
                    + "a- ``I like AT&T''.\""

I was expecting the following tokenization as this is what our unit test for 
OpenNLP produces:
", John, &, Mary, 's, dog, ,, ", Jane, thought, (, to, herself, ), ., ", What, 
a, #, $, %, !, a, -, ``, I, like, AT&T, '', ., "

ClearNLP's output is slightly different:
", John, &, Mary, 's, dog, ,, ", Jane, thought, (, to, herself, ), ., ", What, 
a, #, $, %, !, a, -, `, `, I, like, AT, &, T, ', ', ., "

Specifically, the discrepancies are:
`` vs `,`
AT&T vs AT, &, T
'' vs ', '

Is this just a different style of tokenization or is it incorrect?  Does it 
make a difference for the parser? 

Original issue reported on code.google.com by lee.becker on 27 Oct 2012 at 6:01

[GoogleCodeExporter]

Additional tokenization differences.

Text: I said at 4:45pm.
Expected: [I, said, at, 4:45, pm, .]
ClearNLP: [I, said, at, 4:45pm, .]

Text:
I can't believe they wanna keep 40% of that."
        ``Whatcha think?''
        "I don't --- think so...,"
Expected: [I, ca, n't, believe, they, wan, na, keep, 40, %, of, that, ., ", ``, 
Wha, t, cha, think, ?, '', ", I, do, n't, ---, think, so, ..., ,, "]
ClearNLP: [I, ca, n't, believe, they, wan, na, keep, 40, %, of, that, ., ", `, 
`, Whatcha, think, ?, ', ', ", I, do, n't, ---, think, so, ..., ,, "]

Text: 
You `paid' US$170,000?!
        You should've paid only$16.75.
Expected: [You, `, paid, ', US$, 170,000, ?, !, You should, 've, paid, only, $, 
16.75, .]
ClearNLP: [You, `, paid, ', US, $, 170,000, ?!, You, should, 've, paid, only, 
$, 16.75, .]

Text: 
1. Buy a new Chevrolet (37%-owned in the U.S..) . 15%
Expected: [1, ., Buy, a, new, Chevrolet, (, 37%-owned, in, the, U.S, .., ), ., 
15, %]
ClearNLP: [1, ., Buy, a, new, Chevrolet, (, 37, %, -, owned, in, the, U.S, .., 
), ., 15, %]
** Though for this one, ClearNLP disagrees with the ClearTK tokenizer, and 
agrees with the Stanford parser on "37%-owned". **

Original comment by lee.becker on 27 Oct 2012 at 7:48

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Lee,

Here are my answers.  All the bugfixes will be applied to the following 
version, which is coming in next 2 days.  Thanks for finding these  bugs and 
please let me know if you find more.  I expect the tokenizer  to keep evolving!

1. The cases of `` and '' are bugs, they are fixed now.
2. The case of "AT&T" is a known bug; the LDC tokenizer handles it as the 
ClearNLP tokenizer does.  I made a simple heuristic to get around with it 
("\\p{Upper}\\&\\p{Upper}"); I didn't want to go with the dictionary approach 
for this case since that can cover only so far.
3. The case of "whatcha" should be "what cha" instead of "wha t cha".  This is 
also fixed now.
4. The case of "4:45pm." is a bug; it is fixed now.
5. "US" and "$" are now joined as "US$".
6. The case of "?!" is intentional; all period-like punctuation (e.g., ".", 
"!", and "?") are grouped as one unit in the ClearNLP tokenizer.
7. For the last example ("1. Buy …"), the ClearNLP is doing the same as the 
LDC tokenizer.  The hyphenation is a change that LDC made few years ago which I 
don't think was applied to the OpenNLP.

Jinho

Original comment by jdcho...@gmail.com on 29 Oct 2012 at 11:08