Comparing the UD EWT and Propbank EWT, I found 10 cases of tokenization mismatches. In all cases of the first code block below, I believe UD EWT is doing the right thing. In answers-20111107175720AAlb2TB_ans-0015 maybe we need the CorectForm=basically in the MISC.
1)
# sent_id = answers-20111107175720AAlb2TB_ans-0015
# text = The Irish weather works on the ‘four seasons in a day’ principle, which basic<U+00AD>ally means that you can’t predict a thing when it comes to the behaviour of the sky.
...
17 basic<U+00AD>ally basically ADV RB _ 18 advmod 18:advmod _
...
...
google/ewt/answers/00/20111107175720AAlb2TB_ans.xml 14 16 basic GW (S(ADVP* - - * (ARGM-ADV* * *
google/ewt/answers/00/20111107175720AAlb2TB_ans.xml 14 17 ally RB *) - - * *) * *
...
2)
# sent_id = answers-20111108110012AAK8Azy_ans-0029
# newpar id = answers-20111108110012AAK8Azy_ans-p0004
# text = mm it depends on the size of his tank or cage or whatever you have him in..atleast as far as i'm concerned.
...
19 at at ADP IN ExtPos=ADV 22 advmod 22:advmod CorrectSpaceAfter=Yes|SpaceAfter=No
20 least least ADJ JJS Degree=Sup 19 fixed 19:fixed _
...
...
google/ewt/answers/00/20111108110012AAK8Azy_ans.xml 28 18 atleast RB (ADVP(ADVP* - - (ARGM-ADV* * * *
...
3)
# sent_id = reviews-042012-0006
# text = I will 4-ever be eternally grateful for their hospitality and luv that my Sicilian family showed me when I was there for 3 years.
1 I I PRON PRP Case=Nom|Number=Sing|Person=1|PronType=Prs 6 nsubj 6:nsubj _
2 will will AUX MD VerbForm=Fin 6 aux 6:aux _
3 4-ever forever ADV RB Abbr=Yes 6 advmod 6:advmod _
...
google/ewt/reviews/00/042012.xml 5 0 I PRP (TOP(S(NP*) - - (ARG1*) (ARG0*) * *
google/ewt/reviews/00/042012.xml 5 1 will MD (VP* - - (ARGM-MOD*) (ARGM-MOD*) * *
google/ewt/reviews/00/042012.xml 5 2 4 GW (ADVP* - - (ARGM-TMP* (ARGM-TMP* * *
google/ewt/reviews/00/042012.xml 5 3 - HYPH * - - * * * *
google/ewt/reviews/00/042012.xml 5 4 ever RB *) - - *) *) * *
...
4)
# sent_id = answers-20111107094641AATmyb6_ans-0036
# text = Have a good trip where-ever you end up.
...
5 where-ever wherever ADV WRB Typo=Yes 7 advmod 7:advmod CorrectForm=wherever
...
...
google/ewt/answers/00/20111107094641AATmyb6_ans.xml 35 4 where GW (SBAR(WHADVP* - - * (ARG1* (ARG2*
google/ewt/answers/00/20111107094641AATmyb6_ans.xml 35 5 - HYPH * - - * * *
google/ewt/answers/00/20111107094641AATmyb6_ans.xml 35 6 ever WRB *) - - * * *)
...
5)
# sent_id = email-enronsent32_01-0050
# text = My new address will be: Mary Hain Senior Regulatory Counsel ISO New England Inc.One Sullivan Road Holyoke, MA 01040-2841 (413) 535-4000 mhain@ISO-NE.com
...
15 Inc. Inc. PROPN NNP Number=Sing 8 list 8:list CorrectSpaceAfter=Yes|SpaceAfter=No
16 One One NUM NNP NumType=Card 17 nummod 17:nummod _
...
...
google/ewt/email/00/enronsent32_01.xml 49 14 Inc.One NNP *) - - *
...
For the next block, I just want to confirm the decisions in UD EWT:
6)
# sent_id = reviews-096340-0002
# text = Hancocks is one of four fabric stores in Fort Smith.
1-2 Hancocks _ _ _ _ _ _ _ _
1 Hancock Hancock PROPN NNP Number=Sing 4 nsubj 4:nsubj _
2 s 's PART POS Typo=Yes 1 case 1:case CorrectForm='s
...
google/ewt/reviews/00/096340.xml 1 0 Hancocks NNP (TOP(S(NP*) - - (ARG1*)
google/ewt/reviews/00/096340.xml 1 1 is VBZ (VP* be be.01 (V*)
...
7)
# sent_id = reviews-009389-0003
# text = it is now bislas.
1 it it PRON PRP Case=Nom|Gender=Neut|Number=Sing|Person=3|PronType=Prs 4 nsubj 4:nsubj _
2 is be AUX VBZ Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin 4 cop 4:cop _
3 now now ADV RB _ 4 advmod 4:advmod _
4-5 bislas _ _ _ _ _ _ _ SpaceAfter=No
4 bisla bisla PROPN NNP Number=Sing 0 root 0:root _
5 s 's PART POS Typo=Yes 4 case 4:case CorrectForm='s
6 . . PUNCT . _ 4 punct 4:punct _
google/ewt/reviews/00/009389.xml 2 0 it PRP (TOP(S(NP*) - - (ARG1*)
google/ewt/reviews/00/009389.xml 2 1 is VBZ (VP* be be.01 (V*)
google/ewt/reviews/00/009389.xml 2 2 now RB (ADVP*) - - (ARGM-TMP*)
google/ewt/reviews/00/009389.xml 2 3 bislas NNP (NP*)) - - (ARG2*)
google/ewt/reviews/00/009389.xml 2 4 . . *)) - - *
8)
# sent_id = reviews-209465-0001
# newpar id = reviews-209465-p0001
# text = Pho-nomenal!!
1 Pho-nomenal phenomenal ADJ JJ Degree=Pos|Style=Expr 0 root 0:root CorrectForm=Phenomenal|SpaceAfter=No
2 !! !! PUNCT . _ 1 punct 1:punct _
google/ewt/reviews/00/209465.xml 0 0 Pho NN (TOP(FRAG(ADJP* - -
google/ewt/reviews/00/209465.xml 0 1 - HYPH * - -
google/ewt/reviews/00/209465.xml 0 2 nomenal JJ *) - -
google/ewt/reviews/00/209465.xml 0 3 !! . *)) - -
9)
# sent_id = answers-20111108104724AAuBUR7_ans-0032
# text = Worse comes to worse, I'm out of the $$ and have to put him down anyway or have a pretty pasture ornament.
...
11 $$ $$ SYM $ _ 0 root 0:root _
...
...
google/ewt/answers/00/20111108104724AAuBUR7_ans.xml 31 10 $ $ * - - * * * * *
google/ewt/answers/00/20111108104724AAuBUR7_ans.xml 31 11 $ $ *)))) - - * *) * * *
...
10)
# sent_id = answers-20111106213308AA5Nh2g_ans-0008
# text = You voted on the Dominion Posts website.
1 You you PRON PRP Case=Nom|Person=2|PronType=Prs 2 nsubj 2:nsubj _
2 voted vote VERB VBD Mood=Ind|Number=Sing|Person=2|Tense=Past|VerbForm=Fin 0 root 0:root _
3 on on ADP IN _ 8 case 8:case _
4 the the DET DT Definite=Def|PronType=Art 8 det 8:det _
5 Dominion Dominion PROPN NNP Number=Sing 6 compound 6:compound _
6-7 Posts _ _ _ _ _ _ _ _
6 Post Post PROPN NNP Number=Sing 8 compound 8:compound _
7 s 's PART POS Typo=Yes 6 case 6:case CorrectForm='s
8 website website NOUN NN Number=Sing 2 obl 2:obl:on SpaceAfter=No
9 . . PUNCT . _ 2 punct 2:punct _
google/ewt/answers/00/20111106213308AA5Nh2g_ans.xml 7 0 You PRP (TOP(S(NP*) - - (ARG0*)
google/ewt/answers/00/20111106213308AA5Nh2g_ans.xml 7 1 voted VBD (VP* vote vote.01 (V*)
google/ewt/answers/00/20111106213308AA5Nh2g_ans.xml 7 2 on IN (PP* - - (ARGM-LOC*
google/ewt/answers/00/20111106213308AA5Nh2g_ans.xml 7 3 the DT (NP* - - *
google/ewt/answers/00/20111106213308AA5Nh2g_ans.xml 7 4 Dominion NNP (NML* - - *
google/ewt/answers/00/20111106213308AA5Nh2g_ans.xml 7 5 Posts NNPS *) - - *
google/ewt/answers/00/20111106213308AA5Nh2g_ans.xml 7 6 website NN *))) - - *)
google/ewt/answers/00/20111106213308AA5Nh2g_ans.xml 7 7 . . *)) - - *
Comparing the UD EWT and Propbank EWT, I found 10 cases of tokenization mismatches. In all cases of the first code block below, I believe UD EWT is doing the right thing. In
answers-20111107175720AAlb2TB_ans-0015
maybe we need the CorectForm=basically in the MISC.For the next block, I just want to confirm the decisions in UD EWT: