AngledLuffa
commented
1 year ago Can confirm, this currently happens with the model we trained from GUM + GUMReddit
import stanza
pipe = stanza.Pipeline("en", package="gum", processors="tokenize,mwt")
pipe("When we do the payroll for them, we state to them up front, that uh, we will pay the payroll, we will make the deductions, and then the employer contribution, which is approximately twenty-six percent, over and above the hourly wage, is also deducted, from the um subcontractor's check.")
Describe the bug I trained a custom stanza tokenizer and mwt on UD_English-GUM. When using the tokenizer & mwt for inference, the tokenizer changed the surface form of the word. For example, the word "subcontractor's" is tokenized as "subcontratrr 's" in the sentence:
"The college is a state-funded uh uh remodel, and on state-funded remodels, we're required to pay prevailing wages. Uh prevailing wages, that, um, that indicate different levels of agility, of the different men working. And so, uh a lot of the crews, uh, like Mitchell, who have people that work under him, around town, in regular situations, come to the people like me, and ask us to do payroll for them. When we do the payroll for them, we state to them up front, that uh, we will pay the payroll, we will make the deductions, and then the employer contribution, which is approximately twenty-six percent, over and above the hourly wage, is also deducted, from the um subcontractor's check."
To Reproduce Steps to reproduce the behavior:
Expected behavior subcontractor's -> subcontractor 's
Environment (please complete the following information):
Additional context I have also tried the newest stanza version 1.4.2, while this issue is still there.