Closed jkawamoto closed 1 month ago
Thanks for finding the source of the CI errors. I agree it is our priority to fix them, so we should perhaps merge this as soon as possible. That said, I think I have fixed this already in March in a more general way. Unfortunately, I didn't have time to polish it into a PR and since then @mjpost pushed another solution (not so general) and I have forgotten all the details. I am still too busy to look at this, but if there is a volunteer, please try to merge (and review) https://github.com/martinpopel/sacreBLEU/tree/xml-domains
Thank you for the detailed explanation. I will review the branch you have been working on.
Thank you for merging my commits. Please, fix also the failing test, i.e. add domain=ALL
to the expected output.
I'm trying to fix the errors, but it might take some time.
Thank you for your advice. I was able to pass the tests by adding domain=ALL
to the four lines and removing a couple of lines. I’m not sure if removing those lines is acceptable. I’ll look for another way to solve the CI failures.
The CI errors are fixed. Could you please review this PR?
Thank you very much, once again. The last commits were exactly the missing pieces which prevented me to finish the PR in March. I am happy I could merge it into the master now.
My pleasure. I’m also glad that this PR was finally merged.
I looked through the newest CI errors and could reproduce them locally. The failure of
test_dataset::test_process_to_text
occurs whenwmt22
is chosen (since this test randomly selects 10 datasets, the failure doesn’t always happen). The failure is because the domain files are shorter than the other files in some language pairs. For example,ru-en
has this problem:Then, I checked the original file at https://github.com/wmt-conference/wmt22-news-systems/blob/main/xml/wmttest2022.ru-en.all.xml, and found that not all
doc
elements have thedomain
attribute._unwrap_wmt21_or_later
should check if the length of domains matches the lengths of the other data.