Error when uploading the file with the predictions

[MinionAttack]

Hi,

I've uploaded the submission file and after a while I've seen that the status changed to "Failed".

Looking at the output file I can see in the "Scoring output log" something like this:

monolingual
########################################
SF1 on norec: XXXX
SF1 on multibooked_ca: XXXX
SF1 on multibooked_eu: XXXX
SF1 on opener_en: XXXX
SF1 on opener_es: XXXX

And If I see in the "Scoring error log":

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Traceback (most recent call last):
  File "/tmp/codalab/tmpA_k0h4/run/program/evaluate.py", line 269, in <module>
    main()
  File "/tmp/codalab/tmpA_k0h4/run/program/evaluate.py", line 246, in main
    assert g.issubset(p), "missing some sentences: {}".format(g.difference(p))
AssertionError: missing some sentences: set([u'20020507/22.11.06-28210-40', u'xbank/wsj_0189-15', u'xbank/wsj_0266-8', u'ula/sw2078-UTF16-ms98-a-trans-211', u'ula/sw2078-UTF16-ms98-a-trans-210', u'non_fbis/16.01.33-12919-50', u'20020507/17.55.53-20579-19', u'xbank/wsj_0762-6', u'xbank/wsj_1033-10', u'20020316/20.37.48-18053-23', u'20020316/20.37.48-18053-24', u'xbank/wsj_0679-11', u'non_fbis/16.01.33-12919-49', u'non_fbis/16.01.33-12919-48', u'non_fbis/06.12.31-26764-20', u'non_fbis/12.15.47-5091-24', u'xbank/wsj_0176-7', u'xbank/wsj_0189-17', u'xbank/wsj_0557-20', u'xbank/wsj_0173-10', u'xbank/wsj_0144-9', u'xbank/wsj_0136-7', u'20020517/22.08.22-24562-15', u'20020517/22.08.22-24562-16', u'20020517/22.08.22-24562-17', u'20011130/12.33.55-762-10', u'20011130/12.33.55-762-11', u'20011204/21.34.10-25509-4', u'non_fbis/09.35.06-27851-12', u'xbank/wsj_0068-6', u'20020320/12.04.40-21590-29', u'20020517/22.08.22-24562-18', u'20020302/21.01.08-20603-7', u'ula/sw2078-UTF16-ms98-a-trans-207', u'ula/sw2015-ms98-a-trans-41', u'ula/sw2015-ms98-a-trans-40', u'ula/115CVL035-17', u'ula/115CVL035-16', u'20020411/22.23.02-12197-18', u'ula/110CYL068-31', u'xbank/wsj_0679-10', u'ula/sw2078-UTF16-ms98-a-trans-208', u'ula/sw2078-UTF16-ms98-a-trans-209', u'xbank/wsj_1038-9', u'xbank/wsj_0991-6', u'xbank/wsj_0679-9', u'ula/110CYL200-23', u'20010620/13.40.05-15087-17', u'20020123/21.21.45-6259-23', u'xbank/wsj_0551-4', u'20010706/02.01.27-21386-7', u'xbank/wsj_0068-7', u'xbank/wsj_0189-16', u'20020507/22.11.06-28210-39', u'ula/115CVL035-15', u'xbank/wsj_0173-9', u'xbank/wsj_0811-11', u'ula/115CVL035-14'])

I see that the MPQA and darmstadt_unis datasets are missing, but I've updated the data and executed the pre-processing scripts after the changes made in the 29.11.2021 before training and predicting the files.

Before the submission I've obtained the scores locally with the file evaluate_single_dataset.py and I can get the scores for all datasets in monolingual and for the crosslingual task.

What I'm doing wrong?

Regards.

[jerbarnes]

Hey,

I just downloaded your submission and for me locally I get the same error, so it seems there are some differences in your processed MPQA data. There were some problems that came about from using different stanza versions. In the end, I had to add a requirements.txt for the data processing step as well, as there were several people who were having problems. You should have 2063 dev sentences, so maybe you can check if you have that or not?

[MinionAttack]

Seeing the requirements it's specified stanza=1.1.1 but some time ago I opened and issue #1 and the code was updated to use stanza >=1.2.3 so this is a regression?

Right now I have 2024 dev sentences.

[jerbarnes]

More than a regression, it seems there were some differences between 1.1.1 and 1.2.3 that came up when dealing with further issues. I think I've fixed some of the problems that gave the original error while using 1.1.1 in #1. Could you try with that and see if it helps? Sorry for the back and forth :/

[MinionAttack]

Installing an environment with stanza 1.1.1 gives me a dev.json with 2063 sentences.

[jerbarnes]

Great. Have you tried to resubmit?

[MinionAttack]

Not yet, I'm retraining the models.

[MinionAttack]

I have a question, now that I am using stanza 1.1.1 do I have to retrain all baseline models or just MPQA?

[jerbarnes]

MPQA is the only dataset that was affected by the stanza version issue, as far as I know.

Iago Alonso Alonso @.***> igorleak hau idatzi zuen (2021 abe. 10, or. 13:14):

I have a question, now that I am using stanza 1.1.1 do I have to retrain all baseline models or just MPQA?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jerbarnes/semeval22_structured_sentiment/issues/21#issuecomment-990923196, or unsubscribe https://github.com/notifications/unsubscribe-auth/AENMV6XR2P32EESEBJZPJJ3UQHVLXANCNFSM5JW4OQVQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[MinionAttack]

Done, now it works.