search

jerbarnes / semeval22_structured_sentiment

SemEval-2022 Shared Task 10: Structured Sentiment Analysis
75 stars 42 forks source link

AssertionError: missing some sentences #10

Closed akkarimi closed 3 years ago

akkarimi commented 3 years ago

I'm trying to submit some results, but I get assertion error telling me that some sentences are missing. I tried it with a previously successful submission and I still got the same error. Is there a problem with codalab evaluation?

jerbarnes commented 3 years ago

Hi,

We have recently updated the darmstadt and mpqa datasets, which is probably what led to this error. You can check the sentence ids in the example_submission.zip and make sure that yours coincide. It might be worth making sure that the github is up-to-date. Let me know if that solves the issue.

akkarimi commented 3 years ago

Hi, Thanks for the reply. Trying to test what you said but my submission on codalab has been in the "submitted" status for a couple of hours. Refreshing doesn't help!!

jerbarnes commented 3 years ago

Seems like it might be a codalab problem. I just submitted the example_submission.zip and it worked.

akutuzov commented 3 years ago

Hi @akkarimi In your submissions, there are 1622 sentences in the MPQA dataset, while there should be 2063 of them. Check why is it like that.

akkarimi commented 3 years ago

Hi @akutuzov. Thanks for the update. You're right. I was using the example_submission file from the forked version of this repo.