Missing samples in MPQA dataset

[luxinyu1]

When I am trying to submit submission.zip on CodaLab, this error is raised by the online scoring script:

Traceback (most recent call last):
  File "/tmp/codalab/tmpuNjjC5/run/program/evaluate.py", line 269, in 
    main()
  File "/tmp/codalab/tmpuNjjC5/run/program/evaluate.py", line 246, in main
    assert g.issubset(p), "missing some sentences: {}".format(g.difference(p))
AssertionError: missing some sentences: set([u'20011204/21.34.10-25509-3', u'20020113/03.19.43-8352-7', u'ula/110CYL068-30', u'xbank/wsj_0266-8', u'20020113/03.20.33-11983-4', u'ula/sw2078-UTF16-ms98-a-trans-211', u'ula/sw2078-UTF16-ms98-a-trans-210', u'non_fbis/16.01.33-12919-50', u'ula/110CYL067-46', u'ula/110CYL067-45', u'ula/110CYL067-44', u'20020507/17.55.53-20579-19', u'xbank/wsj_0189-15', u'20020316/20.37.48-18053-23', u'20020316/20.37.48-18053-24', u'xbank/wsj_0679-11', u'non_fbis/16.01.33-12919-49', u'20020411/22.23.02-12197-18', u'non_fbis/06.12.31-26764-20', u'ula/sw2078-UTF16-ms98-a-trans-208', u'xbank/wsj_0176-7', u'20020318/20.48.00-11907-39', u'ula/115CVL035-14', u'xbank/wsj_0557-20', u'xbank/wsj_0173-10', u'20020517/22.08.22-24562-17', u'xbank/wsj_0144-9', u'xbank/wsj_0136-7', u'20020513/21.31.14-23484-9', u'20020517/22.08.22-24562-15', u'20020517/22.08.22-24562-16', u'20020320/12.04.40-21590-29', u'xbank/wsj_0762-6', u'20011130/12.33.55-762-11', u'20011204/21.34.10-25509-4', u'non_fbis/09.35.06-27851-12', u'20020302/21.01.08-20603-7', u'non_fbis/09.35.06-27851-11', u'20020517/22.08.22-24562-18', u'xbank/wsj_1038-9', u'ula/sw2078-UTF16-ms98-a-trans-205', u'ula/sw2078-UTF16-ms98-a-trans-206', u'ula/sw2078-UTF16-ms98-a-trans-207', u'ula/sw2015-ms98-a-trans-41', u'ula/sw2015-ms98-a-trans-40', u'ula/115CVL035-17', u'ula/115CVL035-16', u'ula/110CYL068-31', u'xbank/wsj_0679-10', u'20020516/22.23.24-9583-11', u'ula/sw2078-UTF16-ms98-a-trans-209', u'20011206/21.18.24-28147-9', u'xbank/wsj_0189-17', u'20011221/20.54.40-10484-15', u'20011221/20.54.40-10484-14', u'xbank/wsj_0991-6', u'ula/110CYL200-23', u'20010620/13.40.05-15087-17', u'20020409/22.17.52-18926-11', u'20020123/21.21.45-6259-23', u'xbank/wsj_0551-4', u'non_fbis/06.12.31-26764-19', u'20010706/02.01.27-21386-7', u'xbank/wsj_0068-7', u'xbank/wsj_0189-16', u'ula/115CVL035-15', u'20020206/20.31.05-16359-20', u'xbank/wsj_0811-11', u'non_fbis/08.06.09-13335-9'])

After searching these sent_id in data/mpqa/dev.json， I found that these samples are missing, but these ids can still be found in example_submission.zip. I am wondering that if it is a bug caused by commit e63e80140d8673def09f3471c95790c988d8acd5 which modified the MPQA preprocessing script or I did something wrong while preprocessing this dataset?

[jerbarnes]

Hi,

You are correct, the example_submission.zip still had several sentence ids which had been removed from the official data. I've corrected this now and checked that it is updated. I'll close the issue, but feel free to open again if there is any problem on your end.

[luxinyu1]

Hi,

You are correct, the example_submission.zip still had several sentence ids which had been removed from the official data. I've corrected this now and checked that it is updated. I'll close the issue, but feel free to open again if there is any problem on your end.

Ummm maybe my expression was not clear enough. Actually the redundant ids in example_submission.zip is not a very important bug. The key point is, maybe the online scoring script (or data) should be updated because it forces us to submit the sentences which don't exist in current splits of MPQA and this leads to the failure of submissions.

[jerbarnes]

Sorry about that. The online data was updated before, but I had forgotten to update the example_submission.zip. That's why I assumed it was just a problem there. I just downloaded you submission to check, but it seems like you might be working with outdated data. Please pull and rerun process_mpqa.sh to make sure. You should have 2063 sentences in the dev partition (your submission only has 2008 currently).

Let me know if that doesn't work.

[luxinyu1]

Sorry about that. The online data was updated before, but I had forgotten to update the example_submission.zip. That's why I assumed it was just a problem there. I just downloaded you submission to check, but it seems like you might be working with outdated data. Please pull and rerun process_mpqa.sh to make sure. You should have 2063 sentences in the dev partition (your submission only has 2008 currently).

Let me know if that doesn't work.

Sadly, we have checked we are at the newest commit 968b077e241fbe85e4d39f0c9aeef83bcafe72b6, we even cloned the whole repo and processed the MPQA dataset again. And the dev set still only has 2008 sentences. Have you re-executed the preprocess script? Is this a problem caused by our runtime environment?

[jerbarnes]

I just tried recloning the repo and rerunning the preprocessing script and I have 2063, so it does seem like it could be caused by the runtime env. The first thing that comes to mind is if your version of stanza is the same as the one in the requirements.txt?

[luxinyu1]

I just tried recloning the repo and rerunning the preprocessing script and I have 2063, so it does seem like it could be caused by the runtime env. The first thing that comes to mind is if your version of stanza is the same as the one in the requirements.txt?

The stanza version in our env is 1.3.0. The README.md in data/ tells us to install stanza >=1.2.3 while the requirements.txt in baselines/graph_parser shows the stanza version is 1.1.0. What is the stanza version you use in your preprocessing env?

[luxinyu1]

I just tried recloning the repo and rerunning the preprocessing script and I have 2063, so it does seem like it could be caused by the runtime env. The first thing that comes to mind is if your version of stanza is the same as the one in the requirements.txt?

Maybe you could export a requiements.txt for the preprocessing scripts if you are using different environments while doing preprocessing and training, otherwise, merging these environments is also a good choice.

[jerbarnes]

Sorry, it should have all been stanza 1.1.1 throughout :/ I will update this now.

[jerbarnes]

Let me know if this solves the problem.

[luxinyu1]

Let me know if this solves the problem.

It seems that this problem has been solved through changing the version of stanza. Thanks for your constant attention.

[jerbarnes]

I'm glad to hear that it has been solved and sorry for the misunderstandings and delays. Hope the rest of the shared task goes more smoothly :)

[jerbarnes]

If that's solved the problem, I'll close the issue for now.