Closed juditacs closed 9 years ago
Can you please also add a README with instructions on how to run it, including how our submitted outputs (at least the one for Task 1 that didn't use machines) can be recreated?
The exact same results are not yet produced, because of the missing features. I added a new section to the readme with a very brief description.
(new_machine)recski@nessi6:~/sandbox/semeval$ cat semeval_data/sts_test/test_task2a/STS.input.headlines.txt | python semeval/paraphrases.py -c configs/twitter.cfg > out
Traceback (most recent call last):
File "semeval/paraphrases.py", line 33, in
Yes, I forgot to mention that there are two tagging modes: simple and sts. simple parses the tags from twitter input and sts uses hunpos and nltk ne chunk. Please change the option to sts and the encoding to latin1. I also added additional checking so that instead of failing it adds dummy tags if hunpos is not enabled (simple tagging mode is used).
I made the two changes, plus I changed the value of ngrams to 4 so I could reproduce the bare ngram similarity version for some sts data, and yet:
cat semeval_data/sts_test/test_task2a/STS.input.headlines.txt | python src/align_and_penalize.py --sim-type none --batch > headlines_old.out test_evaluation_task2a/correlation-noconfidence.pl test_evaluation_task2a/STS.gs.headlines.txt headlines_old.out Pearson: 0.79843
cat semeval_data/sts_test/test_task2a/STS.input.headlines.txt | python semeval/paraphrases.py -c configs/sts.cfg > headlines_new.out test_evaluation_task2a/correlation-noconfidence.pl test_evaluation_task2a/STS.gs.headlines.txt headlines_new.out Pearson: 0.59869
General
Missing stuff
What you can do