search

allenai / allennlp-reading-comprehension

81 stars 25 forks source link

Performance decrease after allennlp's update #24

Closed eladsegal closed 4 years ago

eladsegal commented 4 years ago

Not sure if it belongs here or in allennlp's repo.

With the commit https://github.com/allenai/allennlp-reading-comprehension/commit/e4f8e5df4f9fa35287d44e94fc8b26b9cabed0a5 I got

{"em": 0.8045411542100284, "f1": 0.8829992175395974}

on SQuAD with the transformer_squad.jsonnet configuration. I ran it multiple times, also with different seeds and got pretty much the same results.

After the latest commit (https://github.com/allenai/allennlp-reading-comprehension/commit/56c2d7066ee14a3d1cd9cdd4da88acd808a0544e), I got worse results with the same configuration:

{"em": 0.7550614947965941, "f1": 0.8392231569459155}

I ran it multiple times as well and got the same results.

eladsegal commented 4 years ago

Performance is good again after https://github.com/allenai/allennlp-reading-comprehension/commit/e7c217c714d8c0e843719ea320b64691071b4a30