Reader evaluation bug

[BojanKovachki]

There is a bug in the evaluate_reader(cdqa_pipeline, 'path-to-annotated-dataset.json') function.

I tried it in Colab and I get the following error:

KeyError                                  Traceback (most recent call last)
<ipython-input-28-252ef3ba7353> in <module>()
----> 1 evaluate_reader(cdqa_pipeline, 'annotated-dataset.json')

/usr/local/lib/python3.6/dist-packages/cdqa/utils/evaluation.py in evaluate_reader(cdqa_pipeline, dataset_file, expected_version)
    127     with open(dataset_file, "r") as dataset_file:
    128         dataset_json = json.load(dataset_file)
--> 129         #if dataset_json["version"] != expected_version:
    130             #print(
    131                 #"Evaluation expects v-"

KeyError: 'version'

[andrelmfarias]

Hi, Can you please add the following key-value pair to your json / dictionnary? "version": "1.1"

It should work

[BojanKovachki]

Hi @andrelmfarias,

a new error appears:

AttributeError                            Traceback (most recent call last)
<ipython-input-35-d4ebafb56682> in <module>()
----> 1 evaluate_reader(cdqa_pipeline, 'annotated-dataset.json')

2 frames
/usr/local/lib/python3.6/dist-packages/cdqa/reader/bertqa_sklearn.py in write_predictions(all_examples, all_features, all_results, n_best_size, max_answer_length, do_lower_case, output_prediction_file, output_nbest_file, output_null_log_odds_file, verbose_logging, version_2_with_negative, null_score_diff_threshold, retriever_score_weight, n_predictions)
    786         best_dict["title"] = example.title
    787         best_dict["paragraph"] = example.paragraph
--> 788         best_dict["retriever_score"] = example.retriever_score.item()
    789         best_dict["final_score"] = (1 - retriever_score_weight) * (
    790             best_dict["start_logit"] + best_dict["end_logit"]

AttributeError: 'int' object has no attribute 'item'

[andrelmfarias]

Hi,

I know why there's this bug. Il will be opening a PR with a fix for it today.