error in evaluation code

[laura-dietz]

I am one step further. I produced output by running

python ef_evaluate.py /home/ben/trec-car/data/enwiki-20161220/release-v1.4/test200set/all.test200.cbor.hierarchical.qrels output-test200.run

now I get an error in the evaluation code

dietz@jelly:~/datasci/code2/team1$ python ef_evaluate.py /home/ben/trec-car/data/enwiki-20161220/release-v1.4/test200set/all.test200.cbor.hierarchical.qrels output-test200.run runid all test num_q all 2382 num_ret all 238200 num_rel_ret all 5357 Traceback (most recent call last): File "ef_evaluate.py", line 136, in

print("map" + "\t all \t" + str(mean_average_precision())) File "ef_evaluate.py", line 131, in mean_average_precision sum += average_precision(res) File "ef_evaluate.py", line 106, in average_precision obj2 = result_of_each_query[query_id] KeyError: 'Cigna/References'

[laura-dietz]

I evaluated your code with trec_eval -- here the numbers: dietz@jelly:~/datasci/code2/team1$ ../trec_eval /home/ben/trec-car/data/enwiki-20161220/release-v1.4/test200set/all.test200.cbor.hierarchical.qrels output-test200.run runid all test num_q all 1879 num_ret all 187900 num_rel all 4489 num_rel_ret all 80 map all 0.0069 gm_map all 0.0000 Rprec all 0.0063 bpref all 0.0170 recip_rank all 0.0160 iprec_at_recall_0.00 all 0.0161 iprec_at_recall_0.10 all 0.0147 iprec_at_recall_0.20 all 0.0106 iprec_at_recall_0.30 all 0.0087 iprec_at_recall_0.40 all 0.0071 iprec_at_recall_0.50 all 0.0071 iprec_at_recall_0.60 all 0.0033 iprec_at_recall_0.70 all 0.0033 iprec_at_recall_0.80 all 0.0033 iprec_at_recall_0.90 all 0.0033 iprec_at_recall_1.00 all 0.0033 P_5 all 0.0036 P_10 all 0.0019 P_15 all 0.0014 P_20 all 0.0012 P_30 all 0.0009 P_100 all 0.0004 P_200 all 0.0002 P_500 all 0.0001 P_1000 all 0.0000

[gauravpatil93]

Working on tracing the issue, it appears to work with smaller data sets but fails on bigger ones.