-c option not functioning

[voorheese]

The -c option should force trec_eval to output results for every topic in the qrels, even when the run retrieved no documents for that topic. But currently trec_eval reports results only for topics in the intersection of the runs and qrels (the correct default behavior) even if the -c option is given.

When evaluating the run ssample.txt using the qrels file sqrels.txt and the -q and -c options, the output should contain entries for topic 37_3 but it does not. (It correctly does not contain output for 37_9 since there is no qrels for 37_9).

ssample.txt sqrels.txt

[isoboroff]

I've found the problem... in trec_eval.c:369, if the run does not return any results for a query, trec_eval just skips calculating all measures. This is not a problem for averages -- the average is still correct (note that num_q for all is 8 for this example). However, since there is no eval for the query, nothing gets printed in -q mode.

The reasonable behavior is that trec_eval should print the correct "zero value" for each metric for these queries when in -q mode. Let me work on a patch.

[isoboroff]

Revisiting this in the run-up to 10.0 release. We need to insert a zero eval, or a bogus ranking, when the topic is missing from the run file. This is a little tricky given the way trec_eval loads data.

[isoboroff]

Here is the output of my fixed behavior for -c, for the quicktest ./trec_eval -m all_trec -q -c test/qrels.test test/results.trunc. Is this what you're hoping to see?

num_rel                 301 474
num_rel_ret             301 71
map                     301 0.0324
Rprec                   301 0.1456
bpref                   301 0.1230
recip_rank              301 0.1667
iprec_at_recall_0.00    301 0.2857
iprec_at_recall_0.10    301 0.2098
iprec_at_recall_0.20    301 0.0000
iprec_at_recall_0.30    301 0.0000
iprec_at_recall_0.40    301 0.0000
iprec_at_recall_0.50    301 0.0000
iprec_at_recall_0.60    301 0.0000
iprec_at_recall_0.70    301 0.0000
iprec_at_recall_0.80    301 0.0000
iprec_at_recall_0.90    301 0.0000
iprec_at_recall_1.00    301 0.0000
P_5                     301 0.0000
P_10                    301 0.2000
P_15                    301 0.1333
P_20                    301 0.2500
P_30                    301 0.2333
P_100                   301 0.2300
P_200                   301 0.2100
P_500                   301 0.1420
P_1000                  301 0.0710
relstring               301 '0000011000'
recall_5                301 0.0000
recall_10               301 0.0042
recall_15               301 0.0042
recall_20               301 0.0105
recall_30               301 0.0148
recall_100              301 0.0485
recall_200              301 0.0886
recall_500              301 0.1498
recall_1000             301 0.1498
infAP                   301 0.0324
Rprec_mult_0.20         301 0.2211
Rprec_mult_0.40         301 0.2053
Rprec_mult_0.60         301 0.1930
Rprec_mult_0.80         301 0.1711
Rprec_mult_1.00         301 0.1456
Rprec_mult_1.20         301 0.1248
Rprec_mult_1.40         301 0.1069
Rprec_mult_1.60         301 0.0935
Rprec_mult_1.80         301 0.0831
Rprec_mult_2.00         301 0.0749
utility                 301 -358.0000
11pt_avg                301 0.0450
binG                    301 0.0238
G                       301 0.0238
ndcg                    301 0.1584
ndcg_rel                301 0.1651
Rndcg                   301 0.1567
ndcg_cut_5              301 0.0000
ndcg_cut_10             301 0.1518
ndcg_cut_15             301 0.1176
ndcg_cut_20             301 0.1985
ndcg_cut_30             301 0.1975
ndcg_cut_100            301 0.2166
ndcg_cut_200            301 0.2063
ndcg_cut_500            301 0.1584
ndcg_cut_1000           301 0.1584
map_cut_5               301 0.0000
map_cut_10              301 0.0010
map_cut_15              301 0.0010
map_cut_20              301 0.0023
map_cut_30              301 0.0033
map_cut_100             301 0.0118
map_cut_200             301 0.0208
map_cut_500             301 0.0324
map_cut_1000            301 0.0324
relative_P_5            301 0.0000
relative_P_10           301 0.2000
relative_P_15           301 0.1333
relative_P_20           301 0.2500
relative_P_30           301 0.2333
relative_P_100          301 0.2300
relative_P_200          301 0.2100
relative_P_500          301 0.1498
relative_P_1000         301 0.1498
success_1               301 0.0000
success_5               301 0.0000
success_10              301 1.0000
set_P                   301 0.1420
set_relative_P          301 0.1498
set_recall              301 0.1498
set_map                 301 0.0213
set_F                   301 0.1458
num_nonrel_judged_ret   301 188
rbp                     301 0.1861
rbp_resid               301 0.0610
unj_5                   301 0.0000
unj_10                  301 0.0000
unj_20                  301 0.1000
num_ret                 302 0
num_rel                 302 77
num_rel_ret             302 0
map                     302 0.0000
Rprec                   302 0.0000
bpref                   302 0.0000
recip_rank              302 0.0000
iprec_at_recall_0.00    302 0.0000
iprec_at_recall_0.10    302 0.0000
iprec_at_recall_0.20    302 0.0000
iprec_at_recall_0.30    302 0.0000
iprec_at_recall_0.40    302 0.0000
iprec_at_recall_0.50    302 0.0000
iprec_at_recall_0.60    302 0.0000
iprec_at_recall_0.70    302 0.0000
iprec_at_recall_0.80    302 0.0000
iprec_at_recall_0.90    302 0.0000
iprec_at_recall_1.00    302 0.0000
P_5                     302 0.0000
P_10                    302 0.0000
P_15                    302 0.0000
P_20                    302 0.0000
P_30                    302 0.0000
P_100                   302 0.0000
P_200                   302 0.0000
P_500                   302 0.0000
P_1000                  302 0.0000
relstring               302 ''
recall_5                302 0.0000
recall_10               302 0.0000
recall_15               302 0.0000
recall_20               302 0.0000
recall_30               302 0.0000
recall_100              302 0.0000
recall_200              302 0.0000
recall_500              302 0.0000
recall_1000             302 0.0000
infAP                   302 0.0000
Rprec_mult_0.20         302 0.0000
Rprec_mult_0.40         302 0.0000
Rprec_mult_0.60         302 0.0000
Rprec_mult_0.80         302 0.0000
Rprec_mult_1.00         302 0.0000
Rprec_mult_1.20         302 0.0000
Rprec_mult_1.40         302 0.0000
Rprec_mult_1.60         302 0.0000
Rprec_mult_1.80         302 0.0000
Rprec_mult_2.00         302 0.0000
utility                 302 0.0000
11pt_avg                302 0.0000
binG                    302 0.0000
G                       302 0.0000
ndcg                    302 0.0000
ndcg_rel                302 0.0000
Rndcg                   302 0.0000
ndcg_cut_5              302 0.0000
ndcg_cut_10             302 0.0000
ndcg_cut_15             302 0.0000
ndcg_cut_20             302 0.0000
ndcg_cut_30             302 0.0000
ndcg_cut_100            302 0.0000
ndcg_cut_200            302 0.0000
ndcg_cut_500            302 0.0000
ndcg_cut_1000           302 0.0000
map_cut_5               302 0.0000
map_cut_10              302 0.0000
map_cut_15              302 0.0000
map_cut_20              302 0.0000
map_cut_30              302 0.0000
map_cut_100             302 0.0000
map_cut_200             302 0.0000
map_cut_500             302 0.0000
map_cut_1000            302 0.0000
relative_P_5            302 0.0000
relative_P_10           302 0.0000
relative_P_15           302 0.0000
relative_P_20           302 0.0000
relative_P_30           302 0.0000
relative_P_100          302 0.0000
relative_P_200          302 0.0000
relative_P_500          302 0.0000
relative_P_1000         302 0.0000
success_1               302 0.0000
success_5               302 0.0000
success_10              302 0.0000
set_P                   302 0.0000
set_relative_P          302 0.0000
set_recall              302 0.0000
set_map                 302 0.0000
set_F                   302 0.0000
num_nonrel_judged_ret   302 0
rbp                     302 0.0000
rbp_resid               302 0.0000
unj_5                   302 0.0000
unj_10                  302 0.0000
unj_20                  302 0.0000
num_ret                 303 84
num_rel                 303 10
num_rel_ret             303 6
map                     303 0.2723
Rprec                   303 0.4000
bpref                   303 0.3300
recip_rank              303 0.3333
iprec_at_recall_0.00    303 0.6000
iprec_at_recall_0.10    303 0.6000
iprec_at_recall_0.20    303 0.6000
iprec_at_recall_0.30    303 0.6000
iprec_at_recall_0.40    303 0.5714
iprec_at_recall_0.50    303 0.3846
iprec_at_recall_0.60    303 0.3333
iprec_at_recall_0.70    303 0.0000
iprec_at_recall_0.80    303 0.0000
iprec_at_recall_0.90    303 0.0000
iprec_at_recall_1.00    303 0.0000
P_5                     303 0.6000
P_10                    303 0.4000
P_15                    303 0.3333
P_20                    303 0.3000
P_30                    303 0.2000
P_100                   303 0.0600
P_200                   303 0.0300
P_500                   303 0.0120
P_1000                  303 0.0060
relstring               303 '0011101000'
recall_5                303 0.3000
recall_10               303 0.4000
recall_15               303 0.5000
recall_20               303 0.6000
recall_30               303 0.6000
recall_100              303 0.6000
recall_200              303 0.6000
recall_500              303 0.6000
recall_1000             303 0.6000
infAP                   303 0.2723
Rprec_mult_0.20         303 0.0000
Rprec_mult_0.40         303 0.5000
Rprec_mult_0.60         303 0.5000
Rprec_mult_0.80         303 0.5000
Rprec_mult_1.00         303 0.4000
Rprec_mult_1.20         303 0.3333
Rprec_mult_1.40         303 0.3571
Rprec_mult_1.60         303 0.3125
Rprec_mult_1.80         303 0.3333
Rprec_mult_2.00         303 0.3000
utility                 303 -72.0000
11pt_avg                303 0.3354
binG                    303 0.2494
G                       303 0.2494
ndcg                    303 0.4730
ndcg_rel                303 0.4285
Rndcg                   303 0.4182
ndcg_cut_5              303 0.4469
ndcg_cut_10             303 0.3633
ndcg_cut_15             303 0.4211
ndcg_cut_20             303 0.4730
ndcg_cut_30             303 0.4730
ndcg_cut_100            303 0.4730
ndcg_cut_200            303 0.4730
ndcg_cut_500            303 0.4730
ndcg_cut_1000           303 0.4730
map_cut_5               303 0.1433
map_cut_10              303 0.2005
map_cut_15              303 0.2389
map_cut_20              303 0.2723
map_cut_30              303 0.2723
map_cut_100             303 0.2723
map_cut_200             303 0.2723
map_cut_500             303 0.2723
map_cut_1000            303 0.2723
relative_P_5            303 0.6000
relative_P_10           303 0.4000
relative_P_15           303 0.5000
relative_P_20           303 0.6000
relative_P_30           303 0.6000
relative_P_100          303 0.6000
relative_P_200          303 0.6000
relative_P_500          303 0.6000
relative_P_1000         303 0.6000
success_1               303 0.0000
success_5               303 1.0000
success_10              303 1.0000
set_P                   303 0.0714
set_relative_P          303 0.6000
set_recall              303 0.6000
set_map                 303 0.0429
set_F                   303 0.1277
num_nonrel_judged_ret   303 61
rbp                     303 0.3176
rbp_resid               303 0.0049
unj_5                   303 0.0000
unj_10                  303 0.0000
unj_20                  303 0.0000
runid                   all STANDARD
num_q                   all 3
num_ret                 all 584
num_rel                 all 561
num_rel_ret             all 77
map                     all 0.1016
gm_map                  all 0.0045
Rprec                   all 0.1819
bpref                   all 0.1510
recip_rank              all 0.1667
iprec_at_recall_0.00    all 0.2952
iprec_at_recall_0.10    all 0.2699
iprec_at_recall_0.20    all 0.2000
iprec_at_recall_0.30    all 0.2000
iprec_at_recall_0.40    all 0.1905
iprec_at_recall_0.50    all 0.1282
iprec_at_recall_0.60    all 0.1111
iprec_at_recall_0.70    all 0.0000
iprec_at_recall_0.80    all 0.0000
iprec_at_recall_0.90    all 0.0000
iprec_at_recall_1.00    all 0.0000
P_5                     all 0.2000
P_10                    all 0.2000
P_15                    all 0.1556
P_20                    all 0.1833
P_30                    all 0.1444
P_100                   all 0.0967
P_200                   all 0.0800
P_500                   all 0.0513
P_1000                  all 0.0257
recall_5                all 0.1000
recall_10               all 0.1347
recall_15               all 0.1681
recall_20               all 0.2035
recall_30               all 0.2049
recall_100              all 0.2162
recall_200              all 0.2295
recall_500              all 0.2499
recall_1000             all 0.2499
infAP                   all 0.1016
gm_bpref                all 0.0074
Rprec_mult_0.20         all 0.0737
Rprec_mult_0.40         all 0.2351
Rprec_mult_0.60         all 0.2310
Rprec_mult_0.80         all 0.2237
Rprec_mult_1.00         all 0.1819
Rprec_mult_1.20         all 0.1527
Rprec_mult_1.40         all 0.1547
Rprec_mult_1.60         all 0.1353
Rprec_mult_1.80         all 0.1388
Rprec_mult_2.00         all 0.1250
utility                 all -143.3333
11pt_avg                all 0.1268
binG                    all 0.0911
G                       all 0.0911
ndcg                    all 0.2105
ndcg_rel                all 0.1979
Rndcg                   all 0.1916
ndcg_cut_5              all 0.1490
ndcg_cut_10             all 0.1717
ndcg_cut_15             all 0.1796
ndcg_cut_20             all 0.2238
ndcg_cut_30             all 0.2235
ndcg_cut_100            all 0.2299
ndcg_cut_200            all 0.2264
ndcg_cut_500            all 0.2105
ndcg_cut_1000           all 0.2105
map_cut_5               all 0.0478
map_cut_10              all 0.0671
map_cut_15              all 0.0800
map_cut_20              all 0.0915
map_cut_30              all 0.0919
map_cut_100             all 0.0947
map_cut_200             all 0.0977
map_cut_500             all 0.1016
map_cut_1000            all 0.1016
relative_P_5            all 0.2000
relative_P_10           all 0.2000
relative_P_15           all 0.2111
relative_P_20           all 0.2833
relative_P_30           all 0.2778
relative_P_100          all 0.2767
relative_P_200          all 0.2700
relative_P_500          all 0.2499
relative_P_1000         all 0.2499
success_1               all 0.0000
success_5               all 0.3333
success_10              all 0.6667
set_P                   all 0.0711
set_relative_P          all 0.2499
set_recall              all 0.2499
set_map                 all 0.0214
set_F                   all 0.0912
num_nonrel_judged_ret   all 249
rbp                     all 0.1679
rbp_resid               all 0.0220
unj_5                   all 0.0000
unj_10                  all 0.0000
unj_20                  all 0.0333