Calculate precision and recall

[cleong110]

https://stackoverflow.com/questions/28154321/calculating-precision-and-recall-in-click-data

[cleong110]

https://github.com/jcklie/ireval maybe?

[cleong110]

Based on https://stackoverflow.com/questions/28154321/calculating-precision-and-recall-in-click-data I guess I just divide my "match count" by k to get precision

Precision = No. of relevant documents retrieved / No. of total documents retrieved

and match count over number of valid answers to get recall:

Recall = No. of relevant documents retrieved / No. of total relevant documents

...which I can do pretty easy with the code I have

[cleong110]

https://github.com/DavidRFerreira/InformationRetrieval_EvaluationMetrics?tab=readme-ov-file#explanation also

[cleong110]

https://github.com/facebookresearch/fairseq/blob/main/examples/MMPT/mmpt/evaluators/metric.py

[cleong110]

Done for report. Parsed the output logs, extracting the info. Really must refactor the code to output in a dataframe or such. https://docs.google.com/spreadsheets/d/1F-JmD7IEOtNU8Tx8KvYeEJKNvlhqDtFNI46GSveO9RY/edit?gid=0#gid=0

Subset  Model   File    Trials run  Dataset size    K (number of documents retrieved)   No. of total relevant documents Mean matches    Precision = No. of relevant documents retrieved / No. of total documents retrieved  Recall = No. of relevant documents retrieved / No. of total relevant documents
ASL Citizen 400 words, 10 examples each asl-citizen ./search_results_asl_citizen_400_words_10_examples_each_asl-citizen.txt 3990    3990    10  8.965914787 5.649   0.5649  0.6300528317
ASL Citizen 400 words, 10 examples each asl-signs   ./search_results_asl_citizen_400_words_10_examples_each_asl-signs.txt   3990    3990    10  8.965914787 2.52    0.252   0.2810644602
ASL Citizen 400 words, 10 examples each baseline_temporal   ./search_results_asl_citizen_400_words_10_examples_each_baseline_temporal.txt   3990    3990    10  8.965914787 0.351   0.0351  0.0391482641
ASL Citizen 400 words, 10 examples each sem-lex ./search_results_asl_citizen_400_words_10_examples_each_sem-lex.txt 3990    3990    10  8.965914787 4.089   0.4089  0.4560605468
ASL Citizen 400 words, 10 examples each Random guessing N/A 3990    3990    10  8.965914787 0.02    0.002   0.002230670319
ASL Citizen 20 words, 5 examples each   asl-citizen ./search_results_20_words_5_examples_asl-citizen.txt    100 100 10  4   3.8 0.38    0.95
ASL Citizen 20 words, 5 examples each   asl-signs   ./search_results_20_words_5_examples_asl-signs.txt  100 100 10  4   3.35    0.335   0.8375
ASL Citizen 20 words, 5 examples each   baseline_temporal   ./search_results_20_words_5_examples_baseline_temporal.txt  100 100 10  4   1.32    0.132   0.33
ASL Citizen 20 words, 5 examples each   sem-lex ./search_results_20_words_5_examples_sem-lex.txt    100 100 10  4   3.51    0.351   0.8775
ASL Citizen 20 words, 5 examples each   Random guessing N/A 100 100 10  4   0.4 0.04    0.1
ASL Citizen full set    asl-citizen ./search_results_asl_citizen_full_set_using_model_asl-citizen.txt   83116   83116   10  29.66906492 5.262   0.5262  0.177356449
ASL Citizen full set    asl-signs   ./search_results_asl_citizen_full_set_using_model_asl-signs.txt 83116   83116   10  29.66906492 1.939   0.1939  0.06535426732
ASL Citizen full set    baseline_temporal   ./search_results_asl_citizen_full_set_using_model_baseline_temporal.txt 14822   83116   10  29.66906492 0.144   0.01440 0.004854977388
ASL Citizen full set    sem-lex ./search_results_asl_citizen_full_set_using_model_sem-lex.txt   83116   83116   10  29.66906492 3.601   0.3601  0.1213722107
ASL Citizen full set    Random guessing N/A 83116   83116   10  29.66906492 0.004   0.0004  0.0001348205618
ASL citizen Test set    asl-citizen ./search_results_asl_citizen_test_set_using_model_asl-citizen.txt   32917   32917   10  11  2.805   0.2805  0.255
ASL citizen Test set    asl-signs   ./search_results_asl_citizen_test_set_using_model_asl-signs.txt 32917   32917   10  11  1.064   0.1064  0.09672727273
ASL citizen Test set    baseline_temporal   ./search_results_asl_citizen_test_set_using_model_baseline_temporal.txt 32917   32917   10  11  0.083   0.0083  0.007545454545
ASL citizen Test set    sem-lex ./search_results_asl_citizen_test_set_using_model_sem-lex.txt   32917   32917   10  11  2.077   0.2077  0.1888181818
ASL citizen Test set    Random guessing N/A 32917   32917   10  11  0.003   0.0003  0.0002727272727