Drop portuguese, modify defaults to be Spanish. Add command to perform experiments for each function over thresholds - returning optimal threshold and expected F1 score. Add command to perform 1-shot or k-fold evaluations using optimization as integral part.
Experiments used threshold values of .1 to .8 in steps of 0.1. To select optimum the best 4 results were selected (always contiguous) and a cubic polynomial fit the data. Best fit was returned within the range of the 4 best results.
1-shot and k-fold cross-validation was performed over the data. Cross validation in this case requires that sampling be by concept, since pairwise comparisons are made across all languages and words within a concept. There is some added variability to test results due to using concept partitions since borrowing varies somewhat by concept. But far superior to the incorrect results from not partitioning by concept (where F1 score was proportional to 1/k).
Here are use examples:
% cldfbench sabor.run_pairwise_experiments --function SCA INFO Trail result: ['SCA', 0.1, -1, 455, 8752, 16, 1107, 0.966, 0.291, 0.448, 0.891] INFO Trail result: ['SCA', 0.2, -1, 726, 8743, 25, 836, 0.967, 0.465, 0.628, 0.917] INFO Trail result: ['SCA', 0.3, -1, 924, 8718, 50, 638, 0.949, 0.592, 0.729, 0.933] INFO Trail result: ['SCA', 0.4, -1, 1076, 8623, 145, 486, 0.881, 0.689, 0.773, 0.939] INFO Trail result: ['SCA', 0.5, -1, 1180, 8309, 459, 382, 0.72, 0.755, 0.737, 0.919] INFO Trail result: ['SCA', 0.6, -1, 1273, 7477, 1291, 289, 0.496, 0.815, 0.617, 0.847] INFO Trail result: ['SCA', 0.7, -1, 1367, 5676, 3092, 195, 0.307, 0.875, 0.454, 0.682] INFO Trail result: ['SCA', 0.8, -1, 1466, 3013, 5755, 96, 0.203, 0.939, 0.334, 0.434] INFO ## Wrote results to /Users/johnmiller/ling/sabor-installs/sabor/store/exp_results_SCA.tsv INFO Estimated optimum threshold and f1 score (0.40, 0.773) INFO Observed threshold and f1 score (0.40, 0.773)
and
`% cldfbench sabor.cross_validate_pairwise --cv k-fold --k 3 --function NED
2022-05-02 16:40:58,586 [INFO] Data has been written to file </var/folders/79/2ckx3nkx6y76__jfdd13q9tw0000gn/T/tmp5hv83nod/full-wl.tsv>.
partition: 0 3 0 436 8124 3976
INFO Trail result: ['NED', 0.1, None, 77, 5900, 2, 945, 0.975, 0.075, 0.14, 0.863]
INFO Trail result: ['NED', 0.2, None, 143, 5900, 2, 879, 0.986, 0.14, 0.245, 0.873]
INFO Trail result: ['NED', 0.3, None, 287, 5898, 4, 735, 0.986, 0.281, 0.437, 0.893]
INFO Trail result: ['NED', 0.4, None, 384, 5893, 9, 638, 0.977, 0.376, 0.543, 0.907]
INFO Trail result: ['NED', 0.5, None, 497, 5888, 14, 525, 0.973, 0.486, 0.648, 0.922]
INFO Trail result: ['NED', 0.6, None, 644, 5856, 46, 378, 0.933, 0.63, 0.752, 0.939]
INFO Trail result: ['NED', 0.7, None, 755, 5659, 243, 267, 0.757, 0.739, 0.748, 0.926]
INFO Trail result: ['NED', 0.8, None, 827, 5104, 798, 195, 0.509, 0.809, 0.625, 0.857]
INFO ## Wrote results to /Users/johnmiller/ling/sabor-installs/sabor/store/exp_results_NED.tsv
INFO Train result: ['NED', 0.64, None, 700, 5774, 128, 322, 0.845, 0.685, 0.757, 0.935]
INFO Test result: ['NED', 0.64, None, 354, 2803, 63, 186, 0.849, 0.656, 0.74, 0.927]
partition: 1 3 436 872 8014 4086
INFO Trail result: ['NED', 0.1, None, 77, 5812, 2, 950, 0.975, 0.075, 0.139, 0.861]
INFO Trail result: ['NED', 0.2, None, 140, 5811, 3, 887, 0.979, 0.136, 0.239, 0.87]
INFO Trail result: ['NED', 0.3, None, 280, 5807, 7, 747, 0.976, 0.273, 0.426, 0.89]
INFO Trail result: ['NED', 0.4, None, 373, 5801, 13, 654, 0.966, 0.363, 0.528, 0.902]
INFO Trail result: ['NED', 0.5, None, 476, 5801, 13, 551, 0.973, 0.463, 0.628, 0.918]
INFO Trail result: ['NED', 0.6, None, 630, 5760, 54, 397, 0.921, 0.613, 0.736, 0.934]
INFO Trail result: ['NED', 0.7, None, 739, 5569, 245, 288, 0.751, 0.72, 0.735, 0.922]
INFO Trail result: ['NED', 0.8, None, 819, 5041, 773, 208, 0.514, 0.797, 0.625, 0.857]
INFO ## Wrote results to /Users/johnmiller/ling/sabor-installs/sabor/store/exp_results_NED.tsv
INFO Train result: ['NED', 0.64, None, 691, 5690, 124, 336, 0.848, 0.673, 0.75, 0.933]
INFO Test result: ['NED', 0.64, None, 363, 2887, 67, 172, 0.844, 0.679, 0.752, 0.931]
partition: 2 3 872 1308 8062 4038
INFO Trail result: ['NED', 0.1, None, 68, 5818, 2, 1007, 0.971, 0.063, 0.119, 0.854]
INFO Trail result: ['NED', 0.2, None, 129, 5817, 3, 946, 0.977, 0.12, 0.214, 0.862]
INFO Trail result: ['NED', 0.3, None, 275, 5811, 9, 800, 0.968, 0.256, 0.405, 0.883]
INFO Trail result: ['NED', 0.4, None, 385, 5808, 12, 690, 0.97, 0.358, 0.523, 0.898]
INFO Trail result: ['NED', 0.5, None, 503, 5803, 17, 572, 0.967, 0.468, 0.631, 0.915]
INFO Trail result: ['NED', 0.6, None, 654, 5764, 56, 421, 0.921, 0.608, 0.733, 0.931]
INFO Trail result: ['NED', 0.7, None, 768, 5564, 256, 307, 0.75, 0.714, 0.732, 0.918]
INFO Trail result: ['NED', 0.8, None, 856, 5069, 751, 219, 0.533, 0.796, 0.638, 0.859]
INFO ## Wrote results to /Users/johnmiller/ling/sabor-installs/sabor/store/exp_results_NED.tsv
INFO Train result: ['NED', 0.64, None, 717, 5690, 130, 358, 0.847, 0.667, 0.746, 0.929]
INFO Test result: ['NED', 0.64, None, 337, 2887, 61, 150, 0.847, 0.692, 0.762, 0.939]
Dataset Function Threshold tp tn fp fn precision recall F1 score accuracy
Test NED 0.640 354.000 2803.000 63.000 186.000 0.849 0.656 0.740 0.927
Test NED 0.640 363.000 2887.000 67.000 172.000 0.844 0.679 0.752 0.931
Test NED 0.640 337.000 2887.000 61.000 150.000 0.847 0.692 0.762 0.939
Average NED 0.640 351.333 2859.000 63.667 169.333 0.847 0.676 0.751 0.932
StDev NED 0.640 13.204 48.497 3.055 18.148 0.003 0.018 0.011 0.006`
Drop portuguese, modify defaults to be Spanish. Add command to perform experiments for each function over thresholds - returning optimal threshold and expected F1 score. Add command to perform 1-shot or k-fold evaluations using optimization as integral part.
Experiments used threshold values of .1 to .8 in steps of 0.1. To select optimum the best 4 results were selected (always contiguous) and a cubic polynomial fit the data. Best fit was returned within the range of the 4 best results.
1-shot and k-fold cross-validation was performed over the data. Cross validation in this case requires that sampling be by concept, since pairwise comparisons are made across all languages and words within a concept. There is some added variability to test results due to using concept partitions since borrowing varies somewhat by concept. But far superior to the incorrect results from not partitioning by concept (where F1 score was proportional to 1/k).
Here are use examples:
% cldfbench sabor.run_pairwise_experiments --function SCA INFO Trail result: ['SCA', 0.1, -1, 455, 8752, 16, 1107, 0.966, 0.291, 0.448, 0.891] INFO Trail result: ['SCA', 0.2, -1, 726, 8743, 25, 836, 0.967, 0.465, 0.628, 0.917] INFO Trail result: ['SCA', 0.3, -1, 924, 8718, 50, 638, 0.949, 0.592, 0.729, 0.933] INFO Trail result: ['SCA', 0.4, -1, 1076, 8623, 145, 486, 0.881, 0.689, 0.773, 0.939] INFO Trail result: ['SCA', 0.5, -1, 1180, 8309, 459, 382, 0.72, 0.755, 0.737, 0.919] INFO Trail result: ['SCA', 0.6, -1, 1273, 7477, 1291, 289, 0.496, 0.815, 0.617, 0.847] INFO Trail result: ['SCA', 0.7, -1, 1367, 5676, 3092, 195, 0.307, 0.875, 0.454, 0.682] INFO Trail result: ['SCA', 0.8, -1, 1466, 3013, 5755, 96, 0.203, 0.939, 0.334, 0.434] INFO ## Wrote results to /Users/johnmiller/ling/sabor-installs/sabor/store/exp_results_SCA.tsv INFO Estimated optimum threshold and f1 score (0.40, 0.773) INFO Observed threshold and f1 score (0.40, 0.773)and
`% cldfbench sabor.cross_validate_pairwise --cv k-fold --k 3 --function NED 2022-05-02 16:40:58,586 [INFO] Data has been written to file </var/folders/79/2ckx3nkx6y76__jfdd13q9tw0000gn/T/tmp5hv83nod/full-wl.tsv>. partition: 0 3 0 436 8124 3976 INFO Trail result: ['NED', 0.1, None, 77, 5900, 2, 945, 0.975, 0.075, 0.14, 0.863]
INFO Trail result: ['NED', 0.2, None, 143, 5900, 2, 879, 0.986, 0.14, 0.245, 0.873]
INFO Trail result: ['NED', 0.3, None, 287, 5898, 4, 735, 0.986, 0.281, 0.437, 0.893]
INFO Trail result: ['NED', 0.4, None, 384, 5893, 9, 638, 0.977, 0.376, 0.543, 0.907]
INFO Trail result: ['NED', 0.5, None, 497, 5888, 14, 525, 0.973, 0.486, 0.648, 0.922]
INFO Trail result: ['NED', 0.6, None, 644, 5856, 46, 378, 0.933, 0.63, 0.752, 0.939]
INFO Trail result: ['NED', 0.7, None, 755, 5659, 243, 267, 0.757, 0.739, 0.748, 0.926]
INFO Trail result: ['NED', 0.8, None, 827, 5104, 798, 195, 0.509, 0.809, 0.625, 0.857]
INFO ## Wrote results to /Users/johnmiller/ling/sabor-installs/sabor/store/exp_results_NED.tsv INFO Train result: ['NED', 0.64, None, 700, 5774, 128, 322, 0.845, 0.685, 0.757, 0.935]
INFO Test result: ['NED', 0.64, None, 354, 2803, 63, 186, 0.849, 0.656, 0.74, 0.927]
partition: 1 3 436 872 8014 4086 INFO Trail result: ['NED', 0.1, None, 77, 5812, 2, 950, 0.975, 0.075, 0.139, 0.861]
INFO Trail result: ['NED', 0.2, None, 140, 5811, 3, 887, 0.979, 0.136, 0.239, 0.87]
INFO Trail result: ['NED', 0.3, None, 280, 5807, 7, 747, 0.976, 0.273, 0.426, 0.89]
INFO Trail result: ['NED', 0.4, None, 373, 5801, 13, 654, 0.966, 0.363, 0.528, 0.902]
INFO Trail result: ['NED', 0.5, None, 476, 5801, 13, 551, 0.973, 0.463, 0.628, 0.918]
INFO Trail result: ['NED', 0.6, None, 630, 5760, 54, 397, 0.921, 0.613, 0.736, 0.934]
INFO Trail result: ['NED', 0.7, None, 739, 5569, 245, 288, 0.751, 0.72, 0.735, 0.922]
INFO Trail result: ['NED', 0.8, None, 819, 5041, 773, 208, 0.514, 0.797, 0.625, 0.857]
INFO ## Wrote results to /Users/johnmiller/ling/sabor-installs/sabor/store/exp_results_NED.tsv INFO Train result: ['NED', 0.64, None, 691, 5690, 124, 336, 0.848, 0.673, 0.75, 0.933]
INFO Test result: ['NED', 0.64, None, 363, 2887, 67, 172, 0.844, 0.679, 0.752, 0.931]
partition: 2 3 872 1308 8062 4038 INFO Trail result: ['NED', 0.1, None, 68, 5818, 2, 1007, 0.971, 0.063, 0.119, 0.854]
INFO Trail result: ['NED', 0.2, None, 129, 5817, 3, 946, 0.977, 0.12, 0.214, 0.862]
INFO Trail result: ['NED', 0.3, None, 275, 5811, 9, 800, 0.968, 0.256, 0.405, 0.883]
INFO Trail result: ['NED', 0.4, None, 385, 5808, 12, 690, 0.97, 0.358, 0.523, 0.898]
INFO Trail result: ['NED', 0.5, None, 503, 5803, 17, 572, 0.967, 0.468, 0.631, 0.915]
INFO Trail result: ['NED', 0.6, None, 654, 5764, 56, 421, 0.921, 0.608, 0.733, 0.931]
INFO Trail result: ['NED', 0.7, None, 768, 5564, 256, 307, 0.75, 0.714, 0.732, 0.918]
INFO Trail result: ['NED', 0.8, None, 856, 5069, 751, 219, 0.533, 0.796, 0.638, 0.859]
INFO ## Wrote results to /Users/johnmiller/ling/sabor-installs/sabor/store/exp_results_NED.tsv INFO Train result: ['NED', 0.64, None, 717, 5690, 130, 358, 0.847, 0.667, 0.746, 0.929]
INFO Test result: ['NED', 0.64, None, 337, 2887, 61, 150, 0.847, 0.692, 0.762, 0.939]
Dataset Function Threshold tp tn fp fn precision recall F1 score accuracy
Test NED 0.640 354.000 2803.000 63.000 186.000 0.849 0.656 0.740 0.927 Test NED 0.640 363.000 2887.000 67.000 172.000 0.844 0.679 0.752 0.931 Test NED 0.640 337.000 2887.000 61.000 150.000 0.847 0.692 0.762 0.939 Average NED 0.640 351.333 2859.000 63.667 169.333 0.847 0.676 0.751 0.932 StDev NED 0.640 13.204 48.497 3.055 18.148 0.003 0.018 0.011 0.006`