Closed bruffridge closed 3 years ago
I've just added a commit with my results for this issue. The results themselves are copied here as follows:
MAG | MeSH | Everything Else | P@1=nDCG@1 | P@3 | P@5 | nDCG@3 | nDCG@5 |
---|---|---|---|---|---|---|---|
no | no | no | 0.307 ± 0.047 | 0.221 ± 0.036 | 0.193 ± 0.012 | 0.239 ± 0.038 | 0.250 ± 0.026 |
yes | no | no | 0.498 ± 0.034 | 0.393 ± 0.024 | 0.312 ± 0.029 | 0.420 ± 0.026 | 0.415 ± 0.028 |
no | yes | no | 0.409 ± 0.067 | 0.313 ± 0.046 | 0.264 ± 0.029 | 0.338 ± 0.050 | 0.348 ± 0.043 |
yes | yes | no | 0.533 ± 0.065 | 0.432 ± 0.045 | 0.345 ± 0.040 | 0.461 ± 0.044 | 0.455 ± 0.046 |
no | no | yes | 0.582 ± 0.064 | 0.450 ± 0.047 | 0.343 ± 0.042 | 0.486 ± 0.048 | 0.471 ± 0.055 |
yes | no | yes | 0.586 ± 0.104 | 0.443 ± 0.059 | 0.350 ± 0.045 | 0.482 ± 0.069 | 0.475 ± 0.063 |
no | yes | yes | 0.571 ± 0.087 | 0.439 ± 0.064 | 0.344 ± 0.043 | 0.477 ± 0.069 | 0.468 ± 0.063 |
yes | yes | yes | 0.591 ± 0.043 | 0.452 ± 0.036 | 0.359 ± 0.027 | 0.492 ± 0.036 | 0.487 ± 0.033 |
In a nutshell, this suggests that MAG fields of study alone give somewhat more information than MeSH terms alone, although both contribute positively to improving precision/nDCG.
Also note that the standard deviation's rather high (perhaps due to the size of our dataset) -- sometimes for the same config settings (but different folds of the dataset) I get test precisions ranging from 0.44 to 0.72
Eric, can you rerun this ablation study, now that the MAG/MeSH bug is fixed? I want to see if we can just use MAG terms, or if we need MeSH as well. Using just MAG terms makes data collection easier.
<!DOCTYPE html>