kaitlynhair
commented
2 years ago Thanks for this!
I have performed an evaluation of the ASySD shiny app by testing performance across 5 biomedical datasets versus other tools. (Publication coming soon (I hope), preprint here: https://www.biorxiv.org/content/10.1101/2021.05.04.442412v1). I will need to add this into the documentation!
As I mentioned in the other issue - this package has been edited a lot recently to introduce new functionality. Although the method to identify duplicate citations has not changed, I think there was a bug in the new function to merge duplicates and retain additional meta-data (as opposed to only keeping one full citation from each duplicate group). I'm not sure how easy it is to run this again but I expect it may have changed(?). In our evaluation we calculated an overall sensitivity and specificity ~95% but this could be variable across different research domains.
LukasWallrich
Before including it in a registered report, I wanted to evaluate the performance of ASySD - using the method recently published here
Your algorithm does really well on what I see as the most important outcome - there are no false positives, i.e. no entries that get lost. However, on false negatives, it could do better with 86% sensitivity - even though that is already better than any of the common reference managers ... so I am definitely excited about using this going forward.
Details on the benchmark are here - when you are documenting this package, you might want to include something like that? Also, there is a csv of false negatives there (i.e. overlooked duplicates) that might be worth reviewing?