ltalirz
commented
2 years ago Dear Jonathan,
thank you very much for this thoughtful suggestion, as well as your offer to help - I really appreciate it! I think your suggestion is certainly worth considering, let me start with some explanation of the status quo, and then turn to your suggestion.
You rightfully point out that the current approach on atomistic.software of searching for both the name of a key author (or authors) and the name of the code excludes certain mentions of the code. This can happen, for example, when the name of the code author was not mentioned (no traditional "citation"), or when the reference section of the article was not indexed (e.g. code mentioned in title or abstract but full text index missing). For some codes this difference can indeed become quite substantial, e.g. I've seen a factor of 1.5 for LAMMPS [1].
This approach was inherited from the first static incarnation of the list and, I presume, was a natural evolution from wanting to count traditional citations. On the plus side, this puts the codes on almost [2] the same footing and one can almost [3] exclude false positives. On the negative side, you are correct that there is a real danger of false negatives, e.g. when authors start citing a new article/version of the software that does not match the current search query.
Turning to your suggestion.
My basic methodology has been to make the search criteria as broad as possible and then to estimate a false positive rate by sampling the data by hand. The premise is that false positives of an overly generous search are easier to quantify than false negatives of an overly narrow search. In MOPAC's case, I estimated false positive rates for two different, separated periods of time, and found them to be very similar. Thus, I suspect that it is safe to use a constant false positive rate for each code
Could you perhaps elaborate a bit on what makes you confident of the premise of a constant false positive rate?
While your two data points for the case of MOPAC seem to match it, I can't come up with a fundamental reason why the growth rate of code "X" and the term "X" in the rest of Google scholar can be assumed to correlate more generally.
I don't know what the origin of the false positives for MOPAC in your search are, but for codes that are named after words from day-to-day language (WEST, exciting, ...) or perhaps even more so for those named after somewhat rare words (fleur, Amber, ORCA) I can certainly imagine a sudden uptick of false positives because the word appears in some other context that is becoming a hot topic of research. Or, conversely, a strong relative reduction of false positives because the code is young and its user base is growing much faster than the use of the search term in Google Scholar (citations of MOPAC have remained relatively stable over the last decade, while for, say, ORCA, annual citations have grown by one order of magnitude).
This would result in an incorrect representation of the citation trend for that code, which I consider a key aspect of atomistic.software (perhaps more so than highly accurate absolute citation numbers/ranking). It would also be difficult to detect, while mitigating the issue of false negatives with the current approach seems to be more straightforward (by regularly checking for a more recent review paper / version release of a code). On second thought, I believe you are right that it would be easier to detect, but one may have to be very careful and for each update sift through a large portion of the search results for each code, and carry a time-dependent false positive rate (which complicates the approach). There is also the question of what to do with popular packages with >1000 citations where Google only provides links to the first 1000 results.
Secondly, one of the big advantages of using Google scholar is that it allows atomistic.software to provide direct links to the search results (no paywall), so that users can browse the current research done with a code. I would consider it a significant downside to have those results littered with false positives resulting from a more general query.
Given this analysis, I am initially skeptical of implementing this idea - but I certainly remain open to discussion.
At the same time, I am very interested in reducing false negatives in the current approach - a factor of 2 seems high to me, and not very satisfactory. One thing I've started doing is to include, besides the "main author of a code" (which is anyhow not well defined), first authors of the review articles via "OR", since the main author can hide behind an "et al." in references. In my experience so far, this improves the agreement between this way of citation counting and other, independent methods - perhaps this would help in the case of MOPAC as well?
So far, this has been done only for a couple of codes in the list on an "as needed" basis (e.g. for MOLCAS), and I would very much appreciate help in rolling it out more broadly.
[1] https://github.com/ltalirz/atomistic-software/issues/29 [2] Some codes need exceptions, e.g. for a code like WEST the only way may be to directly count the citations of a reference, or for Gaussian where people will just cite the version of the code since this is the preferred citation prescribed by the authors. [3] I try, to varying degree of success.
godotalgorithm
I am maintaining MOPAC, and as part of this activity, I've been exploring how to track its citations more accurately. My best attempt at this using Google Scholar data is: https://openmopac.github.io/_images/plot.pdf . There are two things to note: my estimate of citation counts to MOPAC are about twice what is in your database, and I have error bars.
My basic methodology has been to make the search criteria as broad as possible and then to estimate a false positive rate by sampling the data by hand. The premise is that false positives of an overly generous search are easier to quantify than false negatives of an overly narrow search. In MOPAC's case, I estimated false positive rates for two different, separated periods of time, and found them to be very similar. Thus, I suspect that it is safe to use a constant false positive rate for each code, but each code probably has a different false positive rate because some codes are more successful at inducing standardized citations than others.
Would you be interested in adapting this sort of methodology for atomistic.software? It would require an entry in codes.json for the false positive citation rate for each code, and you may need to adjust query_string somehow to accommodate more complicated nested logical expressions for the Google Scholar search strings. If you are willing to make the appropriate backend changes, then I'd be willing to go through the database entries to expand the search strings and estimate the false positive rates.