search

chapel-lang / chapel

a Productive Parallel Programming Language
https://chapel-lang.org
Other
1.76k stars 414 forks source link

Sanity check download counts scripting / logic / values #18169

Open bradcray opened 2 years ago

bradcray commented 2 years ago

Our download count graphs have been reporting some very stellar numbers over the past few releases, though sometimes I worry that they are unrealistically stellar. For example, given the apparent growth in number of downloads, I don't feel as though we've seen a similar growth in the number of user questions or involvement. ~And then other things seem puzzling like the relatively low number of homebrew downloads for 1.24 as reported by GitHub (~100) compared to previous releases or other means of getting the release.~ [update: determined that Homebrew is more like 648 downloads at present for 1.24, which still seems a bit low, but less surprising... see this comment]. And do DockerHub users really account for such a large fraction of our downloads given that I feel we almost never hear about users installing through docker?

All this leads me to worry "Maybe our scripting is just broken." E.g., could we be doing something like counting a single download that happened on day 1 of the release for every subsequent day since then for one of these configurations?

To that end, I think it'd be valuable for us to audit these numbers by determining what we think the counts for each technology are manually as best we can and comparing to what we're graphing; and if we see disparate numbers, examine the scripts to see if there are bugs, or perhaps changes to what's being reported to us over time which would account for some of the puzzling figures above.

bradcray commented 2 years ago

@Maxrimus : Tagging you on this since you're currently looking at download counts and the thing that caused me to go from musing about it from time to time to filing the issue was the orders of magnitude differences between number of homebrew package downloads reported vs. total number of downloads of 1.24 while thinking about https://github.com/Cray/chapel-private/issues/2341.

bradcray commented 2 years ago

Another reason to feel skeptical about the accuracy of the current download counts: First, it seems odd that the Chapel 1.24.0 docker images have continued to be downloaded after 1.24.1 was released, yet the graphs show both releases being downloaded every day since then. Also, from what I'm seeing, it looks like the difference between the dockerhub-chapel-chapel-1.24.0 and 1.24.1 lines, is exactly 2171 for every day I've checked since 1.24.1 was released, which is the number of downloads 1.24.0 had received on that day. This seems too coincidental to be trustworthy (suggesting that every day, the exact same number of copies of each dockerhub image has been downloaded).