hpjansson
commented
3 years ago
- It's not a very good metric: In practice I find that the "line changed" count is extremely noisy, not useful to study repositories over time. It gets worse when you aggregate different repositories of a community, that may have different practices. (One way to phrase this: I was never able to succesfully interpret the results of
--unit changesin an interesting way for the repositories I was looking at.)
Agreed. I've used it a bit to look for outliers in the data, but in general it's not very useful.
- Performance: If we didn't have to compute this information at
ingesttime, then we could ask git to clone the repository but don't actually download the trees, just the history. This can be done with Partial Clones, I think thatgit clone --filter=blob:nonewould work. This would massively decrease the cloning time to start the analysis, which takes a substantial part of the total analysis time.
In the manual use case, you typically clone once followed by multiple ingest/plot, and repositories can be updated incrementally after that, so I didn't consider it a big performance issue. Additionally, not all servers (http...) support --filter.
That said, maybe you're writing a wrapper script that clones, performs the analysis and then cleans up after itself -- partial clones could save a lot of resources there if the script is going to be run regularly. And on limited/metered connections it could be a life saver.
One could of course consider adding a
--no-changesoption to the tool that would not collect this information and be compatible with those partial clones, and possibly even record this fact in the database to later fail properly if users try to plot changes. But:
This could be invasive, while removing changes altogether makes the script simpler, which is good.
Git partial clones don't fail nicely if you try to ask for changes (or anything blob-related) after the fact, they try to re-download it on demand and the performance is terrible (I've observed one request for each commit when asking
git log --stat). This means that if we fail to protect some blob-depending parts of the script in the codebase, the user experience is going to be horrible in this case. I would find it easier if we could just globally enforce that only the history, not the patches, are accessed by the fornalder ingestion logic.
One of the recurring comments on my GNOME project analysis is that translation commits are fairly numerous there and seen as orthogonal to code contributions. They are also easy to separate mechanically based on file names (po/ subdir, .po extension). Similar arguments could be made for build/release engineering, documentation, icons, etc.
Keeping the --stat would allow us to filter the commits or assign cohorts based on file type (by extension or regexps over file names/paths, perhaps). It might also be useful in tracking efforts to switch to a new programming language (e.g. from C to C++ or Rust -- GCC did the former, and there are several projects experimenting with the latter).
So... I like the --no-changes idea, even with the downsides. Maybe call it --no-blobs as eventually it'll affect more than just the change statistic.
An attractive idea would be to detect filtered repositories on ingest and suppress --stat automatically. Then we wouldn't need a flag at all. We'd probably have to warn that this was happening, and especially when finding a mix of filtered and unfiltered repositories, to avoid silently generating plausible-looking but incorrect plots down the line.
gasche
Thanks again for fornalder, it's a very interesting tool.
I was wondering if you had opinions about the idea of dropping the "lines changed" measurement from the tool, and related interface options. There are two different aspect that make me think this could be a good idea:
It's not a very good metric: In practice I find that the "line changed" count is extremely noisy, not useful to study repositories over time. It gets worse when you aggregate different repositories of a community, that may have different practices. (One way to phrase this: I was never able to succesfully interpret the results of
--unit changesin an interesting way for the repositories I was looking at.)Performance: If we didn't have to compute this information at
ingesttime, then we could ask git to clone the repository but don't actually download the trees, just the history. This can be done with Partial Clones, I think thatgit clone --filter=blob:nonewould work. This would massively decrease the cloning time to start the analysis, which takes a substantial part of the total analysis time.Concrete example: for a long list of repositories I wanted to look at, cloning everything in parallel takes about 30mn, ingesting also takes 30mn (presumably this could be parallelized). Doing
blob:nonepartial clones completes in 3mn. (The difference could be arbitrarily large for repositories that require downloading large amount of data.)One could of course consider adding a
--no-changesoption to the tool that would not collect this information and be compatible with those partial clones, and possibly even record this fact in the database to later fail properly if users try to plot changes. But:git log --stat). This means that if we fail to protect some blob-depending parts of the script in the codebase, the user experience is going to be horrible in this case. I would find it easier if we could just globally enforce that only the history, not the patches, are accessed by the fornalder ingestion logic.