cmlsharp
commented
4 years ago Agreed this would be a good feature. Interestingly, we do actually do this to some extent behind the scenes, but at present it isn't expressed to the user.
In the ranking phase, we keep a frequency map of the k-grams we've seen and use it to do a modified inverse document frequency type thing when computing the score for two submissions. So rarity is considered for the ranking.
The slight trouble with displaying this to the user is in the more intensive comparing phase (which is only run for the top n ranked pairs), we actually expand the matching k-grams. So like if we have a particular k-gram that occurs in two submissions but there are additionally some surrounding characters that happen to match in both documents too, we'll absorb those into the match. While we know the frequency of the original k-gram, we don't know the frequency of the expansion that we end up showing the user. We could sort of fudge it and pretend they're the same, but that seems a but dicey.
benedictbrown
For each matching area, show how many files match ("2 files" means just the current two files), similar to etector. Etector gives details in tooltip, and give larger font size to more unique things. Former is very helpful, but latter can make comparing files difficult.
Knowing how unique a match is is very helpful for determining which cases to refer and articulating to committee how improbable the similarity is.