Open Mr0grog opened 5 years ago
See also edgi-govdata-archiving/web-monitoring-processing#175
Would this be a good good-first-issue
label candidate?
I wish it was, but the HTML diff is an incredibly horrifying mess, and nobody should try and screw with it unless they are ready for a lot of setbacks and a lot of WTFs. That is why it is not already marked with “help wanted.”
This FTP diffing problem made me realize we should probably be splitting tokens in the HTML diff on periods (and maybe other punctuation?), not just on whitespace:
(Of course we don’t really want to use this differ on FTP listings, but that’s a different matter.)
This requires some care, though — we probably want to treat the periods as tokens themselves (in case they change), unlike whitespace. We’ve also talked about this before in terms of general punctuation handling — it would be really useful not only to split this way, but to tag and count punctuation changes separately from other changes. We might not prioritize a punctuation change for analysts to look at like we do a word change, and it would be nice to call out clearly that a change was merely in punctuation.
There are also punctuation changes we might want to treat extra special and even suppress in many cases. For example, changing
’
to'
(apostrophe to prime) is a change we’ve seen before, and not one we generally care about.