Support using an Intl.Segmenter for word tokenization in diffWords

kpdecker / jsdiff

A javascript text differencing implementation.

BSD 3-Clause "New" or "Revised" License

7.75k stars 491 forks source link

Closed ExplodingCabbage closed 2 days ago

ExplodingCabbage commented 7 months ago

This would let us support:

languages without spaces - e.g. Chinese, Japanese
languages where punctuation that is normally a word break can appear in the middle of a word (e.g. colons can appear in the middle of words in Finnish)