asterite
commented
2 years ago I think what's needed here is why one uses graphemes. What's the use case? Maybe it's for showing them in a text editor. I guess it must be something related to visual stuff. Maybe counting how many graphemes there are?
If those are the use cases, then adding more methods to Grapheme, like each_char and each_byte might not be needed. If one wants that, they can turn a Grapheme into a String. That allocates memory if the Grapheme stores a Char, but if these uses cases are rare then it should be fine.
So maybe keeping Grapheme like it's right now is fine.
straight-shoota
beta-ziliani
hutou
Blacksmoke16
The Grapheme API for strings was introduced in #11472.
However, there is still a debate about the actual data format for graphemes. They are a sequence of codepoints, which is typically represented as
String. But we chose to have a dedicated type,String::Graphemefor this. It's a wrapper aroundStringand provides an optimization for grapheme clusters that consist of only a single code point. They are stored asCharto avoid lots of tiny string allocations for the very common single-codepoints graphemes.This was already discussed in the introducing PR (#11472) and its precursor (#10720).
To summarize, I'm copying my original comment about representation options from https://github.com/crystal-lang/crystal/pull/11472#issuecomment-977178131 with some additions:
Char | Stringunion type is a huge improvement for grapheme clusters consisting of a single character. That should be most common in typical use cases, so it saves a big deal of allocations compared toString(or a collection ofCharas in the initial propsal). However, in some contexts, multi-character grapheme clusters are quite ubiquitous. Whether you're dealing with lots of composed emojis or scripts such as Thai, Hebrew or Arabic, or even just many decomposed diacritics, there might be a lot of string allocations for all these graphemes.String#each_grapheme_boundaryas part of the public API. That's trivial and basically cost-free. However, I'd consider it more a low-level API. It can be useful for custom grapheme-based implementations.StringPoolto avoid repeated allocations of identical grapheme clusters. There is no way to release values from a pool, so it would be a bad idea to use a global pool. But individual pools could be useful for single iterations. This could optionally be configurable and the API could allow specifying a custom pool to be used. This would be pretty easy to implement later. We just need to addStringPoolparameters toGrapheme.newand the iteration methods. A pool means we still need to allocate strings for every grapheme, but we can safe on duplicate allocations of the same graphemes. It is likely that the same graphemes show up repeatedly in longer texts of the same context, so that should be expected to be benefitial.Graphemeessentially a pointer to a substring. String allocation would only happen inGrapheme#to_s. The big downside of this approach is that it keeps the original string in memory. That's a serious implication when working with large amounts of texts. If theGraphemeinstaces are only used for iteration, this is not a problem. But if you start storingGraphemessomewhere, they keep pointers to the original strings alive. An example would be collecting frequencies of graphemes over a collection of texts. Assuming every evaluated texts contains a multi-character grapheme that was not encountered previously, all these strings will be kept in memory. The user would need to be aware of this behaviour and take action such as converting grapheme instances to self-owned cluster strings.Char | {Char, Char} | Stringor{size: Int32, data: Char[4]} | String. This would cover more cases, but there is technically no limit to the length of a single cluster, soStringis always needed for those that exceed the length.Stringinstead of a dedicatedGraphemetype. Elixir's string implementation takes this approach. The downside is that every grapheme cluster is heap allocated. This could be combined with 1. and/or 2. for some optimization. Another improvement would be a global pool for common grapheme clusters.