As surfaced in jco/#443, the current string conversion logic in store_string_to_utf8 doesn't quite match what an implementation naturally wants to do. In particular, while the CABI does specify starting with an optimistic allocation that assumes all code points fit into a single byte, the fallback (that reallocs to the worst-case size and does the general transcode) doesn't happen as soon as the first non-fitting code point is discovered; it happens later once the initial allocation is full. This PR tweaks the logic to instead do the realloc immediately when the non-fitting code point is discovered.
Technically, this could be considered a breaking change (since it observably changes when exactly the realloc happens), but it would be surprising if any code depended upon this behavior (at this early stage, without massive usage). Thus, I think we can just make the change.
As surfaced in jco/#443, the current string conversion logic in
store_string_to_utf8doesn't quite match what an implementation naturally wants to do. In particular, while the CABI does specify starting with an optimistic allocation that assumes all code points fit into a single byte, the fallback (that reallocs to the worst-case size and does the general transcode) doesn't happen as soon as the first non-fitting code point is discovered; it happens later once the initial allocation is full. This PR tweaks the logic to instead do the realloc immediately when the non-fitting code point is discovered.Technically, this could be considered a breaking change (since it observably changes when exactly the realloc happens), but it would be surprising if any code depended upon this behavior (at this early stage, without massive usage). Thus, I think we can just make the change.
CC @alexcrichton @guybedford @dbaeumer