BlobCodes
commented
3 months ago Maybe the converstion algorithm from String#to_utf16 could be implemented as a macro method? It's a bit complex, but not too much. I don't think we can explicitly do math operations on 16-bit integers in the macro language, though.
Simply porting #14671 works fine, explicit math operations on 16-bit integers are not needed.
class String
macro utf16_literal(data)
{%
arr = [] of NumberLiteral
data.chars.each do |c|
c = c.ord
if c < 0x1_0000
arr << c
else
c -= 0x1_0000
arr << 0xd800 + ((c >> 10) & 0x3ff)
arr << 0xdc00 + (c & 0x3ff)
end
end
arr << 0
%}
Slice(UInt16).literal({{arr.splat}})[0, {{arr.size - 1}}]
end
end
s = String.utf16_literal("TEST 😐🐙 ±∀ の")
# => Slice[84, 69, 83, 84, 32, 55357, 56848, 55357, 56345, 32, 177, 8704, 32, 12398]
String.from_utf16(s)
# => "TEST 😐🐙 ±∀ の"Encoding 10000 characters takes around 300ms. That's certainly not fast, but probably good enough.
EDIT: Added a final 0 byte
straight-shoota
stakach
ysbaddaden
bcardiff
When working with Windows APIs, it's common that we need UTF-16 strings (instead of Crystal's
Stringwhich is UTF-8).String#to_utf16is available for conversion.But most use cases of this method in stdlib are actually for string literals (e.g.
"Content Type".to_utf16). This is a bit unnecessary because it means the string transformation happens at runtime, while it could be entirely at compile time, avoding extra computation and allocation.A particularly intricate use case is in #14659 where we must not allocate at all. So it ends up with such a mechanism to achive compile time conversion:
UInt16.static_array({% for chr in "CRYSTAL_TRACE".chars %}{{chr.ord}}, {% end %} 0).This certainly works, at least for this limited use case. But it fails for code points outside the Basic Multilingual Plane. So it's not a generic solution.
It would be nice if we had an easy tool for creating UTF-16 encoded strings.
Maybe the converstion algorithm from
String#to_utf16could be implemented as a macro method? It's a bit complex, but not too much. I don't think we can explicitly do math operations on 16-bit integers in the macro language, though.An alternative would be to expose a compiler primitive for UTF-16 conversion.
Related: #2886