Closed ilonatommy closed 1 month ago
Tagging subscribers to this area: @dotnet/area-system-globalization See info in area-owners.md if you want to be subscribed.
Author: | ilonatommy |
---|---|
Assignees: | ilonatommy, mkhamoyan |
Labels: | `area-System.Globalization` |
Milestone: | 8.0.0 |
Tagging subscribers to 'arch-wasm': @lewing See info in area-owners.md if you want to be subscribed.
Author: | ilonatommy |
---|---|
Assignees: | ilonatommy, mkhamoyan |
Labels: | `arch-wasm`, `area-System.Globalization` |
Milestone: | 8.0.0 |
Closing, the planned work for HybridGlobalization
was completed.
The task is to remove as much data from ICU files as possible and exchange ICU4C functions that are using this data with platform native functions - in the case of WASM with Web API. Because we are not able to get rid of ICU datafile completely (some functionalities are not easily replaceable) we will keep loading
icudt.dat
in a reduced form. This mode will be calledHybridGlobalization
and will be by default switched off. User can switch it on by setting MsBuild's<HybridGlobalization>
to true. PoC branch is here: https://github.com/dotnet/runtime/compare/main...ilonatommy:runtime:icu-platform-native.1) Removing
collations
for WASMGlobalizationNative_ChangeCase
+ optimize memory usage - do not create a new string for returning the value but pass the address of buffer reserved on C# size that will hold the result. [ILONA] public API:TextInfo.ToLower
,TextInfo.ToUpper
, TextInfo.ToTitleCase`GlobalizationNative_IndexOf
andGlobalizationNative_LastIndexOf
- will not work for letters that consist of more than one grapheme, issue: https://github.com/tc39/ecma402/issues/506 [ILONA] public API:CompareInfo.IndexOf
,String.IndexOf
,MemoryExtensions.IndexOf
,CompareInfo.LastIndexOf
,String.LastIndexOf
,MemoryExtensions.LastIndexOf
.GlobalizationNative_StartsWith
andGlobalizationNative_EndsWith
[ILONA] public API:CompareInfo.IsSuffix
,String.EndsWidth
,MemoryExtensions.EndsWith
,CompareInfo.IsPrefix
,String.StartsWidth
,MemoryExtensions.StartsWith
.GlobalizationNative_CompareString
(withoutOrdinal
andOrdinalIgnoreCase
,IgnoreKanaType
,IgnoreWidth
) [ILONA] public API:CompareInfo.Compare
,String.Compare
ImplementIgnoreKanaType
andIgnoreWidth
basing onpal_collation.c
code [ILONA]Ordinal
andOrdinalIgnoreCase
. [ILONA]GlobalizationNative_GetSortKey
If much, throw PNSE onGlobalizationNative_GetSortVersion
.HybridGlobalization
from Blazor. Changes in dotnet/sdk might be needed.2) Removing
normalization
for WASM: Removed from planned Hybrid features. Savings from normalization removal on WASM are ~60kB. The removal breaks public APIs: string.Normalize, string.IsNormalized, IdnMapping.GetAsciii, IdnMapping.GetUnicode. Normalize/IsNormalized were succesfully replaced in https://github.com/dotnet/runtime/pull/85510. For GetAscii/GetUnicode replacement, Invariant implementation enhanced by normalization step was used, see branch https://github.com/ilonatommy/runtime/tree/idn-mapping. The mapping still lacked detection of disallowed/ignored/mapped characters and would need access to MappingTables of the current Unicode version to e.g. detect incorrect inputs to throw. One Unicode version mapping table in plain text weights ~900kB. Even if we compressed it, we still would need to maintain it with every Unicode version. Development time spent on correct implementation and chances of real size reduction, taking into cosideration the need to keep the mapping tables, are too small to remove normalization data from ICU.Updateicudt_wasm.dat
and corresponding sharded datafiles.Implement Punycode, ~~might be using this algorithmusing InvariantGlobalization algorithm + normalization function.~~Use normalization from the PoC branch.Update documentations.3) Investigate implications of removing further data batches, e.g. check the effect of removing all collations,
coll_ucadata
,locales_tree
etc.no exception thrown
forCultureInfoAll.LcidTest
,CultureInfoAll.GetCultureTest
,CultureInfoConstructor.Ctor_String
(now we support wider range of locales so we should not expect some of them throw as it was with standard ICU)4) (optional) Enhancement of collations by manual workarounds:
Consider fixing someIgnoreSymbols
by adding static data on JS side.Consider shifting katakana/hiragana and high/low symbols, based onpal_collation.c
code5) (optional) Consider failing a build when HybridGlobalization function is not supported
Tracking issues:
https://github.com/dotnet/runtime/issues/101912 https://github.com/dotnet/runtime/issues/102305 https://github.com/dotnet/runtime/issues/102373 https://github.com/dotnet/runtime/issues/95921 https://github.com/dotnet/runtime/issues/95795 https://github.com/dotnet/runtime/issues/95623