Open sffc opened 1 year ago
General preference for #3
Do we plan to provide these locale-ish APIs in the near term? I actually think future try_new_auto_with_locale()
would be fine
General preference for #2
I'd prefer the default constructor names to be consistent in behavior as much as makes sense. If we believe Segmenter constructors will want to take locale just like all others, lets keep the names for those constructors.
If, in the future, we decide to not add those names, we can always alias the default constructor names to the _invariant
ones.
Why this ICU4C rule isn't merged/requested to UAX#29? Does ICU4C have a plan to file/merge an issue to UAX#29? After merging this change to UAX#29, then #3.
UAX #29 in general doesn't really want to include locale-specific stuff because it wants to leave that up to CLDR.
Suffix suggestions:
_invariant
_root
(wrong: we don't have CLDR root tailorings)_uax
(too restrictive: we want to add CLDR tailorings)_untailored
(wrong: LineBreakOptions has tailorings)_default
(could be restrictive with regard to default data)Since we will want to take locales as parameters (even for segmenters where that isn't implemented yet), IMO we should make that the "normal" case.
From discussion with @aethanyc @makotokato @Manishearth @nordzilla: It is an enhancement to consume CLDR root.xml tailorings, but not necessarily a bug. We would like to see it done in a timely fashion.
Conclusion: use _invariant
For the record we didn't use _invariant
in #3294
Data bundles that contain some language-specific data for sentence segmentation: https://github.com/unicode-org/icu/tree/main/icu4c/source/data/brkitr
These seem to be lists of abbreviations that contain a period that doesn't end a sentence. How bad would it be to merge the lists and use the merged lists across languages?
fi_sv override for word break: https://github.com/unicode-org/icu/blob/main/icu4c/source/data/brkitr/rules/word_fi_sv.txt
It's a bit sad that treating letter, colon, letter as having a word break opportunity after the colon is a case of giving computer syntax needs precedence over natural-language needs. If accommodating computer syntaxes wasn't given priority, the Finnish/Swedish requirement of not treating letter, colon, letter as containing a word break opportunity could be hoisted to root.
el override for sentence break: https://github.com/unicode-org/icu/blob/main/icu4c/source/data/brkitr/rules/sent_el.txt
This seems to be about ASCII semicolon having sentence-ending question mark semantics. Could this be accommodated in the root by triggering the rule on the most recent letter being from the Greek script?
It's a bit sad that treating letter, colon, letter as having a word break opportunity after the colon is a case of giving computer syntax needs precedence over natural-language needs. If accommodating computer syntaxes wasn't given priority, the Finnish/Swedish requirement of not treating letter, colon, letter as containing a word break opportunity could be hoisted to root.
I think German needs this tailoring as well. I don't know why Finnish and Swedish do, but in German a colon is commonly used to form gender-neutral nouns, like Lehrer:in
, which should not contain any word breaks.
What's the current process for updating the tailorings? ICU or CLDR?
@robertbastian this was recently discussed in the CLDR design meeting, CLDR issue: https://unicode-org.atlassian.net/browse/CLDR-15910 / PAG issue: https://github.com/unicode-org/properties/issues/187 (internal)
There's thought that this should actually be made to apply to all languages, since colons without spaces on either side are not really a thing in regular text anyway, and if the space has been removed there's a good chance it's on purpose.
Data bundles that contain some language-specific data for sentence segmentation: https://github.com/unicode-org/icu/tree/main/icu4c/source/data/brkitr
These seem to be lists of abbreviations that contain a period that doesn't end a sentence. How bad would it be to merge the lists and use the merged lists across languages?
That's exactly what they are. https://www.unicode.org/reports/tr35/tr35-general.html#Segmentation_Exceptions for more details.
The lists for one language may not be applicable for others. But you could probably calculate a list that's likely to be generally useful, it might be less useful for any particular language.
I think German needs this tailoring as well.
I think https://unicode-org.atlassian.net/browse/CLDR-15910 should be reverted on the root level so that we don't need tailorings to accommodate natural languages.
I don't know why Finnish and Swedish do
For Finnish, the use case is marking where a sufficiently unusual word body (e.g. acronym) ends and the case suffix starts. For example, English Henri’s would be Henrin in Finnish but English ICU4X’s would be ICU4X:n in Finnish. The use case for Swedish seems to be also about applying suffixes (though not case suffixes) to sufficiently unusual word bodies. (Consider an analog English Londoner but with with the suffix applied to e.g. a sports team acronym.)
Okay, for 2.0 purposes, which of the four segmenters requires a locale parameter?
The "language of the text" would be more appropriate to provide in the terminal segment
function since it is an attribute of the text, but since that requires data loading, it might be more appropriate to specify it in the constructor. Alternatively, we could stuff all sentence language tailorings into a single data key which is always loaded when making a sentence segmenter, as we do for the word break segmenter.
Grapheme: Are there locale-specific CLDR tailorings for graphemes?
Not yet. Please put it into the API. I was doing planning on a work item to move this forward. This is for example languages that want to keep "ch" together etc.
Grapheme
Please put it into the API.
On the flip side, putting this in the API really requires making ECMA-402 have a way to explicitly ask for root and to default to root.
Some users getting a different definition of extended grapheme clusters based on the browser UI locale would likely be bad, after developers having assumed for years that extended grapheme clusters are a Unicode-level concept and not a locale-level concept. Also, it would be bad to have to assume that English is always going to be the untailored language and to teach every developer to ask for a grapheme segmenter for English in order to get behavior on a similar level of stability that one would expect of e.g Swift strings.
This is for example languages that want to keep "ch" together etc.
What languages do you mean and why do they want to keep "ch" together for the kind of purposes that extended grapheme clusters are used for, such as denying the selection of only "c" or only "h"? Czech treats "ch" as a collation unit, but do users of the language expect not to be able to select "c" and "h" individually?
We (poles and chechs) expect to be able to select each letter separately but would really prefer not to break lines between those two letters when they're next to each other.
also generally speaking not having a locale option is likely to disproportionately cause challenges for minority users of the script. Yes, you might seem to have a common solution for majority users, but then there isn't a customization option if something goes against the majority use. sss-Thai
line breaking is an example.
Discuss with:
Optional:
:
thing back into root.std::format
, ... so trying to make it language-dependent is a ship that sailed in 2003. If there is someone who wanted to tailor grapheme cluster segmentation, we need to understand their use case and given them their own API. I have fewer opinions about word segmentation.lang
attribute, and that's it.https://developer.mozilla.org/en-US/docs/Web/HTML/Global_attributes/lang
https://github.com/unicode-org/cldr/blob/main/common/segments/ja.xml
qu
, ambiguous quotation marks. I think different languages use quotation marks in different ways. There are heuristics baked into the algorithm. There are also single quotation marks, which are used very differently across languages. I intend to improve the heuristics, but at the end of the day they are just heuristics, and a language hint could help, and I understand that this is done by Apple.Initial recommendations:
LGTM: @sffc @eggrobin @hsivonen @Manishearth
LGTM: @sffc @hsivonen @eggrobin @Manishearth
.segment()
function, dependent on investigation of the call sites in order to optimize data slicing.LGTM: @sffc @hsivonen @Manishearth
colloquial Cantonese
Henri later provided me context: https://github.com/tc39/proposal-intl-segmenter/issues/133#issuecomment-779777282
Since "colloquial cantonese" is ambiguous, I'll expand on this a bit: the distinction being talked about is "standard written Chinese" (which can be used to write spoken Cantonese but uses a lot more Mandarin vocabulary — it's complicated), vs "written Cantonese", which is more of a 1-1 mapping to spoken Cantonese and may use Cantonese-specific words and characters.
It's also sometimes called "written vernacular Cantonese" however this can be somewhat ambiguous since Written Vernacular Chinese is something else.
CLDR has decided that they're putting the : thing back into root.
Yes, but the Apple rep in the meeting, who is originally from Sweden, insisted that CLDR keep the fi/sv word break tailorings, because he thinks that even the future keyword selecting technical usage should keep fi/sv words together across colon.
No language parameter for grapheme cluster segmenter
+1
Language parameter for the other three segmenters
+1
Plan to put the parameter on the .segment() function
That seems weird both from looking at usage and thinking about data loading.
The conclusions from the discussion of this issue with the CLDR design group:
My suggested path forward for this issue, then, is to add an options bag to the WordSegmenter, LineSegmenter, and SentenceSegmenter constructors with an optional content_locale
field of type &LanguageIdentifier
.
I'm moving this back into 1.5 because the constructor can be drafted and bikeshed ahead of time, and then in 2.0 we can do the minimal change of making the new constructor the default one.
Grapheme clusters should not be language-specific; baked into much low-level processing (e.g., Swift, font mappings) which we don’t want to be language-specific
This makes no sense and contradicts the long standing requests. ( https://unicode-org.atlassian.net/browse/CLDR-2992 which I am working on scheduling ) I would have joined, did not realize this was coming up today.
Perusing the notes it's not clear that the previous requirements and recent discussion from the segmentation summary last year were included here.
Based on additional discussion in the email thread, I would like to move forward with the recommendation in https://github.com/unicode-org/icu4x/issues/3284#issuecomment-2030731133, with the additional understanding that we may add support for locale-based grapheme segmentation in the future if CLDR adds data for this, but it might take the form of another (fifth) segmenter type.
Concretely:
new
or try_new
function without an options bagtry_new_with_options
function that includes a content_locale
optionWhen looking ICU4C brkiter rule files for word and sentence, UAX#29's property of this isn't same each locale. But rules seem to be same. So if we modify datagen (with a few changes of toml data file), we can generate rules data per locale.
In the API review, @markusicu pointed out that ICU takes a locale in the segmenter, and the locale affects the behavior in certain cases, such as those in the data files below:
Why don't we support these in ICU4X Segmenter, and should we add them?
For 1.2 purposes, we have a few choices:
_invariant
to the constructor names, so that in the futuretry_new_auto_invariant()
creates the locale-invariant segmenter andtry_new_auto(locale!("el"))
creates the locale-specific segmenterThoughts?
@aethanyc @makotokato @Manishearth