Open frivoal opened 5 years ago
Section 4.5 talks about canonicalising items that are marked in the registry as deprecated (eg. grandfathered tags) or that contain extlang subtags (eg. zh-yue
-> yue
). These are things that can be determined automatically by using the information in the registry. But it doesn't include zh-HK
.
Section 3.2 mentions zh-CN as possible equivalent to zh-Hans, but as i read this as a configuration option offered to the user, not an assumption that they meant the same thing in all contexts (which is what canonicalisation means) or that the author intended one while writing the other. And I think assuming that zh-HK
means yue
is more of a stretch than the zh-CN/zh-Hans
assumption.
I think we need to be careful about making inappropriate assumptions on the behalf of the content author. zh-HK
may be used to mean Mandarin chinese, but with traditional script, or possibly written text that includes the few additional characters that are used in Hong Kong - in fact, i thought that the predominant legacy usage for zh-HK
arose from an early workaround before zh-Hant
existed - not for yue
. Of course, it could also be used for Cantonese. But it's possible that it's even used for Hakka or Minnan Chinese as spoken in Hong Kong. It depends, really, on what the author needed and intended. It may also depend on whether it is used for a spoken or written phrase.
So i think it makes sense to assume equivalence for language tags that can be automatically paired using information in the registry, but not for things like zh-HK
.
The CSS Working Group just discussed canonicalization of :lang() selectors
.
Thanks for the opportunity to discuss at TPAC. I'm gonig to add some personal notes here that hopefully will help further discussion.
The section in BCP47 on canonicalization can be found here. As noted, this process includes several operations for dealing with different types of mapping for grandfathered/redundant tags as well as changes to region or language subtags over time. All of these mappings further matching operations in a positive way and should be part of CSS Selectors.
The canonical form of language tags is without extlangs: this was the intention when extlangs were created. The extlang form exists because there are cases where content authors may find some utility in using them, but BCP47 implementations are encouraged to "lose" the enclosing primary language subtag. CSS may still choose to do differently.
The matching challenge has multiple considerations. I'll try to illustrate using the yue
(Cantonese) and zh
(Chinese) subtags. Suppose you have a stylesheet and content like so:
:lang(zh) { /* something */ }
:lang(yue) { /*something else */}
<p lang="zh-Hans">...
<p lang="yue">...
With the basic canonicalization of language tags, the range zh
only matches the first
and the range yue
only the second. With the extlang canonicalization, we see the above transformed to the following for matching:
:lang(zh) { }
:lang(zh-yue) { }
<p lang="zh-Hans">...
<p lang="zh-yue">...
Now the first selector matches both content items (the second still only matches one item). This is probably not the intention of the author. Obviously, as was pointed out in the meeting, the opposite case exists (if one started with zh-yue
and wrote styles for :lang(zh)
).
Note that the tag zh-yue
was registered during the RFC3066 era and is one of the so-called "redundant" grandfathered tags. It's replacement is in the BCP47 registry as yue
(and not zh-yue
). However, tags such as zh-yue-Hant
or zh-yue-CN
or such are well-formed and valid since RFC4646.
The question of which sort of content-to-stylesheet canonicalization incompatibility to incur is what is at question here. Note there are quite a few macrolanguages in the registry--this affects more than the Chinese complex of languages and the degree to which the extlang form matches or interferes with use varies by the macrolanguages. Arabic or Malay, to pick two examples, probably do not want the extlang form in as many cases as Chinese--the enclosed languages really have different needs.
I will supply additional links and some further guidance anon: I've solicited more input from the CLDR community.
The PR got accidentally merged, but we didn't have a resolution on this.
Reviewing as part of w3c/i18n-action#117.
In reviewing #4212, I disagree with using the extlang form. The matching of extlangs to the primary language is too complicated for most page authors to deal with and it's a little weird to create subtags out of nothing (e.g. yue
=> zh-yue
) when the created subtags interfere with prefix-based matching. CLDR and BCP47 contain data to allow some subtag inference (notably, the script subtag)
Both the [=content language=] and the [=language range=] must be canonicalized and converted to extlang form as per section 4.5 of [[!RFC5646]] prior to the extended filtering operation. The matching is performed case-insensitively within the ASCII range.
Would be better off as:
Both the [=content language=] and the [=language range=] must be canonicalized as per section 4.5 of [[!RFC5646]] prior to the extended filtering operation. Such matching is performed [=ASCII case-insensitively=].
Happy to discuss. Don't forget that at least one of the BCP47 authors 🙈 is available to you for consultation when dealing with these issues.
At least 2!
On Thu, Aug 29, 2024, 15:17 Addison Phillips @.***> wrote:
Reviewing as part of w3c/i18n-action#117.
In reviewing #4212 https://github.com/w3c/csswg-drafts/pull/4212, I disagree with using the extlang form. The matching of extlangs to the primary language is too complicated for most page authors to deal with and it's a little weird to create subtags out of nothing (e.g. yue => zh-yue) when the created subtags interfere with prefix-based matching. CLDR and BCP47 contain data to allow some subtag inference (notably, the script subtag)
Both the [=content language=] and the [=language range=] must be canonicalized and converted to extlang form as per section 4.5 of [[!RFC5646]] prior to the extended filtering operation. The matching is performed case-insensitively within the ASCII range.
Would be better off as:
Both the [=content language=] and the [=language range=] must be canonicalized as per section 4.5 of [[!RFC5646]] prior to the extended filtering operation. Such matching is performed [=ASCII case-insensitively=].
Happy to discuss. Don't forget that at least one of the BCP47 authors 🙈 is available to you for consultation when dealing with these issues.
— Reply to this email directly, view it on GitHub https://github.com/w3c/csswg-drafts/issues/4154#issuecomment-2319146095, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACJLEMFPV45255SIFYDIGBTZT6MYDAVCNFSM4IHK4XH2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMZRHEYTINRQHE2Q . You are receiving this because you are subscribed to this thread.Message ID: @.***>
Cantonese may be described with any of the following language tags:
zh-HK
(most common, legacy)zh-yue
(also legacy)yue
(current most appropriate value)Even though these refer to the same language,
:lang(yue)
would only match the last one.RFC 5646 section 4.5 together with the IANA Language Subtag Registry define canonicalization and mappings that would allow to match the last two (by extlang form). RFC4647 section 3.2 says we should use this.
The last paragraph of that same section 3.2 also says we may also want to consider mappings like zh-HK to zh-yue, and that too could seem appropriate, but as far as I know there’s no equivalent to IANA Language Subtag registry for such mappings.
While it is possible for authors to deal with this themself by using
:lang(yue, zh-yue, zh-HK)
, to the extend we can automate this, they shouldn't have to deal with this.(I'm taking Cantonese as an example, but the same can be said about other languages)