Closed p5pRT closed 9 years ago
p5pRT
commented
9 years ago The document was using "utf8"\, "UTF8" and "UTF-8" interchangably\, with the latter by far the most common. Make them all consistently "UTF-8"\, except of course when referring to the actual names of things (e.g. macros and functions).
p5pRT
commented
9 years ago p5pRT
commented
9 years ago TonyC pointed out on IRC that the description of the «Wide character (U+%X) in %s» warning was copied from perldiag\, so update it there too.
There are a few more instances of "UTF8" in perldiag\, but those should be in a separate ticket/patch (possibly together with a more wide-reaching overhaul).
-- "I use RMS as a guide in the same way that a boat captain would use a lighthouse. It's good to know where it is\, but you generally don't want to find yourself in the same spot." - Tollef Fog Heen
p5pRT
commented
9 years ago Once more\, with attachment!
ilmari@ilmari.org (Dagfinn Ilmari MannsÄker) writes:
TonyC pointed out on IRC that the description of the «Wide character (U+%X) in %s» warning was copied from perldiag\, so update it there too.
There are a few more instances of "UTF8" in perldiag\, but those should be in a separate ticket/patch (possibly together with a more wide-reaching overhaul).
-- "I use RMS as a guide in the same way that a boat captain would use a lighthouse. It's good to know where it is\, but you generally don't want to find yourself in the same spot." - Tollef Fog Heen
p5pRT
commented
9 years ago p5pRT
commented
9 years ago On Tue May 19 19:31:05 2015\, ilmari wrote:
Once more\, with attachment!
ilmari@ilmari.org (Dagfinn Ilmari MannsÄker) writes:
TonyC pointed out on IRC that the description of the «Wide character (U+%X) in %s» warning was copied from perldiag\, so update it there too.
There are a few more instances of "UTF8" in perldiag\, but those should be in a separate ticket/patch (possibly together with a more wide-reaching overhaul).
Thanks\, applied as 50ea4745c8ab3dc6c2e7bfcf895c892b27dae6b4.
Tony
p5pRT
commented
9 years ago The RT System itself - Status changed from 'new' to 'open'
p5pRT
commented
9 years ago @tonycoz - Status changed from 'open' to 'resolved'
p5pRT
commented
9 years ago On 20 May 2015 at 05:26\, Tony Cook via RT \perlbug\-followup@​perl\.org wrote:
On Tue May 19 19:31:05 2015\, ilmari wrote:
Once more\, with attachment!
ilmari@ilmari.org (Dagfinn Ilmari MannsÄker) writes:
TonyC pointed out on IRC that the description of the «Wide character (U+%X) in %s» warning was copied from perldiag\, so update it there too.
There are a few more instances of "UTF8" in perldiag\, but those should be in a separate ticket/patch (possibly together with a more wide-reaching overhaul).
Thanks\, applied as 50ea4745c8ab3dc6c2e7bfcf895c892b27dae6b4.
I dont know about this change actually. Sorry to say.
See perldoc Encode and look for the section "UTF-8 vs. utf8 vs. UTF8" (quoted below)
In short all three term have subtly different definitions.
At least one of the changes in this patch appears to change the correct use of "utf8" to the incorrect "UTF-8":
@@ -1174\,7 +1174\,7 @@ L\<\C is deprecated in regex|perldiag/"\C is deprecated in regex; marked by \<-- H (D deprecated) The C\<\< /\C/ >> character class was deprecated in v5.20\, and now emits a warning. It is intended that it will become an error in v5.24. This character class matches a single byte even if it appears within a -multi-byte character\, breaks encapsulation\, and can corrupt utf8 +multi-byte character\, breaks encapsulation\, and can corrupt UTF-8 strings.
The original was correct. The regex engine does not care about UTF-8\, it cares about utf8.
I think this patch should be reverted until each change can be reviewed to see if it refers to "UTF-8" the formal definition from Unicode\, or "utf8" the internal encoding used by Perl (a supeset of UTF-8) or if it refers to the UTF8 flag (which indicates the scalar contains "utf8" not "UTF-8".)
cheers\, Yves
UTF-8 vs. utf8 vs. UTF8 ....We now view strings not as sequences of bytes\, but as sequences of numbers in the range 0 .. 2**32-1 (or in the case of 64-bit computers\, 0 .. 2**64-1) -- Programming Perl\, 3rd ed.
That has historically been Perl's notion of UTF-8\, as that is how UTF-8 was first conceived by Ken Thompson when he invented it. However\, thanks to later revisions to the applicable standards\, official UTF-8 is now rather stricter than that. For example\, its range is much narrower (0 .. 0x10_FFFF to cover only 21 bits instead of 32 or 64 bits) and some sequences are not allowed\, like those used in surrogate pairs\, the 31 non-character code points 0xFDD0 .. 0xFDEF\, the last two code points in any plane (0xXX_FFFE and 0xXX_FFFF)\, all non-shortest encodings\, etc.
The former default in which Perl would always use a loose interpretation of UTF-8 has now been overruled:
From: Larry Wall \larry@​wall\.org Date: December 04\, 2004 11:51:58 JST To: perl-unicode@perl.org Subject: Re: Make Encode.pm support the real UTF-8 Message-Id: \20041204025158\.GA28754@​wall\.org
On Fri\, Dec 03\, 2004 at 10:12:12PM +0000\, Tim Bunce wrote: : I've no problem with 'utf8' being perl's unrestricted uft8 encoding\, : but "UTF-8" is the name of the standard and should give the : corresponding behaviour.
For what it's worth\, that's how I've always kept them straight in my head.
Also for what it's worth\, Perl 6 will mostly default to strict but make it easy to switch back to lax.
Larry
Got that? As of Perl 5.8.7\, "UTF-8" means UTF-8 in its current sense\, which is conservative and strict and security-conscious\, whereas "utf8" means UTF-8 in its former sense\, which was liberal and loose and lax. "Encode" version 2.10 or later thus groks this subtle but critically important distinction between "UTF-8" and "utf8".
encode("utf8"\, "\x{FFFF_FFFF}"\, 1); # okay encode("UTF-8"\, "\x{FFFF_FFFF}"\, 1); # croaks
In the "Encode" module\, "UTF-8" is actually a canonical name for "utf-8-strict". That hyphen between the "UTF" and the "8" is critical; without it\, "Encode" goes "liberal" and (perhaps overly-)permissive:
find_encoding("UTF-8")->name # is 'utf-8-strict' find_encoding("utf-8")->name # ditto. names are case insensitive find_encoding("utf_8")->name # ditto. "_" are treated as "-" find_encoding("UTF8")->name # is 'utf8'.
Perl's internal UTF8 flag is called "UTF8"\, without a hyphen. It indicates whether a string is internally encoded as "utf8"\, also without a hyphen.
-- perl -Mre=debug -e "/just|another|perl|hacker/"
p5pRT
commented
9 years ago @tonycoz - Status changed from 'resolved' to 'open'
p5pRT
commented
9 years ago On Wed\, May 20\, 2015 at 07:16:52AM +0200\, demerphq wrote:
On 20 May 2015 at 05:26\, Tony Cook via RT \perlbug\-followup@​perl\.org wrote:
On Tue May 19 19:31:05 2015\, ilmari wrote:
Once more\, with attachment!
ilmari@ilmari.org (Dagfinn Ilmari MannsÄker) writes:
TonyC pointed out on IRC that the description of the «Wide character (U+%X) in %s» warning was copied from perldiag\, so update it there too.
There are a few more instances of "UTF8" in perldiag\, but those should be in a separate ticket/patch (possibly together with a more wide-reaching overhaul).
Thanks\, applied as 50ea4745c8ab3dc6c2e7bfcf895c892b27dae6b4.
I dont know about this change actually. Sorry to say.
See perldoc Encode and look for the section "UTF-8 vs. utf8 vs. UTF8" (quoted below)
In short all three term have subtly different definitions.
Using minor typographical differences to differentiate them is kind of dumb.
At least one of the changes in this patch appears to change the correct use of "utf8" to the incorrect "UTF-8":
@@ -1174\,7 +1174\,7 @@ L\<\C is deprecated in regex|perldiag/"\C is deprecated in regex; marked by \<-- H (D deprecated) The C\<\< /\C/ >> character class was deprecated in v5.20\, and now emits a warning. It is intended that it will become an error in v5.24. This character class matches a single byte even if it appears within a -multi-byte character\, breaks encapsulation\, and can corrupt utf8 +multi-byte character\, breaks encapsulation\, and can corrupt UTF-8 strings.
The original was correct. The regex engine does not care about UTF-8\, it cares about utf8.
Anyone reading it should realize it's working with perl's internal unicode representation\, which is an extended UTF-8. For anyone who doesn't the difference is likely meaningless anyway.
I've reverted the commit.
Tony
p5pRT
commented
9 years ago On 20 May 2015 at 08:27\, Tony Cook \tony@​develop\-help\.com wrote:
On Wed\, May 20\, 2015 at 07:16:52AM +0200\, demerphq wrote:
On 20 May 2015 at 05:26\, Tony Cook via RT \perlbug\-followup@​perl\.org wrote:
On Tue May 19 19:31:05 2015\, ilmari wrote:
Once more\, with attachment!
ilmari@ilmari.org (Dagfinn Ilmari MannsÄker) writes:
TonyC pointed out on IRC that the description of the «Wide character (U+%X) in %s» warning was copied from perldiag\, so update it there too.
There are a few more instances of "UTF8" in perldiag\, but those should be in a separate ticket/patch (possibly together with a more wide-reaching overhaul).
Thanks\, applied as 50ea4745c8ab3dc6c2e7bfcf895c892b27dae6b4.
I dont know about this change actually. Sorry to say.
See perldoc Encode and look for the section "UTF-8 vs. utf8 vs. UTF8" (quoted below)
In short all three term have subtly different definitions.
Using minor typographical differences to differentiate them is kind of dumb.
I agree\, more or less. On the other hand the flag being name UTF8 is kinda unavoidable\, as hyphens aren't allowed in C identifiers.
At least one of the changes in this patch appears to change the correct use of "utf8" to the incorrect "UTF-8":
@@ -1174\,7 +1174\,7 @@ L\<\C is deprecated in regex|perldiag/"\C is deprecated in regex; marked by \<-- H (D deprecated) The C\<\< /\C/ >> character class was deprecated in v5.20\, and now emits a warning. It is intended that it will become an error in v5.24. This character class matches a single byte even if it appears within a -multi-byte character\, breaks encapsulation\, and can corrupt utf8 +multi-byte character\, breaks encapsulation\, and can corrupt UTF-8 strings.
The original was correct. The regex engine does not care about UTF-8\, it cares about utf8.
Anyone reading it should realize it's working with perl's internal unicode representation\, which is an extended UTF-8. For anyone who doesn't the difference is likely meaningless anyway.
Sorry\, I didnt have time to review it closely to see how much it really matters. My point was just that changing all 'utf8'\, 'UTF8' to 'UTF-8' will subtly change the meaning of what was written.
I've reverted the commit.
Sorry about that. I somehow feel like a party pooper for bringing this up.
cheers. Yves
-- perl -Mre=debug -e "/just|another|perl|hacker/"
p5pRT
commented
9 years ago Oops\, forgot to push the revert\, I'll hold off on it for now.
On Wed\, May 20\, 2015 at 07:16:39PM +0200\, demerphq wrote:
Sorry about that. I somehow feel like a party pooper for bringing this up.
I may have overreacted\, sorry.
Here's the way I think about it:
- unless we need to specifically distinguish between them (as Encode does)\, calling perl's internal encoding UTF-8 is no big deal\, since its intent is to represent Unicode. If we do need to distinguish between them in perldelta then something like "perl's extended UTF-8" is more useful to most readers than "utf8".
- the name of the flag is SVf_UTF8\, but it can be described as the "UTF-8 flag"\, consider the comment in the source:
#define SVf_UTF8 0x20000000 /* SvPV is UTF-8 encoded This is also set on RVs whose overloaded stringification is UTF-8. This might only happen as a side effect of SvPV() */
Using "the UTF8 flag" seems silly to me - name it or describe it\, not something half-way between.
Here's the chunks and my rationale:
-=head2 Better heuristics on older platforms for determining locale UTF8ness +=head2 Better heuristics on older platforms for determining locale UTF-8ness
On platforms that implement neither the C99 standard nor the POSIX 2001 -standard\, determining if the current locale is UTF8 or not depends on +standard\, determining if the current locale is UTF-8 or not depends on heuristics. These are improved in this release.
In this case we're talking about whether the locales support UTF-8 or not. This has nothing to do with perl's internal SVf_UTF8 flag or internal encoding.
I think it belongs.
(D deprecated) The C\<\< /\C/ >> character class was deprecated in v5.20\, and now emits a warning. It is intended that it will become an error in v5.24. This character class matches a single byte even if it appears within a -multi-byte character\, breaks encapsulation\, and can corrupt utf8 +multi-byte character\, breaks encapsulation\, and can corrupt UTF-8 strings.
This is probably a mistake if perldelta needs to distinguish utf8 vs UTF-8.
(W locale) While in a single-byte locale (I\<i.e.>\, a non-UTF-8 one)\, a multi-byte character was encountered. Perl considers this -character to be the specified Unicode code point. Combining non-UTF8 +character to be the specified Unicode code point. Combining non-UTF-8 locales and Unicode is dangerous. Almost certainly some characters will have two different representations. For example\, in the ISO 8859-7 (Greek) locale\, the code point 0xC3 represents a Capital Gamma. But so @@ -2133\,7 +2133\,7 @@ David Mitchell for future work on vtables.
We're talking about whether locales are UTF-8 or not again\, and the paragraph is inconsistent.
I think it belongs.
-Pad names are now always UTF8. The C\
This might need to be "utf8" instead of "UTF8" under the canon according to Encode\, but I think "UTF-8" is better.
-In Perl 5.20.0\, C\<$^N> accidentally had the internal UTF8 flag turned off +In Perl 5.20.0\, C\<$^N> accidentally had the internal UTF-8 flag turned off
Per my attitude above\, I think this change is correct. Or be "had the C\<SVf_UTF8> flag turned off".
if accessed from a code block within a regular expression\, effectively -UTF8-encoding the value. This has been fixed. +UTF-8-encoding the value. This has been fixed. L\<[perl #123135]|https://rt-archive.perl.org/perl5/Ticket/Display.html?id=123135>.
This would need to be "utf8-encoding".
On some systems\, such as VMS\, C\
Under canon the first UTF8 was wrong and the second was correct. I think they should both be "UTF-8".
-C\<\< s///e >> on tainted utf8 strings corrupted C\<\< pos() >>. This bug\, +C\<\< s///e >> on tainted UTF-8 strings corrupted C\<\< pos() >>. This bug\, introduced in 5.20\, is now fixed. L\<[perl #122148]|https://rt-archive.perl.org/perl5/Ticket/Display.html?id=122148>.
Correct under canon.
-Loading UTF8 tables during a regular expression match could cause assertion +Loading UTF-8 tables during a regular expression match could cause assertion failures under debugging builds if the previous match used the very same regular expression. L\<[perl #122747]|https://rt-archive.perl.org/perl5/Ticket/Display.html?id=122747>
This one may have been just plain incorrect. If I understand correctly we load tables that map unicode code points to properties\, not UTF-8 or perl-UTF-8 to properties.
So this should refer to "Loading Unicode tables".
Tony
p5pRT
commented
9 years ago On 05/20/2015 06:44 PM\, Tony Cook wrote:
Oops\, forgot to push the revert\, I'll hold off on it for now.
On Wed\, May 20\, 2015 at 07:16:39PM +0200\, demerphq wrote:
Sorry about that. I somehow feel like a party pooper for bringing this up.
I may have overreacted\, sorry.
Here's the way I think about it:
- unless we need to specifically distinguish between them (as Encode does)\, calling perl's internal encoding UTF-8 is no big deal\, since its intent is to represent Unicode. If we do need to distinguish between them in perldelta then something like "perl's extended UTF-8" is more useful to most readers than "utf8".
+1
- the name of the flag is SVf_UTF8\, but it can be described as the "UTF-8 flag"\, consider the comment in the source:
#define SVf_UTF8 0x20000000 /* SvPV is UTF-8 encoded This is also set on RVs whose overloaded stringification is UTF-8. This might only happen as a side effect of SvPV() */
Using "the UTF8 flag" seems silly to me - name it or describe it\, not something half-way between.
Here's the chunks and my rationale:
-=head2 Better heuristics on older platforms for determining locale UTF8ness +=head2 Better heuristics on older platforms for determining locale UTF-8ness
On platforms that implement neither the C99 standard nor the POSIX 2001 -standard\, determining if the current locale is UTF8 or not depends on +standard\, determining if the current locale is UTF-8 or not depends on heuristics. These are improved in this release.
In this case we're talking about whether the locales support UTF-8 or not. This has nothing to do with perl's internal SVf_UTF8 flag or internal encoding.
I think it belongs.
+1
(D deprecated) The C\<\< /\C/ >> character class was deprecated in v5.20\, and now emits a warning. It is intended that it will become an error in v5.24. This character class matches a single byte even if it appears within a -multi-byte character\, breaks encapsulation\, and can corrupt utf8 +multi-byte character\, breaks encapsulation\, and can corrupt UTF-8 strings.
This is probably a mistake if perldelta needs to distinguish utf8 vs UTF-8.
I don't think perldelta needs to so distinguish. And in particular\, the above should be "UTF-8"
(W locale) While in a single-byte locale (I\<i.e.>\, a non-UTF-8 one)\, a multi-byte character was encountered. Perl considers this -character to be the specified Unicode code point. Combining non-UTF8 +character to be the specified Unicode code point. Combining non-UTF-8 locales and Unicode is dangerous. Almost certainly some characters will have two different representations. For example\, in the ISO 8859-7 (Greek) locale\, the code point 0xC3 represents a Capital Gamma. But so @@ -2133\,7 +2133\,7 @@ David Mitchell for future work on vtables.
We're talking about whether locales are UTF-8 or not again\, and the paragraph is inconsistent.
I think it belongs.
+1
-Pad names are now always UTF8. The C\
macro always returns +Pad names are now always UTF-8. The C\ macro always returns true. Previously\, this was effectively the case already\, but any support for two different internal representations of pad names has now been removed. This might need to be "utf8" instead of "UTF8" under the canon according to Encode\, but I think "UTF-8" is better.
UTF-8 is better.
-In Perl 5.20.0\, C\<$^N> accidentally had the internal UTF8 flag turned off +In Perl 5.20.0\, C\<$^N> accidentally had the internal UTF-8 flag turned off
Per my attitude above\, I think this change is correct.
+1
Or be "had the
C\<SVf_UTF8> flag turned off".
if accessed from a code block within a regular expression\, effectively -UTF8-encoding the value. This has been fixed. +UTF-8-encoding the value. This has been fixed. L\<[perl #123135]|https://rt-archive.perl.org/perl5/Ticket/Display.html?id=123135>.
This would need to be "utf8-encoding".
I hate the sentence anyway. It doesn't make intuitive sense that turning off a flag is the same thing as 'encoding'. To me 'encoding' and 'decoding' have arbitrary non-intuitive meanings which I always have to look up. It's better to not use the terms\, but say something that makes sense to most of the readers who I don't believe have the definitions ingrained.
On some systems\, such as VMS\, C\
can return a non-ASCII string. If a -scalar assigned to had contained a UTF8 string previously\, then C\ -would not turn off the UTF8 flag\, thus corrupting the return value. This +scalar assigned to had contained a UTF-8 string previously\, then C\ +would not turn off the UTF-8 flag\, thus corrupting the return value. This would happen with C\<$lexical = crypt ...>. Under canon the first UTF8 was wrong and the second was correct. I think they should both be "UTF-8".
+1
-C\<\< s///e >> on tainted utf8 strings corrupted C\<\< pos() >>. This bug\, +C\<\< s///e >> on tainted UTF-8 strings corrupted C\<\< pos() >>. This bug\, introduced in 5.20\, is now fixed. L\<[perl #122148]|https://rt-archive.perl.org/perl5/Ticket/Display.html?id=122148>.
Correct under canon.
I prefer UTF-8.
-Loading UTF8 tables during a regular expression match could cause assertion +Loading UTF-8 tables during a regular expression match could cause assertion failures under debugging builds if the previous match used the very same regular expression. L\<[perl #122747]|https://rt-archive.perl.org/perl5/Ticket/Display.html?id=122747>
This one may have been just plain incorrect. If I understand correctly we load tables that map unicode code points to properties\, not UTF-8 or perl-UTF-8 to properties.
So this should refer to "Loading Unicode tables".
Yes
The bottom line is I think we should say UTF-8 in almost every circumstance. The whole Encode thing was a big mistake that should be corrected in 5.24. We now know the perils of not checking input UTF-8 for well-formedness\, and at the time those decisions were made\, those perils were not understood. To put it in terms currently in the news\, we should issue a safety recall on the Encode API in this regard.
p5pRT
commented
9 years ago On 21 May 2015 at 04:01\, Karl Williamson \public@​khwilliamson\.com wrote:
On 05/20/2015 06:44 PM\, Tony Cook wrote:
Oops\, forgot to push the revert\, I'll hold off on it for now.
On Wed\, May 20\, 2015 at 07:16:39PM +0200\, demerphq wrote:
Sorry about that. I somehow feel like a party pooper for bringing this up.
I may have overreacted\, sorry.
Here's the way I think about it:
- unless we need to specifically distinguish between them (as Encode does)\, calling perl's internal encoding UTF-8 is no big deal\, since its intent is to represent Unicode. If we do need to distinguish between them in perldelta then something like "perl's extended UTF-8" is more useful to most readers than "utf8".
+1
I don't agree really. This is a long held distinction.
- the name of the flag is SVf_UTF8\, but it can be described as the "UTF-8 flag"\, consider the comment in the source:
#define SVf_UTF8 0x20000000 /* SvPV is UTF-8 encoded This is also set on RVs whose overloaded stringification is UTF-8. This might only happen as a side effect of SvPV() */
Using "the UTF8 flag" seems silly to me - name it or describe it\, not something half-way between.
I would argue the comment is wrong and should be changed to "utf8".
Here's the chunks and my rationale:
-=head2 Better heuristics on older platforms for determining locale UTF8ness +=head2 Better heuristics on older platforms for determining locale UTF-8ness
On platforms that implement neither the C99 standard nor the POSIX 2001 -standard\, determining if the current locale is UTF8 or not depends on +standard\, determining if the current locale is UTF-8 or not depends on heuristics. These are improved in this release.
In this case we're talking about whether the locales support UTF-8 or not. This has nothing to do with perl's internal SVf_UTF8 flag or internal encoding.
I think it belongs.
+1
No argument on this one.
(D deprecated) The C\<\< /\C/ >> character class was deprecated in v5.20\, and now emits a warning. It is intended that it will become an error in v5.24. This character class matches a single byte even if it appears within a -multi-byte character\, breaks encapsulation\, and can corrupt utf8 +multi-byte character\, breaks encapsulation\, and can corrupt UTF-8 strings.
This is probably a mistake if perldelta needs to distinguish utf8 vs UTF-8.
I don't think perldelta needs to so distinguish. And in particular\, the above should be "UTF-8"
I disagree.
(W locale) While in a single-byte locale (I\<i.e.>\, a non-UTF-8 one)\, a multi-byte character was encountered. Perl considers this -character to be the specified Unicode code point. Combining non-UTF8 +character to be the specified Unicode code point. Combining non-UTF-8 locales and Unicode is dangerous. Almost certainly some characters will have two different representations. For example\, in the ISO 8859-7 (Greek) locale\, the code point 0xC3 represents a Capital Gamma. But so @@ -2133\,7 +2133\,7 @@ David Mitchell for future work on vtables.
We're talking about whether locales are UTF-8 or not again\, and the paragraph is inconsistent.
I think it belongs.
+1
I have no objection to this.
-Pad names are now always UTF8. The C\
macro always returns +Pad names are now always UTF-8. The C\ macro always returns true. Previously\, this was effectively the case already\, but any support for two different internal representations of pad names has now been removed. This might need to be "utf8" instead of "UTF8" under the canon according to Encode\, but I think "UTF-8" is better.
UTF-8 is better.
If we enforce that varnames must be valid UTF-8 (and I think we should) then fine. If we don't then not fine.
For the record (Karl I know you know this)\, UTF-8 is both an encoding\, and also a specification of which codepoints are legal. Not all utf8 sequences are valid UTF-8. I think the distinction is important.
-In Perl 5.20.0\, C\<$^N> accidentally had the internal UTF8 flag turned off +In Perl 5.20.0\, C\<$^N> accidentally had the internal UTF-8 flag turned off
Per my attitude above\, I think this change is correct.
+1
Disagree. It should be "utf8" unless $^N is guaranteed to contain valid UTF-8.
Or be "had the
C\<SVf_UTF8> flag turned off".
if accessed from a code block within a regular expression\, effectively -UTF8-encoding the value. This has been fixed. +UTF-8-encoding the value. This has been fixed. L\<[perl #123135]|https://rt-archive.perl.org/perl5/Ticket/Display.html?id=123135>.
This would need to be "utf8-encoding".
I hate the sentence anyway. It doesn't make intuitive sense that turning off a flag is the same thing as 'encoding'. To me 'encoding' and 'decoding' have arbitrary non-intuitive meanings which I always have to look up. It's better to not use the terms\, but say something that makes sense to most of the readers who I don't believe have the definitions ingrained.
I dont mind using a more descriptive sentence. I do mind conflating UTF-8 and utf8. I can just see someone saying "why does perl let me put surrogate pair code points in a UTF-8 string?".
On some systems\, such as VMS\, C\
can return a non-ASCII string. If a -scalar assigned to had contained a UTF8 string previously\, then C\ -would not turn off the UTF8 flag\, thus corrupting the return value. This +scalar assigned to had contained a UTF-8 string previously\, then C\ +would not turn off the UTF-8 flag\, thus corrupting the return value. This would happen with C\<$lexical = crypt ...>. Under canon the first UTF8 was wrong and the second was correct. I think they should both be "UTF-8".
+1
Disagree.
-C\<\< s///e >> on tainted utf8 strings corrupted C\<\< pos() >>. This bug\, +C\<\< s///e >> on tainted UTF-8 strings corrupted C\<\< pos() >>. This bug\, introduced in 5.20\, is now fixed. L\<[perl #122148]|https://rt-archive.perl.org/perl5/Ticket/Display.html?id=122148>.
Correct under canon.
I prefer UTF-8.
Disagree.
-Loading UTF8 tables during a regular expression match could cause assertion +Loading UTF-8 tables during a regular expression match could cause assertion failures under debugging builds if the previous match used the very same regular expression. L\<[perl #122747]|https://rt-archive.perl.org/perl5/Ticket/Display.html?id=122747>
This one may have been just plain incorrect. If I understand correctly we load tables that map unicode code points to properties\, not UTF-8 or perl-UTF-8 to properties.
So this should refer to "Loading Unicode tables".
Yes
The bottom line is I think we should say UTF-8 in almost every circumstance. The whole Encode thing was a big mistake that should be corrected in 5.24. We now know the perils of not checking input UTF-8 for well-formedness\,
When you say that do you mean that the sequences are wellformed\, or that the codepoints that they map to are properly validated?
I agree about sequence well formedness\, i dont agree about validation. I consider the following to be a perfectly valid program:
perl -wle'my $s=chr(0x10000);'
However $s will not contain UTF-8\, but instead contain utf8.
and at the time those decisions were made\, those perils were not understood. To put it in terms currently in the news\, we should issue a safety recall on the Encode API in this regard.
I think this is really going too far\, and goes against *years* of practice in the Perl community. This is the age old argument about whether strings are arrays of Unicode Codepoints\, or are they packed arrays of integers which happen to us the same encoding rules as that of UTF-8. I don't think we will ever settle that argument. And I don't think you can throw away a decade or more of this distinction just like that.
In short\, as long as we allow UTF-8 forbidden codepoints (eg\, surrogate pairs and codepoints higher than Unicode allows) in our strings then I don't think we should call it UTF-8.
cheers\, Yves
-- perl -Mre=debug -e "/just|another|perl|hacker/"
p5pRT
commented
9 years ago Top posting to cut to the chase:
The use of capitalization and presence or absence of a dash to indicate whether we accept malformed utf8 or not was wrong. Subtle distinctions\, especially ones like these that can be easily overlooked\, shouldn't have such severe consequences. The same argument applies as to whether we accept wellformed utf8 that is in one of the 3 problematic Unicode classes (surrogates\, non-characters\, and above-Unicode code points). That API should be fixed before more damage is done.
I think we should apply "UTF-8" to everything\, and forget about the distinctions. I wouldn't object to uniformly getting rid of the dash. I don't believe we would get sued for doing these things.
The fact that these different spellings have shown up in our documentation is proof that even perl porters don't pay much attention to the distinctions\, so the average programmer is not going to notice at all.
I have no doubt that if Unicode ran out of code points\, that they would simply increase the number available\, with all previous protestations to the contrary becoming null and void. And there are discussions in the Unicode mailing list about doing this that come up from time to time. But that's not going to happen anytime soon. They've assigned about a quarter of the 2**21 so far allocated\, in more than 20 years. At that rate\, it would be more than 60 more years before they would fill up\, even ignoring the fact that the rate of assignment has been decreasing. There could always be some new technology that would gobble up code points: emoji might end up doing that\, but Unicode is hoping to get out of the new emoji business\, and there are prospects of new technology allowing this to happen.
The bottom line IMO is that saying utf8 to mean one thing and UTF-8 to mean a more restricted thing is outrageously wrong\, although try as I might\, I can't quite blame World War I on this decision ;)
On 05/20/2015 11:39 PM\, demerphq wrote:
On 21 May 2015 at 04:01\, Karl Williamson \public@​khwilliamson\.com wrote:
On 05/20/2015 06:44 PM\, Tony Cook wrote:
Oops\, forgot to push the revert\, I'll hold off on it for now.
On Wed\, May 20\, 2015 at 07:16:39PM +0200\, demerphq wrote:
Sorry about that. I somehow feel like a party pooper for bringing this up.
I may have overreacted\, sorry.
Here's the way I think about it:
- unless we need to specifically distinguish between them (as Encode does)\, calling perl's internal encoding UTF-8 is no big deal\, since its intent is to represent Unicode. If we do need to distinguish between them in perldelta then something like "perl's extended UTF-8" is more useful to most readers than "utf8".
+1
I don't agree really. This is a long held distinction.
- the name of the flag is SVf_UTF8\, but it can be described as the "UTF-8 flag"\, consider the comment in the source:
#define SVf_UTF8 0x20000000 /* SvPV is UTF-8 encoded This is also set on RVs whose overloaded stringification is UTF-8. This might only happen as a side effect of SvPV() */
Using "the UTF8 flag" seems silly to me \- name it or describe it\, not something half\-way between\.I would argue the comment is wrong and should be changed to "utf8".
Here's the chunks and my rationale:
-=head2 Better heuristics on older platforms for determining locale UTF8ness +=head2 Better heuristics on older platforms for determining locale UTF-8ness
On platforms that implement neither the C99 standard nor the POSIX 2001 -standard\, determining if the current locale is UTF8 or not depends on +standard\, determining if the current locale is UTF-8 or not depends on heuristics. These are improved in this release.
In this case we're talking about whether the locales support UTF-8 or not. This has nothing to do with perl's internal SVf_UTF8 flag or internal encoding.
I think it belongs.
+1
No argument on this one.
(D deprecated) The C\<\< /\C/ >> character class was deprecated in v5.20\, and now emits a warning. It is intended that it will become an error in v5.24. This character class matches a single byte even if it appears within a -multi-byte character\, breaks encapsulation\, and can corrupt utf8 +multi-byte character\, breaks encapsulation\, and can corrupt UTF-8 strings.
This is probably a mistake if perldelta needs to distinguish utf8 vs UTF-8.
I don't think perldelta needs to so distinguish. And in particular\, the above should be "UTF-8"
I disagree.
(W locale) While in a single-byte locale (I\<i.e.>\, a non-UTF-8 one)\, a multi-byte character was encountered. Perl considers this -character to be the specified Unicode code point. Combining non-UTF8 +character to be the specified Unicode code point. Combining non-UTF-8 locales and Unicode is dangerous. Almost certainly some characters will have two different representations. For example\, in the ISO 8859-7 (Greek) locale\, the code point 0xC3 represents a Capital Gamma. But so @@ -2133\,7 +2133\,7 @@ David Mitchell for future work on vtables.
We're talking about whether locales are UTF-8 or not again\, and the paragraph is inconsistent.
I think it belongs.
+1
I have no objection to this.
-Pad names are now always UTF8. The C\
macro always returns +Pad names are now always UTF-8. The C\ macro always returns true. Previously\, this was effectively the case already\, but any support for two different internal representations of pad names has now been removed. This might need to be "utf8" instead of "UTF8" under the canon according to Encode\, but I think "UTF-8" is better.
UTF-8 is better.
If we enforce that varnames must be valid UTF-8 (and I think we should) then fine. If we don't then not fine.
For the record (Karl I know you know this)\, UTF-8 is both an encoding\, and also a specification of which codepoints are legal. Not all utf8 sequences are valid UTF-8. I think the distinction is important.
-In Perl 5.20.0\, C\<$^N> accidentally had the internal UTF8 flag turned off +In Perl 5.20.0\, C\<$^N> accidentally had the internal UTF-8 flag turned off
Per my attitude above\, I think this change is correct.
+1
Disagree. It should be "utf8" unless $^N is guaranteed to contain valid UTF-8.
Or be "had the
C\<SVf_UTF8> flag turned off".
if accessed from a code block within a regular expression\, effectively -UTF8-encoding the value. This has been fixed. +UTF-8-encoding the value. This has been fixed. L\<[perl #123135]|https://rt-archive.perl.org/perl5/Ticket/Display.html?id=123135>.
This would need to be "utf8-encoding".
I hate the sentence anyway. It doesn't make intuitive sense that turning off a flag is the same thing as 'encoding'. To me 'encoding' and 'decoding' have arbitrary non-intuitive meanings which I always have to look up. It's better to not use the terms\, but say something that makes sense to most of the readers who I don't believe have the definitions ingrained.
I dont mind using a more descriptive sentence. I do mind conflating UTF-8 and utf8. I can just see someone saying "why does perl let me put surrogate pair code points in a UTF-8 string?".
On some systems\, such as VMS\, C\
can return a non-ASCII string. If a -scalar assigned to had contained a UTF8 string previously\, then C\ -would not turn off the UTF8 flag\, thus corrupting the return value. This +scalar assigned to had contained a UTF-8 string previously\, then C\ +would not turn off the UTF-8 flag\, thus corrupting the return value. This would happen with C\<$lexical = crypt ...>. Under canon the first UTF8 was wrong and the second was correct. I think they should both be "UTF-8".
+1
Disagree.
-C\<\< s///e >> on tainted utf8 strings corrupted C\<\< pos() >>. This bug\, +C\<\< s///e >> on tainted UTF-8 strings corrupted C\<\< pos() >>. This bug\, introduced in 5.20\, is now fixed. L\<[perl #122148]|https://rt-archive.perl.org/perl5/Ticket/Display.html?id=122148>.
Correct under canon.
I prefer UTF-8.
Disagree.
-Loading UTF8 tables during a regular expression match could cause assertion +Loading UTF-8 tables during a regular expression match could cause assertion failures under debugging builds if the previous match used the very same regular expression. L\<[perl #122747]|https://rt-archive.perl.org/perl5/Ticket/Display.html?id=122747>
This one may have been just plain incorrect. If I understand correctly we load tables that map unicode code points to properties\, not UTF-8 or perl-UTF-8 to properties.
So this should refer to "Loading Unicode tables".
Yes
The bottom line is I think we should say UTF-8 in almost every circumstance. The whole Encode thing was a big mistake that should be corrected in 5.24. We now know the perils of not checking input UTF-8 for well-formedness\,
When you say that do you mean that the sequences are wellformed\, or that the codepoints that they map to are properly validated?
I agree about sequence well formedness\, i dont agree about validation. I consider the following to be a perfectly valid program:
perl -wle'my $s=chr(0x10000);'
However $s will not contain UTF-8\, but instead contain utf8.
and at the time those decisions were made\, those perils were not understood. To put it in terms currently in the news\, we should issue a safety recall on the Encode API in this regard.
I think this is really going too far\, and goes against *years* of practice in the Perl community. This is the age old argument about whether strings are arrays of Unicode Codepoints\, or are they packed arrays of integers which happen to us the same encoding rules as that of UTF-8. I don't think we will ever settle that argument. And I don't think you can throw away a decade or more of this distinction just like that.
In short\, as long as we allow UTF-8 forbidden codepoints (eg\, surrogate pairs and codepoints higher than Unicode allows) in our strings then I don't think we should call it UTF-8.
cheers\, Yves
p5pRT
commented
9 years ago On Wed May 20 22:39:33 2015\, demerphq wrote:
I think this is really going too far\, and goes against *years* of practice in the Perl community. This is the age old argument about whether strings are arrays of Unicode Codepoints\, or are they packed arrays of integers which happen to us the same encoding rules as that of UTF-8. I don't think we will ever settle that argument. And I don't think you can throw away a decade or more of this distinction just like that.
In short\, as long as we allow UTF-8 forbidden codepoints (eg\, surrogate pairs and codepoints higher than Unicode allows) in our strings then I don't think we should call it UTF-8.
How about the attached?
These only touch the sentences modified by the original patch that you didn't agree with and makes no attempt to correct other uses of "UTF-8".
Tony no attempt to update other
p5pRT
commented
9 years ago p5pRT
commented
9 years ago p5pRT
commented
9 years ago * Karl Williamson \public@​khwilliamson\.com [2015-05-21T23:50:55]
Top posting to cut to the chase:
The use of capitalization and presence or absence of a dash to indicate whether we accept malformed utf8 or not was wrong. Subtle distinctions\, especially ones like these that can be easily overlooked\, shouldn't have such severe consequences.
I agree completely.
(Also\, although I do not think anybody is arguing this\, I want to get it said that I'm not proposing we take any direct action against 5.22.0 as a result of this agreement!) :)
I think we should apply "UTF-8" to everything\, and forget about the distinctions. I wouldn't object to uniformly getting rid of the dash. I don't believe we would get sued for doing these things.
WellâŠ
Okay\, I think it's fair to say that I basically agree with this\, but I think it's something we have to do correctly. I don't think it's enough to just do a mechanical replacement. (I don't mean to imply that you suggested this.)
Here is a very quick from-the-hip summary of thoughts:
* we actually need to talk about UTF-8 pretty rarely\, since we normally only need to talk about "strings" * when we talk about I/O boundaries where we truck in UTF-8\, we should be explicit in what is allowed or not; "will decode UTF-8\, permitting overlong sequences and trans-0x10FFFF characters" or other variants on that * when we talk about representation in memory\, we should do the same\, but having a short name for the form of UTF-8-ish encoding used for all internal representations seems useful; it just shouldn't be "utf8" * we absolutely must document the difference between UTF-8 and utf8 for the sake of understanding documentation that still uses this distinction * I would *love* if we could always refer to "scalars with the SvUTF8 flag" as such\, rather than anything that makes the flag sound like something useful to the person writing Perl. If I thought we could rename SvUTF8 to SvWIMR or something\, I would point hungrily at that solution. * probably we will find an impulse to agonize on each doc update; I predict that in most cases\, it will not be worth it; the right answer will be obvious\, if we have clearer terms
So\, I don't think this is a zero-effort change. I think it is worth it\, though\, because it is confusing to newcomers for no reason. This isn't a situation where "if you don't surprise the beginner sometimes\, you'll end up surprising the experts instead." This is just a shibboleth.
In short\, I think we can find clearer language\, and I think that in many cases\, we worry about distinctions where they are irrelevant\, and can probably settle on UTF-8 in many more cases than we do.
-- rjbs
p5pRT
commented
9 years ago On Tue\, May 26\, 2015 at 07:53:53AM -0400\, Ricardo Signes wrote:
In short\, I think we can find clearer language\, and I think that in many cases\, we worry about distinctions where they are irrelevant\, and can probably settle on UTF-8 in many more cases than we do.
I posted a patch to the ticket hopefully doing that that\, but it didn't seem to make it through to the list:
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=125221#txn-1350100
Tony
p5pRT
commented
9 years ago On Tue May 26 04:54:36 2015\, perl.p5p@rjbs.manxome.org wrote:
* Karl Williamson \public@​khwilliamson\.com [2015-05-21T23:50:55]
Top posting to cut to the chase:
The use of capitalization and presence or absence of a dash to indicate whether we accept malformed utf8 or not was wrong. Subtle distinctions\, especially ones like these that can be easily overlooked\, shouldn't have such severe consequences.
I agree completely.
(Also\, although I do not think anybody is arguing this\, I want to get it said that I'm not proposing we take any direct action against 5.22.0 as a result of this agreement!) :)
I think we should apply "UTF-8" to everything\, and forget about the distinctions. I wouldn't object to uniformly getting rid of the dash. I don't believe we would get sued for doing these things.
WellâŠ
Okay\, I think it's fair to say that I basically agree with this\, but I think it's something we have to do correctly. I don't think it's enough to just do a mechanical replacement. (I don't mean to imply that you suggested this.)
Here is a very quick from-the-hip summary of thoughts:
* we actually need to talk about UTF-8 pretty rarely\, since we normally only need to talk about "strings" * when we talk about I/O boundaries where we truck in UTF-8\, we should be explicit in what is allowed or not; "will decode UTF-8\, permitting overlong sequences and trans-0x10FFFF characters" or other variants on that * when we talk about representation in memory\, we should do the same\, but having a short name for the form of UTF-8-ish encoding used for all internal representations seems useful; it just shouldn't be "utf8" * we absolutely must document the difference between UTF-8 and utf8 for the sake of understanding documentation that still uses this distinction * I would *love* if we could always refer to "scalars with the SvUTF8 flag" as such\, rather than anything that makes the flag sound like something useful to the person writing Perl. If I thought we could rename SvUTF8 to SvWIMR or something\, I would point hungrily at that solution. * probably we will find an impulse to agonize on each doc update; I predict that in most cases\, it will not be worth it; the right answer will be obvious\, if we have clearer terms
So\, I don't think this is a zero-effort change. I think it is worth it\, though\, because it is confusing to newcomers for no reason. This isn't a situation where "if you don't surprise the beginner sometimes\, you'll end up surprising the experts instead." This is just a shibboleth.
In short\, I think we can find clearer language\, and I think that in many cases\, we worry about distinctions where they are irrelevant\, and can probably settle on UTF-8 in many more cases than we do.
The original purpose of this ticket is complete\, as the patch to the 5.22.0 perldelta has been applied\, and 5.22.0 has been released.
Should a new ticket be opened for discussing the future use of UTF-8/utf8/Unicode?
Tony
p5pRT
commented
9 years ago On Mon Jul 20 22:37:40 2015\, tonyc wrote:
The original purpose of this ticket is complete\, as the patch to the 5.22.0 perldelta has been applied\, and 5.22.0 has been released.
So closing this ticket.
Should a new ticket be opened for discussing the future use of UTF- 8/utf8/Unicode?
No response\, no new ticket.
Tony
p5pRT
commented
9 years ago @tonycoz - Status changed from 'open' to 'resolved'
Migrated from rt.perl.org#125221 (status was 'resolved')
Searchable as RT125221$