Closed p5pRT closed 13 years ago
This series of commits delivers the Unicode 6.0 db\, and upgrades Perl to use it. There may still be some work to do in Unicode::UCD to support the new characters (which I'll investigate)\, but the rest of the Perl core should fully support it.
The few code changes are attached to this email\, but the bulk of the changes (along with the attachments here)\, too large to email\, are located at git://github.com/khwilliamson/perl.git branch mktables
Those changes are essentially entirely official Unicode data\, except for the MANIFEST\, perldelta\, version\, and a couple data changes in UCD.t
On Tue\, 12 Oct 2010 21:56:17 -0700\, karl williamson (via RT) \perlbug\-followup@​perl\.org wrote:
This series of commits delivers the Unicode 6.0 db\, and upgrades Perl to use it. There may still be some work to do in Unicode::UCD to support the new characters (which I'll investigate)\, but the rest of the Perl core should fully support it.
Wow.
Don't forget to also update Module::CoreList
$ corelist -a Unicode
Unicode was first released with perl v5.6.2 v5.6.2 3.0.1 v5.8.0 3.2.0 v5.8.1 4.0.0 v5.8.2 4.0.0 v5.8.3 4.0.0 v5.8.4 4.0.1 v5.8.5 4.0.1 v5.8.6 4.0.1 v5.8.7 4.1.0 v5.8.8 4.1.0 v5.8.9 5.1.0 v5.9.0 4.0.0 v5.9.1 4.0.0 v5.9.2 4.0.1 v5.9.3 4.1.0 v5.9.4 4.1.0 v5.9.5 5.0.0 v5.10.0 5.0.0 v5.10.1 5.1.0 v5.11.0 5.1.0 v5.11.1 5.1.0 v5.11.2 5.1.0 v5.11.3 5.2.0 v5.11.4 5.2.0 v5.11.5 5.2.0 v5.12.0 5.2.0 v5.12.1 5.2.0 v5.12.2 5.2.0 v5.13.0 5.2.0 v5.13.1 5.2.0 v5.13.2 5.2.0 v5.13.3 5.2.0 v5.13.4 5.2.0
-- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using 5.00307 through 5.12 and porting perl5.13.x on HP-UX 10.20\, 11.00\, 11.11\, 11.23 and 11.31\, OpenSuSE 10.1\, 11.0 .. 11.3 and AIX 5.2 and 5.3. http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
The RT System itself - Status changed from 'new' to 'open'
I added a little more detail in perldelta\, and attached is some extra wording on steps to take to deliver a new Unicode DB\, including changing pod.lst to the new number. Everything is pushed to git://github.com/khwilliamson/perl.git branch mktables
H.Merijn Brand wrote:
Don't forget to also update Module::CoreList
That looks to me to be part of the monthly release manager's job. The attached patch mentions the need to update it. Thanks for pointing it out.
Also\, I misspoke when I said\,
but the rest of the Perl core should fully support it.
We do not fully support the Unicode standard; what I meant was 6.0 is supported as well as 5.2.
On Wed\, 13 Oct 2010 09:48:48 -0600\, karl williamson \public@​khwilliamson\.com wrote:
H.Merijn Brand wrote:
Don't forget to also update Module::CoreList
That looks to me to be part of the monthly release manager's job.
Yes\, but as Unicode is a property of the implementation\, and not a separate module\, it is most likely not auto-detected.
The attached patch mentions the need to update it. Thanks for pointing it out.
Also\, I misspoke when I said\,
but the rest of the Perl core should fully support it.
We do not fully support the Unicode standard; what I meant was 6.0 is supported as well as 5.2.
-- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using 5.00307 through 5.12 and porting perl5.13.x on HP-UX 10.20\, 11.00\, 11.11\, 11.23 and 11.31\, OpenSuSE 10.1\, 11.0 .. 11.3 and AIX 5.2 and 5.3. http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
On Tue Oct 12 21:56:17 2010\, public@khwilliamson.com wrote:
This series of commits delivers the Unicode 6.0 db\, and upgrades Perl to use it. There may still be some work to do in Unicode::UCD to support the new characters (which I'll investigate)\, but the rest of the Perl core should fully support it.
The few code changes are attached to this email\, but the bulk of the changes (along with the attachments here)\, too large to email\, are located at git://github.com/khwilliamson/perl.git branch mktables
Those changes are essentially entirely official Unicode data\, except for the MANIFEST\, perldelta\, version\, and a couple data changes in UCD.t
I’ve applied the first patch as 92f9d56c66. With the Unicode 6 database I get a test failure:
$ curl http://github.com/khwilliamson/perl/commit/35e84e1c3151243.patch | git am [...] $ cd t $ ./perl harness -v ../lib/charnames.t [...] not ok 17078 - Verify string_vianame("BELL") is chr(0x1F514) # Failed at ../lib/charnames.t line 105 # got "\a" # expected "\x{1f514}"
Father Chrysostomos via RT wrote:
On Tue Oct 12 21:56:17 2010\, public@khwilliamson.com wrote:
This series of commits delivers the Unicode 6.0 db\, and upgrades Perl to use it. There may still be some work to do in Unicode::UCD to support the new characters (which I'll investigate)\, but the rest of the Perl core should fully support it.
The few code changes are attached to this email\, but the bulk of the changes (along with the attachments here)\, too large to email\, are located at git://github.com/khwilliamson/perl.git branch mktables
Those changes are essentially entirely official Unicode data\, except for the MANIFEST\, perldelta\, version\, and a couple data changes in UCD.t
I’ve applied the first patch as 92f9d56c66. With the Unicode 6 database I get a test failure:
$ curl http://github.com/khwilliamson/perl/commit/35e84e1c3151243.patch | git am [...] $ cd t $ ./perl harness -v ../lib/charnames.t [...] not ok 17078 - Verify string_vianame("BELL") is chr(0x1F514) # Failed at ../lib/charnames.t line 105 # got "\a" # expected "\x{1f514}"
I'm afraid this is what I consider to be a flaw in the new standard\, though they wouldn't; I regret that I did not find it before it was too late; as your tests are the first it surfaced. I'm not sure Unicode would have listened to me anyway\, but we would have known about this earlier.
Your tests showed the problem and my tests didn't\, because of the random sampling of the tests\, because it would take too long to go through all million possible code points each time; and my tests just didn't try that combination yet.
I'm not sure what to do; suggestions welcome.
The problem stems from the fact that the Standard does not give names to the control characters\, such as ACK and BEL. It did in version 1.0\, and it still publishes those names as the "Unicode_1_Name" property. That name for character 0x07\, known by the acronym BEL\, is "BELL". What Perl does is to use the Unicode 1 names when there is no current. All was fine until 6.0 came along and re-used BELL for a different character.
But as far as Unicode is concerned\, there isn't a problem\, as BEL has no official name. It is Perl who has persisted in using this old name. I don't know why Unicode removed the names; and it seems eminently reasonable to give them names; but here we are.
The only option I can think of that doesn't violate our stability policies is to\, in 5.14\, keep the old BELL meaning\, but deprecate it\, saying to use BEL instead\, which was added in 5.13 as a synonym for it. This means that in 5.14 we don't accept that one new Unicode character\, except by ordinal value. In 5.16\, we convert to use Unicode.
In the meantime\, I will propose that Unicode adopt a policy of not doing this again\, and perhaps an alias that gives a somewhat different name\, just to clear up future confusion.
karl williamson wrote:
Father Chrysostomos via RT wrote:
On Tue Oct 12 21:56:17 2010\, public@khwilliamson.com wrote:
This series of commits delivers the Unicode 6.0 db\, and upgrades Perl to use it. There may still be some work to do in Unicode::UCD to support the new characters (which I'll investigate)\, but the rest of the Perl core should fully support it.
The few code changes are attached to this email\, but the bulk of the changes (along with the attachments here)\, too large to email\, are located at git://github.com/khwilliamson/perl.git branch mktables
Those changes are essentially entirely official Unicode data\, except for the MANIFEST\, perldelta\, version\, and a couple data changes in UCD.t
I’ve applied the first patch as 92f9d56c66. With the Unicode 6 database I get a test failure:
$ curl http://github.com/khwilliamson/perl/commit/35e84e1c3151243.patch | git am [...] $ cd t $ ./perl harness -v ../lib/charnames.t [...] not ok 17078 - Verify string_vianame("BELL") is chr(0x1F514) # Failed at ../lib/charnames.t line 105 # got "\a" # expected "\x{1f514}"
I'm afraid this is what I consider to be a flaw in the new standard\, though they wouldn't; I regret that I did not find it before it was too late; as your tests are the first it surfaced. I'm not sure Unicode would have listened to me anyway\, but we would have known about this earlier.
Your tests showed the problem and my tests didn't\, because of the random sampling of the tests\, because it would take too long to go through all million possible code points each time; and my tests just didn't try that combination yet.
I'm not sure what to do; suggestions welcome.
The problem stems from the fact that the Standard does not give names to the control characters\, such as ACK and BEL. It did in version 1.0\, and it still publishes those names as the "Unicode_1_Name" property. That name for character 0x07\, known by the acronym BEL\, is "BELL". What Perl does is to use the Unicode 1 names when there is no current. All was fine until 6.0 came along and re-used BELL for a different character.
But as far as Unicode is concerned\, there isn't a problem\, as BEL has no official name. It is Perl who has persisted in using this old name. I don't know why Unicode removed the names; and it seems eminently reasonable to give them names; but here we are.
The only option I can think of that doesn't violate our stability policies is to\, in 5.14\, keep the old BELL meaning\, but deprecate it\, saying to use BEL instead\, which was added in 5.13 as a synonym for it. This means that in 5.14 we don't accept that one new Unicode character\, except by ordinal value. In 5.16\, we convert to use Unicode.
In the meantime\, I will propose that Unicode adopt a policy of not doing this again\, and perhaps an alias that gives a somewhat different name\, just to clear up future confusion.
The attached patches work around this problem by deprecating \N{BELL} for 5.14\, and giving the new name \N{ALERT} to it. The new character with that name will be unnamed. This means that Perl 5.14 doesn't quite support Unicode 6.0.
The patches are also available at: git://github.com/khwilliamson/perl.git branch uni6
which includes the entire series of unicode 6 patches.
On 13 October 2010 19:16\, karl williamson \public@​khwilliamson\.com wrote:
In the meantime\, I will propose that Unicode adopt a policy of not doing this again\, and perhaps an alias that gives a somewhat different name\, just to clear up future confusion.
And register a *very* *strong* note of protest at the current action.
They need to look at their standard in the same way we do in terms of backwards compatibility.
Once published it cant be removed for two major releases. And must go through a deprecation cycle.
I think that you need to be very aggressive about the last point.
cheers\, Yves
-- perl -Mre=debug -e "/just|another|perl|hacker/"
On Tue Nov 16 17:44:57 2010\, public@khwilliamson.com wrote:
karl williamson wrote:
Father Chrysostomos via RT wrote:
On Tue Oct 12 21:56:17 2010\, public@khwilliamson.com wrote:
This series of commits delivers the Unicode 6.0 db\, and upgrades Perl to use it. There may still be some work to do in Unicode::UCD to support the new characters (which I'll investigate)\, but the rest of the Perl core should fully support it.
The few code changes are attached to this email\, but the bulk of the changes (along with the attachments here)\, too large to email\, are located at git://github.com/khwilliamson/perl.git branch mktables
Those changes are essentially entirely official Unicode data\, except for the MANIFEST\, perldelta\, version\, and a couple data changes in UCD.t
I’ve applied the first patch as 92f9d56c66. With the Unicode 6 database I get a test failure:
$ curl http://github.com/khwilliamson/perl/commit/35e84e1c3151243.patch | git am [...] $ cd t $ ./perl harness -v ../lib/charnames.t [...] not ok 17078 - Verify string_vianame("BELL") is chr(0x1F514) # Failed at ../lib/charnames.t line 105 # got "\a" # expected "\x{1f514}"
I'm afraid this is what I consider to be a flaw in the new standard\, though they wouldn't; I regret that I did not find it before it was too late; as your tests are the first it surfaced. I'm not sure Unicode would have listened to me anyway\, but we would have known about this earlier.
Your tests showed the problem and my tests didn't\, because of the random sampling of the tests\, because it would take too long to go through all million possible code points each time; and my tests just didn't try that combination yet.
I'm not sure what to do; suggestions welcome.
The problem stems from the fact that the Standard does not give names to the control characters\, such as ACK and BEL. It did in version 1.0\, and it still publishes those names as the "Unicode_1_Name" property. That name for character 0x07\, known by the acronym BEL\, is "BELL". What Perl does is to use the Unicode 1 names when there is no current. All was fine until 6.0 came along and re-used BELL for a different character.
But as far as Unicode is concerned\, there isn't a problem\, as BEL has no official name. It is Perl who has persisted in using this old name. I don't know why Unicode removed the names; and it seems eminently reasonable to give them names; but here we are.
The only option I can think of that doesn't violate our stability policies is to\, in 5.14\, keep the old BELL meaning\, but deprecate it\, saying to use BEL instead\, which was added in 5.13 as a synonym for it. This means that in 5.14 we don't accept that one new Unicode character\, except by ordinal value. In 5.16\, we convert to use Unicode.
In the meantime\, I will propose that Unicode adopt a policy of not doing this again\, and perhaps an alias that gives a somewhat different name\, just to clear up future confusion.
The attached patches work around this problem by deprecating \N{BELL} for 5.14\, and giving the new name \N{ALERT} to it. The new character with that name will be unnamed. This means that Perl 5.14 doesn't quite support Unicode 6.0.
The patches are also available at: git://github.com/khwilliamson/perl.git branch uni6
which includes the entire series of unicode 6 patches.
Thank you. All applied.
(Why did you not use ALARM?)
@cpansprout - Status changed from 'open' to 'resolved'
Father Chrysostomos via RT wrote:
On Tue Nov 16 17:44:57 2010\, public@khwilliamson.com wrote:
karl williamson wrote:
Father Chrysostomos via RT wrote:
On Tue Oct 12 21:56:17 2010\, public@khwilliamson.com wrote:
This series of commits delivers the Unicode 6.0 db\, and upgrades Perl to use it. There may still be some work to do in Unicode::UCD to support the new characters (which I'll investigate)\, but the rest of the Perl core should fully support it.
The few code changes are attached to this email\, but the bulk of the changes (along with the attachments here)\, too large to email\, are located at git://github.com/khwilliamson/perl.git branch mktables
Those changes are essentially entirely official Unicode data\, except for the MANIFEST\, perldelta\, version\, and a couple data changes in UCD.t I’ve applied the first patch as 92f9d56c66. With the Unicode 6 database I get a test failure:
$ curl http://github.com/khwilliamson/perl/commit/35e84e1c3151243.patch | git am [...] $ cd t $ ./perl harness -v ../lib/charnames.t [...] not ok 17078 - Verify string_vianame("BELL") is chr(0x1F514) # Failed at ../lib/charnames.t line 105 # got "\a" # expected "\x{1f514}"
I'm afraid this is what I consider to be a flaw in the new standard\, though they wouldn't; I regret that I did not find it before it was too late; as your tests are the first it surfaced. I'm not sure Unicode would have listened to me anyway\, but we would have known about this earlier.
Your tests showed the problem and my tests didn't\, because of the random sampling of the tests\, because it would take too long to go through all million possible code points each time; and my tests just didn't try that combination yet.
I'm not sure what to do; suggestions welcome.
The problem stems from the fact that the Standard does not give names to the control characters\, such as ACK and BEL. It did in version 1.0\, and it still publishes those names as the "Unicode_1_Name" property. That name for character 0x07\, known by the acronym BEL\, is "BELL". What Perl does is to use the Unicode 1 names when there is no current. All was fine until 6.0 came along and re-used BELL for a different character.
But as far as Unicode is concerned\, there isn't a problem\, as BEL has no official name. It is Perl who has persisted in using this old name. I don't know why Unicode removed the names; and it seems eminently reasonable to give them names; but here we are.
The only option I can think of that doesn't violate our stability policies is to\, in 5.14\, keep the old BELL meaning\, but deprecate it\, saying to use BEL instead\, which was added in 5.13 as a synonym for it. This means that in 5.14 we don't accept that one new Unicode character\, except by ordinal value. In 5.16\, we convert to use Unicode.
In the meantime\, I will propose that Unicode adopt a policy of not doing this again\, and perhaps an alias that gives a somewhat different name\, just to clear up future confusion.
The attached patches work around this problem by deprecating \N{BELL} for 5.14\, and giving the new name \N{ALERT} to it. The new character with that name will be unnamed. This means that Perl 5.14 doesn't quite support Unicode 6.0.
The patches are also available at: git://github.com/khwilliamson/perl.git branch uni6
which includes the entire series of unicode 6 patches.
Thank you. All applied.
(Why did you not use ALARM?)
Thanks for applying these.
Alert is a semi-standard name for it\, used in K&R\, for example. I have never hear of "ALARM".
On Thu Nov 18 14:29:39 2010\, public@khwilliamson.com wrote:
Alert is a semi-standard name for it\, used in K&R\, for example. I have never hear of "ALARM".
$ ack -i '\\a.*alarm' pod pod/perlop.pod 1029: \a alarm (bell) (BEL)
pod/perlre.pod 230: \a alarm (bell) (BEL)
pod/perlrebackslash.pod 67: \a Alarm or bell. 120: \a 7 07 BEL \cG alarm or bell
pod/perlreref.pod 86: \a Alarm (beep)
So I thought ‘alarm’ was standard. But ‘alert’ makes more sense.
Migrated from rt.perl.org#78354 (status was 'resolved')
Searchable as RT78354$