Closed p5pRT closed 10 years ago
This is a bug report for perl from fawaka@gmail.com\, generated with the help of perlbug 1.39 running under perl 5.16.1.
I recently took the liberty to convert our asian readme files from euc-* to utf-8\, but I didn't update their contents. I can't understand them\, but they still seem to advocate «use encoding». Someone who actually groks the languages should probably update their content.
Flags: category=docs severity=low
Site configuration information for perl 5.16.1:
Configured by leon at Thu Aug 9 12:30:13 EEST 2012.
Summary of my perl5 (revision 5 version 16 subversion 1) configuration:
Platform: osname=linux\, osvers=3.2.0-27-generic\, archname=x86_64-linux-thread-multi uname='linux leon-laptop 3.2.0-27-generic #43-ubuntu smp fri jul 6 14:25:57 utc 2012 x86_64 x86_64 x86_64 gnulinux ' config_args='-de -Dprefix=/home/leon/perl5/perlbrew/perls/perl-5.16.1 -Duseshrplib -Dusethreads' hint=recommended\, useposix=true\, d_sigaction=define useithreads=define\, usemultiplicity=define useperlio=define\, d_sfio=undef\, uselargefiles=define\, usesocks=undef use64bitint=define\, use64bitall=define\, uselongdouble=undef usemymalloc=n\, bincompat5005=undef Compiler: cc='cc'\, ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'\, optimize='-O2'\, cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include' ccversion=''\, gccversion='4.6.3'\, gccosandvers='' intsize=4\, longsize=8\, ptrsize=8\, doublesize=8\, byteorder=12345678 d_longlong=define\, longlongsize=8\, d_longdbl=define\, longdblsize=16 ivtype='long'\, ivsize=8\, nvtype='double'\, nvsize=8\, Off_t='off_t'\, lseeksize=8 alignbytes=8\, prototype=define Linker and Libraries: ld='cc'\, ldflags =' -fstack-protector -L/usr/local/lib' libpth=/usr/local/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/lib libs=-lnsl -ldb -ldl -lm -lcrypt -lutil -lpthread -lc perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc libc=\, so=so\, useshrplib=true\, libperl=libperl.so gnulibc_version='2.15' Dynamic Linking: dlsrc=dl_dlopen.xs\, dlext=so\, d_dlsymun=undef\, ccdlflags='-Wl\,-E -Wl\,-rpath\,/home/leon/perl5/perlbrew/perls/perl-5.16.1/lib/5.16.1/x86_64-linux-thread-multi/CORE' cccdlflags='-fPIC'\, lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector'
Locally applied patches:
@INC for perl 5.16.1: /home/leon/perl5/perlbrew/perls/perl-5.16.1/lib/site_perl/5.16.1/x86_64-linux-thread-multi /home/leon/perl5/perlbrew/perls/perl-5.16.1/lib/site_perl/5.16.1 /home/leon/perl5/perlbrew/perls/perl-5.16.1/lib/5.16.1/x86_64-linux-thread-multi /home/leon/perl5/perlbrew/perls/perl-5.16.1/lib/5.16.1 .
Environment for perl 5.16.1: HOME=/home/leon LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_ADDRESS=en_US.UTF-8 LC_IDENTIFICATION=en_US.UTF-8 LC_MEASUREMENT=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_NAME=en_US.UTF-8 LC_NUMERIC=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_TELEPHONE=en_US.UTF-8 LC_TIME=en_US.UTF-8 LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/leon/perl5/perlbrew/bin:/home/leon/perl5/perlbrew/perls/perl-5.16.1/bin:/home/leon/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games PERLBREW_HOME=/home/leon/.perlbrew PERLBREW_PATH=/home/leon/perl5/perlbrew/bin:/home/leon/perl5/perlbrew/perls/perl-5.16.1/bin PERLBREW_PERL=perl-5.16.1 PERLBREW_ROOT=/home/leon/perl5/perlbrew PERLBREW_VERSION=0.25 PERL_BADLANG (unset) SHELL=/bin/bash
On Thu Apr 25 13:28:15 2013\, LeonT wrote:
This is a bug report for perl from fawaka@gmail.com\, generated with the help of perlbug 1.39 running under perl 5.16.1.
-----------------------------------------------------------------
I recently took the liberty to convert our asian readme files from euc-* to utf-8\, but I didn't update their contents. I can't understand them\, but they still seem to advocate �use encoding�. Someone who actually groks the languages should probably update their content.
Are these all the files you were referring to?
README.cn README.jp README.ko
The RT System itself - Status changed from 'new' to 'open'
On 05/08/2013 07:24 PM\, James E Keenan via RT wrote:
On Thu Apr 25 13:28:15 2013\, LeonT wrote:
This is a bug report for perl from fawaka@gmail.com\, generated with the help of perlbug 1.39 running under perl 5.16.1.
-----------------------------------------------------------------
I recently took the liberty to convert our asian readme files from euc-* to utf-8\, but I didn't update their contents. I can't understand them\, but they still seem to advocate �use encoding�. Someone who actually groks the languages should probably update their content.
Are these all the files you were referring to?
README.cn README.jp README.ko
--- via perlbug: queue: perl5 status: new https://rt-archive.perl.org/perl5/Ticket/Display.html?id=117765
I don't know if there are more\, but these 3 are definitely ones he modified. (I used git blame on the files to check.)
On 05/08/2013 08:20 PM\, Karl Williamson wrote:
On 05/08/2013 07:24 PM\, James E Keenan via RT wrote:
On Thu Apr 25 13:28:15 2013\, LeonT wrote:
This is a bug report for perl from fawaka@gmail.com\, generated with the help of perlbug 1.39 running under perl 5.16.1.
-----------------------------------------------------------------
I recently took the liberty to convert our asian readme files from euc-* to utf-8\, but I didn't update their contents. I can't understand them\, but they still seem to advocate �use encoding�. Someone who actually groks the languages should probably update their content.
Are these all the files you were referring to?
README.cn README.jp README.ko
--- via perlbug: queue: perl5 status: new https://rt-archive.perl.org/perl5/Ticket/Display.html?id=117765
I don't know if there are more\, but these 3 are definitely ones he modified. (I used git blame on the files to check.)
I ran perldoc -o html README.jp > jp.html
then read that file with my browser\, and ran Google translate on it\, and got the following:
NAME
perljp - Japanese Perl Guide Description
Welcome to the world of Perl!
From Perl 5.8.0\, Unicode support is greatly enhanced support for character code of non-Latin characters have been added\, including (Chinese\, Japanese\, Korean) CJK the result. Unicode is a standard that aims to treat the character code of a single character in the world\, (Greek\, Cyrillic\, Arabic\, Hebrew\, Deevana gari character character in between west\, or even from the east\, etc.)\, and\, so far has already includes Macintosh) and (PC character the OS vendor has established its own.
Perl itself works with Unicode. Regular expression or string literal a Perl script is assumed Unicode. And for input and output\, module corresponding to the different character code\, which had been used\, "Encode" has been standard equipment\, it has to make it easy interconversion of these two character codes to Unicode and You.
Character code Encode supports at this time are as follows.
7bit-jis AdobeStandardEncoding AdobeSymbol AdobeZdingbat ascii big5 big5-hkscs cp1006 cp1026 cp1047 cp1250 cp1251 cp1252 cp1253 cp1254 cp1255 cp1256 cp1257 cp1258 cp37 cp424 cp437 cp500 cp737 cp775 cp850 cp852 cp855 cp856 cp857 cp860 cp861 cp862 cp863 cp864 cp865 cp866 cp869 cp874 cp875 cp932 cp936 cp949 cp950 dingbats euc-cn euc-jp euc-kr gb12345-raw gb2312-raw gsm0338 hp-roman8 hz iso-2022-jp iso-2022-jp-1 iso-8859-1 iso-8859-10 iso-8859-11 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-ir-165 jis0201-raw jis0208-raw jis0212-raw johab koi8-f koi8-r koi8-u ksc5601-raw MacArabic MacCentralEurRoman MacChineseSimp MacChineseTrad MacCroatian MacCyrillic MacDingbats MacFarsi MacGreek MacHebrew MacIcelandic MacJapanese MacKorean MacRoman MacRomanian MacRumanian MacSami MacSymbol MacThai MacTurkish MacUkrainian nextstep posix-bc shiftjis symbol UCS-2BE UCS-2LE UTF-16 UTF-16BE UTF-16LE UTF-32 UTF-32BE UTF-32LE utf8 viscii
(All 114 types)
For example\, to convert files to UTF-8 character encoding FOO\, do the following.
perl-Mencoding = FOO\, STDOUT\, utf8-pe1 \<file.FOO> file.utf8
In addition\, the Perl\, character code conversion utility written in Perl is all\, piconv Because the attached\, I can be as follows.
piconv-f FOO-t utf8 \<file.FOO> file.utf8 piconv-f utf8-t FOO \<file.utf8> file.FOO
About (jcode.pl | Jcode.pm | JPerl)
I was able to handle only literal of 5.8 previous script is in the EUC-JP. In addition\, Jcode.pm is (http://openlab.ring.gr.jp/Jcode/)\, jcode.pl exists as each utility perl4 for the CGI to handle the Japanese language as a module to handle the input and output It seems that you know also whether not a few that is often used. However\, you can handle both of regular expressions in Japanese was not possible.
The Perl 5.005 earlier\, localized versions that specializes in Japanese\, Jperl there were the (http://homepage2.nifty.com/kipp/perl/jperl/index.html). In addition\, Perl of Mac OS 9.x/Classic for Japanese version of MacPerl also were present as MacJPerl. It was also possible to treat regular expressions in Japanese Shift_JIS that also treat the addition to EUC-JP as the character code\, and also possible in (http://habilis.net/macjperl/). These.
In Perl5.8\, you can on these features can be realized only by Perl itself all\, all\, dealing at the same time And the character code above 114 not only Japanese. In addition\, we can now also be easily used to get a new module of the character code for the like CPAN.
Input and output
The following example will output Izure be converted to EUC-JP input Shift_JIS.
# Jcode.pl require "jcode.pl"; while (\<>) { jcode :: convert (* _\, 'euc'\, 'sjis'); print; } # Jcode.pm use Jcode; while (\<>) { print Jcode-> new ($ _\, 'sjis') -> euc; } # Perl 5.8 use Encode; while (\<>) { from_to ($ _\, 'shiftjis'\, 'euc-jp'); print; } # 5.8 Perl - by using the encoding use encoding 'euc-jp'\, STDIN => 'shiftjis'; while (\<>) { print; }
Jperl compatible script
It seems you only need to change the so-called "shebang"\, that it is available without changing most of the script of Jperl for.
#! / Path / to / jperl ↓ #! / Path / to / perl-Mencoding = euc-jp
Please refer to the perldoc encoding for more information.
Learn More
Tomes comes with a Perl\, (unfortunately\, although there is little in English)\, such as the use of Encode module and Unicode support\, new features and Perl has been covered over. It is possible to view some of them with the following command.
Unicode support in general perldoc perlunicode # Perl respect perldoc Encode # Encode module respect perldoc Encode :: JP # out Japanese character code
The URL about Perl in general
http://www.perl.com/
Perl Home Page (O'Reilly and Associates) http://www.cpan.org/
CPAN (Comprehensive Perl Archive Network) http://lists.perl.org/
Perl mailing list collection
URL to help you learn the Perl
http://www.oreilly.com.tw/
Perl Books of O'Reilly's (Traditional Chinese) http://www.oreilly.com.cn/
Perl Books of O'Reilly's (Simplified Chinese) http://www.oreilly.co.jp/catalog/
Perl Books of O'Reilly (Japanese)
Perl user group
http://www.pm.org/groups/asia.html
URL of the Unicode-related
http://www.unicode.org/
(Select organization of the Unicode standard) Unicode Consortium http://www.cl.cam.ac.uk/% 7Emgk25/unicode.html
UTF-8 and Unicode FAQ for Unix / Linux http://wiki.kldp.org/Translations/html/UTF8-Unicode-KLDP/UTF8-Unicode-KLDP.html
UTF-8 and Unicode FAQ for Unix / Linux (Korean translation)
AUTHORS
Jarkko Hietaniemi \jhi@​iki\.fi Dan Kogai (small shepherd bullet) \dankogai@​dan\.co\.jp
On Thu\, May 9\, 2013 at 3:24 AM\, James E Keenan via RT \perlbug\-followup@​perl\.org wrote:
Are these all the files you were referring to?
README.cn README.jp README.ko
Yeah\, and also README.tw.
Leon
On Thu May 09 01:44:48 2013\, LeonT wrote:
On Thu\, May 9\, 2013 at 3:24 AM\, James E Keenan via RT \perlbug\-followup@​perl\.org wrote:
Are these all the files you were referring to?
README.cn README.jp README.ko
Yeah\, and also README.tw.
Leon
I removed all references of encoding in READEM.tw
The first one was rewritten with equivalent "Encode" code to demonstrate the same functionality. The second part was rewritten to demonstrate how perl can do character-based operations on utf8 source code.
The patch is attached.
On Sat May 11 11:11:32 2013\, gugod@gugod.org wrote:
The patch is attached.
Thanks\, applied as e4ea314.
I also subsequently applied 55b2202 and d7b1992. Welcome to the perl 5 AUTHORS file\, gugod!
-- rjbs
On Sun Jun 09 08:40:43 2013\, rjbs wrote:
On Sat May 11 11:11:32 2013\, gugod@gugod.org wrote:
The patch is attached.
Thanks\, applied as e4ea314.
I also subsequently applied 55b2202 and d7b1992. Welcome to the perl 5 AUTHORS file\, gugod!
Do we have any criteria for determining when this ticket becomes closable?
Thank you very much. Jim Keenan
* James E Keenan via RT \perlbug\-followup@​perl\.org [2013-06-13T22:11:53]
On Sun Jun 09 08:40:43 2013\, rjbs wrote:
On Sat May 11 11:11:32 2013\, gugod@gugod.org wrote:
The patch is attached.
Thanks\, applied as e4ea314.
I also subsequently applied 55b2202 and d7b1992. Welcome to the perl 5 AUTHORS file\, gugod!
Do we have any criteria for determining when this ticket becomes closable?
Yes. I have put out a call for review by native readers. Once I get that review for each of the relevant files and take action accordingly\, I will close this ticket.
I haven't set a deadline yet\, but I may keep this open through September so I can call for review in person at YAPC::Asia if needed.
-- rjbs
On Thu Jun 13 19:17:04 2013\, perl.p5p@rjbs.manxome.org wrote:
I haven't set a deadline yet\, but I may keep this open through September so I can call for review in person at YAPC::Asia if needed.
The patch for korean version of README is attached.
- Removed old-fashioned encoding related example - Translated original README into README.ko - Updated several local Perl links for Korean
On Thu Jul 04 10:25:33 2013\, keedi wrote:
On Thu Jun 13 19:17:04 2013\, perl.p5p@rjbs.manxome.org wrote:
I haven't set a deadline yet\, but I may keep this open through September so I can call for review in person at YAPC::Asia if needed.
The patch for korean version of README is attached.
- Removed old-fashioned encoding related example - Translated original README into README.ko - Updated several local Perl links for Korean
Thank you. Applied as 277d344a7. I added you to the AUTHORS file in e9da8ca7b8.
--
Father Chrysostomos
Father Chrysostomos notes:
At the very least\, we should rewrite the command lines that make use of encoding.pm. E.g.\, README.ko has this:
perl -Mencoding=euc-kr\,STDOUT\,utf8 -pe1 \< file.euc-kr > file.utf8
-- rjbs
I'd like to find volunteers for README review of the outstanding languages\, but I don't think this ticket is the right way forward.
-- rjbs
@rjbs - Status changed from 'open' to 'resolved'
Migrated from rt.perl.org#117765 (status was 'resolved')
Searchable as RT117765$