Perl UTF-8

[damonkohler]

From @GoogleCodeExporter on May 31, 2015 11:23

What steps will reproduce the problem?
1. utf8::encode("text"); executes fine
2. utf8::encode("täxt"); fails

What is the expected output? What do you see instead?

I hoped this will print utf8 encoded text (actually into an output file in
my real scenario).

Instead I got a perl interpreter backtrace. See below.

What version of the product are you using? On what operating system?

ase r13, perl 5.10.0

Please provide any additional information below.

Here's the backtrace:

$ export AP_PORT="58112"
/data/data/com.google.ase/perl/perl /sdcard/ase/scripts/now.pl
$ i 2009/09/26 21:38:35 /sdcard/ase/scripts/now.pl starts
o 2009/09/26 21:38:35
i 2009/09/26 21:39:12 noch einmal
i 2009/09/26 21:39:29 hmmh
i 2009/09/26 21:40:42 ah, problem tritt by ascii text nicht auf
Can't locate utf8.pm

in @INC (@INC contains:
/data/data/com.google.ase/perl/5.10.0/arm-eabi-linux
/data/data/com.google.ase/perl/5.10.0
/data/data/com.google.ase/perl/site_perl/5.10.0/arm-eabi-linux
/sdcard/ase/extras/perl/site_perl/5.10.0 /sdcard/ase/extras/perl/site_perl
.) at /sdcard/ase/extras/perl/site_perl/JSON/PP.pm line 1100, <GEN0> line 5.
BEGIN failed--compilation aborted, <GEN0> line 5.
$

utf8 support for ASE perl would be very important because text editors I've
also used on the android phone seem to all standardize on utf8 encoding
(which is a good idea, I think).

Any hope to get this soon?

Have phone, will test.

Adrian

Original issue reported on code.google.com by adrian.a...@gmail.com on 26 Sep 2009 at 8:17

Copied from original issue: damonkohler/android-scripting#108

[damonkohler]

From @GoogleCodeExporter on May 31, 2015 11:23

I have switched to beanshell since I can't see how this would work in perl with 
utf8
not available in the installation.

I saw the same problem, but with following naive patch I can produce files with 
utf8
encoding from a .bsh script:

e.g.
PrintWriter out = new PrintWriter(new OutputStreamWriter(timelog, "UTF-8"), 
true);

--- e:\ase\extras\bsh\android.bsh.orig  2009-09-27 13:29:24.000000000 +0200
+++ e:\ase\extras\bsh\android.bsh   2009-09-27 23:17:50.000000000 +0200
@@ -18,11 +18,11 @@
   String AP_PORT = System.getenv().get("AP_PORT");
   Socket conn = new Socket("127.0.0.1", Integer.decode(AP_PORT));
   BufferedReader in = new BufferedReader(
-      new InputStreamReader(conn.getInputStream(), "8859_1"));
+      new InputStreamReader(conn.getInputStream(), "UTF-8"));
   OutputStream out_stream = new BufferedOutputStream(
       conn.getOutputStream());
   PrintWriter out = new PrintWriter(
-      new OutputStreamWriter(out_stream, "8859_1"), true);
+      new OutputStreamWriter(out_stream, "UTF-8"), true);
   int id = 0;

   call(String method, JSONArray params) {

Original comment by adrian.a...@gmail.com on 1 Oct 2009 at 7:05

[damonkohler]

From @GoogleCodeExporter on May 31, 2015 11:23

Original comment by damonkoh...@gmail.com on 1 Oct 2009 at 7:16

[damonkohler]

From @GoogleCodeExporter on May 31, 2015 11:23

I added utf8 to the list of modules-to-package, so whenever Damon cuts a new 
release it should be there 
(http://code.google.com/p/android-scripting/source/detail?
r=65eff52aff3a0bd7ac064156d5061c2592b744a9)

However, I feel that I have to point out that the utf8 module is very rarely 
the thing ou want for "utf8" or 
"Unicode".  Please read "perluniintro" if this doesn't sound familiar.  (As 
with the beanshell code above, it's 
usually more of a question of doing Unicode I/O.

I'm transferring this ticket back to Damon for the bsh fix.

Original comment by jhietaniemi on 7 Oct 2009 at 10:32

[damonkohler]

From @GoogleCodeExporter on May 31, 2015 11:23

Original comment by jhietaniemi on 7 Oct 2009 at 10:33

[damonkohler]

From @GoogleCodeExporter on May 31, 2015 11:23

Original comment by jhietaniemi on 7 Oct 2009 at 10:35

• Changed title: bsh UTF-8 fix

[damonkohler]

From @GoogleCodeExporter on May 31, 2015 11:23

In Python you need to keep in mind to declare your encoding.
# -*- coding: utf-8 -*-

Otherwise it crashes when you encounter foreign characters.
print('über cool')

SyntaxError: Non-ASCII character '\xc3' in file ...

Original comment by pdietz84 on 2 Dec 2009 at 5:09

[damonkohler]

From @GoogleCodeExporter on May 31, 2015 11:23

Is there any news on getting a fix for Perl sorted anytime soon? JSON/PP.pm 
makes 
*heavy* use of various encode/decode UTF8 routines. As JSON is a main route for 
traffic between Perl, ASE, and the Android stack, it's quite a big deal that 
there's 
no utf8 support yet.

I'll happily look into fixing this issue if no one else is currently...

Original comment by alexel...@gmail.com on 27 May 2010 at 9:56

[damonkohler]

From @GoogleCodeExporter on May 31, 2015 11:23

Hi alexelder,

Sawyer would probably be interested in any help you're will to provide.

Original comment by damonkoh...@gmail.com on 28 May 2010 at 7:19

• Changed title: Perl UTF-8

[damonkohler]

From @GoogleCodeExporter on May 31, 2015 11:23

If the only thing missing is utf8.pm, that shouldn't be a problem at all.

I've added an assert script for this on my perl-android-scripts github repo.

Original comment by xsawy...@gmail.com on 28 May 2010 at 10:10