[Fwd: Patch for perl utf8-related bug]

[p5pRT]

Migrated from rt.perl.org#18932 (status was 'resolved')

Searchable as RT18932$

[p5pRT]

From rbh00@utsglobal.com

Hi\, Hugo

I hope you're the right person to send this repeated email to.

Please see the attached email. I sent it November 6\, and as near as I can tell it has not been seen by you. In fact\, now that I search my records\, I don't think I even got an automated response for it.

There is a bug in test case t/op/pat.t when run on a Perl 5.8.0 built for the UTS operating system that runs on an IBM S/390 mainframe. The attached email reports the bug and explains it and presents a patch that will fix the problem. It's a platform problem\, and the fix is good for all platforms.

I hope this email gets through. I'll give it a week only\, this time\, and then I'll send it again.

Cheers to you and thanks for your work on perl.

Richard Hitt rbh00@​utsglobal.com

[p5pRT]

From rbh00@utsglobal.com

Message RFC822: From - Wed Nov 6 16:28:48 2002 X-Mozilla-Status2: 00000000 Received: from sysdump.utsglobal.com (sysdump.utsglobal.com [129.212.1.39]) by mail.utsglobal.com (Switch-2.0.1/Switch-2.0.1) with ESMTP id gA70aSG12888 for rbh00@utsglobal.com; Wed, 6 Nov 2002 16:36:28 -0800 (PST) Received: (from rbh00@localhost) by sysdump.utsglobal.com (8.11.2/8.11.2) id gA70RXj08618; Wed, 6 Nov 2002 16:27:33 -0800 Date: Wed, 6 Nov 2002 16:27:33 -0800 From: Richard Hitt rbh00@utsglobal.com Message-Id: 200211070027.gA70RXj08618@sysdump.utsglobal.com To: perlbug@perl.org Subject: Patch for perl utf8-related bug Cc: rbh00@utsglobal.com

Hi

Here is a patch to fix a utf8-related bug in perl. It applies cleanly to perl-5.8.0-RC3. It should work with all platforms.

The problem is with saving and restoring of PL_reg_match_utf8. This object is defined as a bool (see thrdvar.h). But in regcomp.c at line 5071 in Perl_save_re_context() it is saved as a char (with SAVEI8).

The bool type is variously defined as an int, a char, and an enum (see handy.h), so using SAVEI8 would work in the char case, and for little-endian platforms also in the int and probably enum case. But our platform is uts on the IBM Mainframe, big-endian, and bool is an int for us. SAVEI8 saves the byte at the wrong end of our int.

What's needed is a SAVEBOOL() and the attendant infrastructure, and this patch provides it, and of course changes line 5071 to do a SAVEBOOL instead of a SAVEI8.

Here is the diffstat: embed.h | 1 + perl.h | 1 + proto.h | 1 + regcomp.c | 2 +- scope.c | 13 +++++++++++++ scope.h | 4 ++++ 6 files changed, 21 insertions, 1 deletion

I hope you'll see fit to include this patch in your Perl tree. As I say, it checks out under our uts platform; and also it checks out under Linux on Intel. I looked for a Perl CVS repository to check it against but found none; is there one somewhere?

Thanks.

Richard Hitt rbh00@utsglobal.com

-------- start patch diff -urN VANILLAperl-5.8.0-RC3/embed.h perl-5.8.0-RC3/embed.h --- VANILLAperl-5.8.0-RC3/embed.h Sat Jul 13 17:02:35 2002 +++ perl-5.8.0-RC3/embed.h Wed Nov 6 13:25:37 2002 @@ -592,6 +592,7 @@

define save_iv Perl_save_iv

define save_list Perl_save_list

define save_long Perl_save_long

+#define save_bool Perl_save_bool

define save_mortalizesv Perl_save_mortalizesv

define save_nogv Perl_save_nogv

define save_op Perl_save_op

diff -urN VANILLAperl-5.8.0-RC3/perl.h perl-5.8.0-RC3/perl.h --- VANILLAperl-5.8.0-RC3/perl.h Thu Jun 20 06:30:34 2002 +++ perl-5.8.0-RC3/perl.h Wed Nov 6 13:25:37 2002 @@ -2163,6 +2163,7 @@ I32 any_i32; IV any_iv; long any_long;

• bool any_bool; void (any_dptr) (void); void (anydxptr) (pTHX void); }; diff -urN VANILLAperl-5.8.0-RC3/proto.h perl-5.8.0-RC3/proto.h --- VANILLAperl-5.8.0-RC3/proto.h Sat Jul 13 17:02:32 2002 +++ perl-5.8.0-RC3/proto.h Wed Nov 6 13:57:02 2002 @@ -690,6 +690,7 @@ PERL_CALLCONV void Perl_saveiv(pTHX IV* iv); PERL_CALLCONV void Perl_savelist(pTHX SV* sarg, I32 maxsarg); PERL_CALLCONV void Perl_savelong(pTHX long longp); +PERL_CALLCONV void Perl_savebool(pTHX bool boolp); PERL_CALLCONV void Perl_savemortalizesv(pTHX SV sv); PERL_CALLCONV void Perl_savenogv(pTHX GV gv); PERL_CALLCONV void Perl_save_op(pTHX); diff -urN VANILLAperl-5.8.0-RC3/regcomp.c perl-5.8.0-RC3/regcomp.c --- VANILLAperl-5.8.0-RC3/regcomp.c Mon Jul 8 13:10:49 2002 +++ perl-5.8.0-RC3/regcomp.c Wed Nov 6 13:25:37 2002 @@ -5068,7 +5068,7 @@ SAVEVPTR(PL_reg_re); / from regexec.c / SAVEPPTR(PL_reg_ganch); / from regexec.c / SAVESPTR(PL_reg_sv); / from regexec.c */
• SAVEI8(PL_reg_match_utf8); / from regexec.c /

• SAVEBOOL(PL_reg_match_utf8); / from regexec.c / SAVEVPTR(PL_reg_magic); / from regexec.c / SAVEI32(PL_reg_oldpos); / from regexec.c / SAVEVPTR(PL_reg_oldcurpm); / from regexec.c / diff -urN VANILLAperl-5.8.0-RC3/scope.c perl-5.8.0-RC3/scope.c --- VANILLAperl-5.8.0-RC3/scope.c Sat Jun 1 10:03:45 2002 +++ perl-5.8.0-RC3/scope.c Wed Nov 6 13:25:37 2002 @@ -391,6 +391,15 @@ }

  void +Perl_savebool(pTHX bool *boolp) +{

• SSCHECK(3);
• SSPUSHBOOL(*boolp);
• SSPUSHPTR(boolp);
• SSPUSHINT(SAVEt_BOOL); +}

• +void Perl_saveI32(pTHX I32 intp) { SSCHECK(3); @@ -794,6 +803,10 @@ case SAVEt_LONG: / long reference / ptr = SSPOPPTR; (long*)ptr = (long)SSPOPLONG;

• break;
• case SAVEt_BOOL: / bool reference /
• ptr = SSPOPPTR;

• (bool)ptr = (bool)SSPOPBOOL; break; case SAVEt_I32: / I32 reference / ptr = SSPOPPTR; diff -urN VANILLAperl-5.8.0-RC3/scope.h perl-5.8.0-RC3/scope.h --- VANILLAperl-5.8.0-RC3/scope.h Sat Jun 1 10:03:45 2002 +++ perl-5.8.0-RC3/scope.h Wed Nov 6 13:25:37 2002 @@ -45,6 +45,7 @@

  define SAVEt_PADSV 35

  define SAVEt_MORTALIZESV 36

  define SAVEt_SHARED_PVREF 37

  +#define SAVEt_BOOL 38

  ifndef SCOPE_SAVES_SIGNAL_MASK

  define SCOPE_SAVES_SIGNAL_MASK 0

  @@ -53,12 +54,14 @@

  define SSCHECK(need) if (PL_savestack_ix + need > PL_savestack_max) savestack_grow()

  define SSPUSHINT(i) (PL_savestack[PL_savestack_ix++].any_i32 = (I32)(i))

  define SSPUSHLONG(i) (PL_savestack[PL_savestack_ix++].any_long = (long)(i))

  +#define SSPUSHBOOL(p) (PL_savestack[PL_savestack_ix++].any_bool = (p))

  define SSPUSHIV(i) (PL_savestack[PL_savestack_ix++].any_iv = (IV)(i))

  define SSPUSHPTR(p) (PL_savestack[PL_savestack_ix++].any_ptr = (void*)(p))

  define SSPUSHDPTR(p) (PL_savestack[PL_savestack_ix++].any_dptr = (p))

  define SSPUSHDXPTR(p) (PL_savestack[PL_savestack_ix++].any_dxptr = (p))

  define SSPOPINT (PL_savestack[--PL_savestack_ix].any_i32)

  define SSPOPLONG (PL_savestack[--PL_savestack_ix].any_long)

  +#define SSPOPBOOL (PL_savestack[--PL_savestack_ix].any_bool)

  define SSPOPIV (PL_savestack[--PL_savestack_ix].any_iv)

  define SSPOPPTR (PL_savestack[--PL_savestack_ix].any_ptr)

  define SSPOPDPTR (PL_savestack[--PL_savestack_ix].any_dptr)

  @@ -116,6 +119,7 @@

  define SAVEINT(i) save_int(SOFT_CAST(int*)&(i))

  define SAVEIV(i) save_iv(SOFT_CAST(IV*)&(i))

  define SAVELONG(l) save_long(SOFT_CAST(long*)&(l))

  +#define SAVEBOOL(b) save_bool(SOFT_CAST(bool*)&(b))

  define SAVESPTR(s) save_sptr((SV**)&(s))

  define SAVEPPTR(s) save_pptr(SOFT_CAST(char**)&(s))

  define SAVEVPTR(s) save_vptr((void*)&(s))

  -------- end patch

[p5pRT]

From @hvds

:Hi\, Hugo : :I hope you're the right person to send this repeated email to.

I am\, but note that the address you sent to is that of the bug database rather than me. You can also send patches direct to the perl porters mailing list\, as above.

:Please see the attached email. I sent it November 6\, and as near as I :can tell it has not been seen by you. In fact\, now that I search my :records\, I don't think I even got an automated response for it.

I'm not sure why I didn't see it\, nor why I didn't see this second copy​: the bug database should have forwarded it to the mailing list automatically. Luckily a colleague noticed it and pointed me at it.

:Here is a patch to fix a utf8-related bug in perl. It applies cleanly :to perl-5.8.0-RC3. It should work with all platforms. : :The problem is with saving and restoring of PL_reg_match_utf8. :This object is defined as a bool (see thrdvar.h). But in regcomp.c at :line 5071 in Perl_save_re_context() it is saved as a char (with SAVEI8). : :The bool type is variously defined as an int\, a char\, and an enum :(see handy.h)\, so using SAVEI8 would work in the char case\, and for :little-endian platforms also in the int and probably enum case. But our :platform is uts on the IBM Mainframe\, big-endian\, and bool is an int :for us. SAVEI8 saves the byte at the wrong end of our int. : :What's needed is a SAVEBOOL() and the attendant infrastructure\, and :this patch provides it\, and of course changes line 5071 to do a SAVEBOOL :instead of a SAVEI8.

Thanks for an excellent analysis\, and a well thought out patch. Note that the changes to embed.h and proto.h are better done by changing embed.fnc and running 'make regen_headers'; that'll make sure that details of the new function are propagated in the right way to all the necessary files.

I've applied your patch (with suitable modifications to embed.fnc and dependencies) to the development sources as change #18312.

:I looked for a Perl CVS repository to check it against but :found none; is there one somewhere?

The perl source is currently held under a perforce repository but with somewhat restricted access. However you can get hold of the individual changes via FTP\, and the latest bleading-edge source is rsyncable; see the section on 'Keeping in sync' in the perlhack manpage.

Hugo

[p5pRT]

From @jhi

The patch got applied\, so I'm marking the problem ticket as resolved.

[p5pRT]

@jhi - Status changed from 'new' to 'resolved'