pack "w" /* this cannot happen ;-) */ is fallacious

[p5pRT]

Migrated from rt.perl.org#17772 (status was 'resolved')

Searchable as RT17772$

[p5pRT]

From @nwc10

I can't remember if this $expletive pair of bugs ever got a bug number

1​: 5.6.x pp.c has this comment​:

  do {   double next = floor(adouble / 128);   *--in = (unsigned char)(adouble - (next * 128)) | 0x80;   if (in \<= buf) /* this cannot happen ;-) */   DIE(aTHX_ "Cannot compress integer");   in--;   adouble = next;   } while (adouble > 0);

er\, right\, for some value of cannot​:

$ perl5.6.1-64-Os -le 'pack "w"\, ~0' Cannot compress integer at -e line 1.

Secondly\, it's an undocumented warning​:

$ perl5.6.1-64-Os -Mdiagnostics -le 'print pack "w"\, ~0' Uncaught exception from user code​:   Cannot compress integer at -e line 1.

While the first is fixed in 5.8\, the second is not\, although it's harder to provoke​:

perl5.9.0 -Mdiagnostics -le 'print length pack "w"\, 2**800' Uncaught exception from user code​:   Cannot compress integer at -e line 1.

perl5.6.1-64-Os -V is​:

Summary of my perl5 (revision 5.0 version 6 subversion 1) configuration​:   Platform​:   osname=freebsd\, osvers=4.5-stable\, archname=i386-freebsd-64int   uname='freebsd thinking-cap.moo 4.5-stable freebsd 4.5-stable #1​: wed feb 6 16​:15​:14 gmt 2002 nick@​thinking-cap.moo​:stuffusrsrcsyscompilethinkingcap i386 '   config_args='-e'   hint=recommended\, useposix=true\, d_sigaction=define   usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef   useperlio=undef d_sfio=undef uselargefiles=define usesocks=undef   use64bitint=define use64bitall=undef uselongdouble=undef   Compiler​:   cc='ccache gcc'\, ccflags ='-fno-strict-aliasing -I/usr/local/include'\,   optimize='-Os'\,   cppflags='-fno-strict-aliasing -I/usr/local/include'   ccversion=''\, gccversion='2.95.3 20010315 (release) [FreeBSD]'\, gccosandvers=''   intsize=4\, longsize=4\, ptrsize=4\, doublesize=8\, byteorder=12345678   d_longlong=define\, longlongsize=8\, d_longdbl=define\, longdblsize=12   ivtype='long long'\, ivsize=8\, nvtype='double'\, nvsize=8\, Off_t='off_t'\, lseeksize=8   alignbytes=4\, usemymalloc=n\, prototype=define   Linker and Libraries​:   ld='ccache gcc'\, ldflags ='-Wl\,-E -L/usr/local/lib'   libpth=/usr/lib /usr/local/lib   libs=-lm -lc -lcrypt -lutil   perllibs=-lm -lc -lcrypt -lutil   libc=\, so=so\, useshrplib=false\, libperl=libperl.a   Dynamic Linking​:   dlsrc=dl_dlopen.xs\, dlext=so\, d_dlsymun=undef\, ccdlflags=' '   cccdlflags='-DPIC -fpic'\, lddlflags='-shared -L/usr/local/lib'

Characteristics of this binary (from libperl)​:   Compile-time options​: USE_64_BIT_INT USE_LARGE_FILES   Built under freebsd   Compiled at May 18 2002 14​:48​:12   @​INC​:   /usr/local/lib/perl5/5.6.1/i386-freebsd-64int   /usr/local/lib/perl5/5.6.1   /usr/local/lib/perl5/site_perl/5.6.1/i386-freebsd-64int   /usr/local/lib/perl5/site_perl/5.6.1   /usr/local/lib/perl5/site_perl/5.005   /usr/local/lib/perl5/site_perl   .

This is a bug report mainly to let off steam because I can't find a way to compare the bit pattern of all 64 bits of an integer on 5.6.1 built with 64 bits.

All large (20 decimal digit) strings get converted to integers via this code in 5.6.1's sv.c​:

  if (numtype & IS_NUMBER_NOT_IV) {   /* May be not an integer. Need to cache NV if we cache IV   * - otherwise future conversion to NV will be wrong. */   NV d;

  d = Atof(SvPVX(sv));

  if (SvTYPE(sv) \< SVt_PVNV)   sv_upgrade(sv\, SVt_PVNV);   SvNVX(sv) = d;   (void)SvNOK_on(sv);   (void)SvIOK_on(sv); #if defined(USE_LONG_DOUBLE)   DEBUG_c(PerlIO_printf(Perl_debug_log\,   "0x%"UVxf" 2nv(%" PERL_PRIgldbl ")\n"\,   PTR2UV(sv)\, SvNVX(sv))); #else   DEBUG_c(PerlIO_printf(Perl_debug_log\,   "0x%"UVxf" 2nv(%g)\n"\,   PTR2UV(sv)\, SvNVX(sv))); #endif   if (SvNVX(sv) \< -0.5) {   SvIVX(sv) = I_V(SvNVX(sv));   goto ret_zero;   } else {   SvUVX(sv) = U_V(SvNVX(sv));   SvIsUV_on(sv);   }

which has just subjected it to an intermediate NV cast. ${expletive}\, as there go my low order bits. That code path is used even *inside* use integer;

And my last hope\, pack "w"\, which does decimal string arithmetic\, doesn't help me because of this​:

perl5.6.1-64-Os -le 'pack "w"\, 18446744073709551615' Cannot compress integer at -e line 1.

(although if I write it as a string\, it works

perl5.6.1-64-Os -le 'pack "w"\, "18446744073709551615"' )

but then that doesn't solve my problems\, because I have negative numbers to compare\, and -1 isn't "-1"​:

perl -wle '$a = -1; $b = "-1"; print "$a "\,$a; print "$b "\,$b' -1 4294967295 -1 -1

Who said ~~ was the scalar context operator? \

(remember\, I can't use numeric -\, as that forces a conversion via sv_2uv\, and sv_2uv is converting via floating point for some of the values I'm interested in)

Oops. That was a rant. use 5.8.0; # 64 bit stuff works

Nicholas Clark -- Even better than the real thing​: http​://nms-cgi.sourceforge.net/

[p5pRT]

From @nwc10

On Sun\, Oct 06\, 2002 at 10​:14​:46PM -0000\, Nicholas Clark wrote​:

Secondly\, it's an undocumented warning​:

$ perl5.6.1-64-Os -Mdiagnostics -le 'print pack "w"\, ~0' Uncaught exception from user code​: Cannot compress integer at -e line 1.

While the first is fixed in 5.8\, the second is not\, although it's harder to provoke​:

perl5.9.0 -Mdiagnostics -le 'print length pack "w"\, 2**800' Uncaught exception from user code​: Cannot compress integer at -e line 1.

Let's make that 3​:

$ perl5.9.0 -Mdiagnostics -le 'print length pack "w"\, -1' Uncaught exception from user code​:   Cannot compress negative numbers at -e line 1. $ perl5.9.0 -Mdiagnostics -le 'print length pack "w"\, "11111111111e0"' Uncaught exception from user code​:   can compress only unsigned integer at -e line 1. $ perl5.9.0 -Mdiagnostics -le 'print length pack "w"\, 2**800' Uncaught exception from user code​:   Cannot compress integer at -e line 1.

There are two code paths that can croak with "can compress only unsigned integer" \, and I believe I'm hitting the first. I think I'd need a tied value to hit the second.

Appended patch makes it only possible to hit "cannot happen" (the third) for compressing Inf\, documents all three in perldiag.pod\, and changes the   "can compress only unsigned integer" to   "Can only compress unsigned integers" to be better English and capitalised consistently with all the other warnings in perldiag.pod (It's not documented. Why would anyone be depending on the exact wording)

For an IEEE double the buffer is 147 bytes. The calculation is a constant\, so the compiler's constant folding should get it right. I believe the calculation is optimally correct\, because the position of the croak() in the loop was wrong (it wasn't using 1 byte of the buffer) and before I fixed that I could croak it at 2**1022 and 2**1023.

Call me pointlessly obsessed\, but perl doesn't favour arbitrary limitations\, and having a 16 or so byte buffer is arbitrary\, particularly when I can calculate the longest buffer needed\, and that buffer is not excessively large. Even if it makes no sense to pass numbers that big to pack 'w' :-)

I believe my "alphabetical" ordering for perldiag is correct - it seems to be tr/A-Za-z/a-za-z/dc and then sort (case insensitive\, drop all non alphabetics)

(sorry\, I'm getting speed obsessed\, and that should be faster than a regexp)

Nicholas Clark -- Even better than the real thing​: http​://nms-cgi.sourceforge.net/

Inline Patch

```diff --- pod/perldiag.pod.orig Thu Oct 3 22:46:34 2002 +++ pod/perldiag.pod Tue Oct 8 21:23:52 2002 @@ -466,6 +466,24 @@ checking. Alternatively, if you are cer function correctly, you may put an ampersand before the name to avoid the warning. See L. +=item Can only compress unsigned integers + +(F) An argument to pack("w",...) was not an integer. The BER compressed +integer format can only be used with positive integers, and you attempted +to compress something else. See L. + +=item Cannot compress integer + +(F) An argument to pack("w",...) was too large to compress. The BER +compressed integer format can only be used with positive integers, and you +attempted to compress Infinity or a very large number (> 1e308). +See L. + +=item Cannot compress negative numbers + +(F) An argument to pack("w",...) was negative. The BER compressed integer +format can only be used with positive integers. See L. + =item / cannot take a count (F) You had an unpack template indicating a counted-length string, but --- pp_pack.c.orig Fri Jun 14 02:44:56 2002 +++ pp_pack.c Tue Oct 8 21:37:29 2002 @@ -2286,7 +2286,7 @@ Perl_pack_cat(pTHX_ SV *cat, char *pat, /* Copy string and check for compliance */ from = SvPV(fromstr, len); if ((norm = is_an_int(from, len)) == NULL) - Perl_croak(aTHX_ "can compress only unsigned integer"); + Perl_croak(aTHX_ "Can only compress unsigned integers"); New('w', result, len, char); in = result + len; @@ -2299,15 +2299,25 @@ Perl_pack_cat(pTHX_ SV *cat, char *pat, SvREFCNT_dec(norm); /* free norm */ } else if (SvNOKp(fromstr)) { - char buf[sizeof(NV) * 2]; /* 8/7 <= 2 */ + /* 10**NV_MAX_10_EXP is the largest power of 10 + so 10**(NV_MAX_10_EXP+1) is definately unrepresentable + given 10**(NV_MAX_10_EXP+1) == 128 ** x solve for x: + x = (NV_MAX_10_EXP+1) * log (10) / log (128) + And with that many bytes only Inf can overflow. + */ +#ifdef NV_MAX_10_EXP + char buf[1 + (int)((NV_MAX_10_EXP + 1) * 0.47456)]; +#else + char buf[1 + (int)((308 + 1) * 0.47456)]; +#endif char *in = buf + sizeof(buf); anv = Perl_floor(anv); do { NV next = Perl_floor(anv / 128); - *--in = (unsigned char)(anv - (next * 128)) | 0x80; if (in <= buf) /* this cannot happen ;-) */ Perl_croak(aTHX_ "Cannot compress integer"); + *--in = (unsigned char)(anv - (next * 128)) | 0x80; anv = next; } while (anv > 0); buf[sizeof(buf) - 1] &= 0x7f; /* clear continue bit */ @@ -2322,7 +2332,7 @@ Perl_pack_cat(pTHX_ SV *cat, char *pat, /* Copy string and check for compliance */ from = SvPV(fromstr, len); if ((norm = is_an_int(from, len)) == NULL) - Perl_croak(aTHX_ "can compress only unsigned integer"); + Perl_croak(aTHX_ "Can only compress unsigned integers"); New('w', result, len, char); in = result + len; --- t/op/pack.t.orig Sun Jun 30 03:25:03 2002 +++ t/op/pack.t Tue Oct 8 21:43:55 2002 @@ -6,7 +6,7 @@ BEGIN { require './test.pl'; } -plan tests => 5819; +plan tests => 5825; use strict; use warnings; @@ -170,6 +170,44 @@ sub list_eq ($$) { eval { $x = unpack 'w', pack 'C*', 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff}; like($@, qr/^Unterminated compressed integer/); + + eval { $x = pack 'w', -1 }; + like ($@, qr/^Cannot compress negative numbers/); + + eval { $x = pack 'w', '1'x(1 + length ~0) . 'e0' }; + like ($@, qr/^Can only compress unsigned integers/); + + SKIP: { + # Is this a stupid thing to do on VMS, VOS and other unusual platforms? + my $inf = eval '2**10000'; + + skip "Couldn't generate infinity - got error '$@'" + unless defined $inf and $inf == $inf / 2; + + eval { $x = pack 'w', $inf }; + like ($@, qr/^Cannot compress integer/); + } + + SKIP: { + # This should be about the biggest thing possible on an IEEE double + my $big = eval '2**1023'; + + skip "Couldn't generate 2**1023 - got error '$@'" + unless defined $big and $big != $big / 2; + + eval { $x = pack 'w', $big }; + is ($@, '', "Should be able to pack 'w', $big # 2**1023"); + + my $y = eval {unpack 'w', $x}; + is ($@, '', + "Should be able to unpack 'w' the result of pack 'w', $big # 2**1023"); + + # I'm getting about 1e-16 on FreeBSD + my $quotient = int (100 * ($y - $big) / $big); + ok($quotient < 2 && $quotient > -2, + "Round trip pack, unpack 'w' of $big is withing 1% ($quotient%)"); + } + } # ```

[p5pRT]

From @hvds

Nicholas Clark \nick@&#8203;unfortu\.net wrote​: :Appended patch makes it only possible to hit "cannot happen" (the third) for :compressing Inf\, documents all three in perldiag.pod\, and changes the : "can compress only unsigned integer" to : "Can only compress unsigned integers" :to be better English and capitalised consistently with all the other warnings :in perldiag.pod

Thanks\, applied as #18010.

:(sorry\, I'm getting speed obsessed [...])

No apologies needed. :)

Hugo

[p5pRT]

From @nwc10

On Sat\, Oct 12\, 2002 at 05​:41​:26PM +0100\, hv@​crypt.org wrote​:

Nicholas Clark \nick@&#8203;unfortu\.net wrote​: :Appended patch makes it only possible to hit "cannot happen" (the third) for :compressing Inf\, documents all three in perldiag.pod\, and changes the : "can compress only unsigned integer" to : "Can only compress unsigned integers" :to be better English and capitalised consistently with all the other warnings :in perldiag.pod

Thanks\, applied as #18010.

Benjamin Goldberg observed that I failed to pass the number of tests to skip at one point.

Nicholas Clark -- Brainfuck better than perl? http​://www.perl.org/advocacy/spoofathon/

Inline Patch

```diff --- t/op/pack.t.orig Sun Oct 20 16:07:56 2002 +++ t/op/pack.t Sun Oct 27 20:10:35 2002 @@ -192,7 +192,7 @@ sub list_eq ($$) { # This should be about the biggest thing possible on an IEEE double my $big = eval '2**1023'; - skip "Couldn't generate 2**1023 - got error '$@'" + skip "Couldn't generate 2**1023 - got error '$@'", 3 unless defined $big and $big != $big / 2; eval { $x = pack 'w', $big }; ```

[p5pRT]

From @rgs

Nicholas Clark wrote​:

--- t/op/pack.t.orig Sun Oct 20 16​:07​:56 2002 +++ t/op/pack.t Sun Oct 27 20​:10​:35 2002

Thanks\, applied as #18069.

@​@​ -192\,7 +192\,7 @​@​ sub list_eq ($$) { # This should be about the biggest thing possible on an IEEE double my $big = eval '2**1023';

- skip "Couldn't generate 2**1023 - got error '$@​'" + skip "Couldn't generate 2**1023 - got error '$@​'"\, 3 unless defined $big and $big != $big / 2;

 eval \{ $x = pack 'w'\, $big \};

[p5pRT]

From @jhi

Fixed\, patched\, resolved.

[p5pRT]

@jhi - Status changed from 'new' to 'resolved'