Open p5pRT opened 5 years ago
In the following versions of Perl:
This is perl\, v5.10.1 (*) built for x86_64-linux-thread-multi (CentOS 6) This is perl 5\, version 14\, subversion 2 (v5.14.2) built for i686-linux-gnu-thread-multi-64int (Ubuntu 12.04) This is perl 5\, version 16\, subversion 3 (v5.16.3) built for x86_64-linux-thread-multi (CentOS 7) This is perl 5\, version 26\, subversion 1 (v5.26.1) built for x86_64-linux-gnu-thread-multi (Ubuntu 18.04)
This code fragment:
------------------------
#!/usr/bin/perl use strict; use warnings FATAL => qw(all);
use JSON::PP; use Scalar::Util qw(reftype);
use constant VERSION => '9.1';
print("constant in JSON before comparison is " . JSON::PP->new()->allow_nonref()->encode(VERSION) . "\n");
if (VERSION > 9.0) { print("constant in JSON after comparison is " . JSON::PP->new()->allow_nonref()->encode(VERSION) . "\n"); }
my $version = '9.2';
print("var in JSON before comparison is " . JSON::PP->new()->allow_nonref()->encode($version) . "\n");
if ($version > 9.1) { print("var in JSON after comparison is " . JSON::PP->new()->allow_nonref()->encode($version) . "\n"); }
------------------------
Returns:
$ ./bug.pl constant in JSON before comparison is "9.1" constant in JSON after comparison is "9.1" var in JSON before comparison is "9.2" var in JSON after comparison is "9.2"
But in "This is perl 5\, version 28\, subversion 2 (v5.28.2) built for x86_64-linux-thread-multi" (Fedora 30) it returns:
$ ./bug.pl constant in JSON before comparison is 9.1 constant in JSON after comparison is 9.1 var in JSON before comparison is "9.2" var in JSON after comparison is 9.2
vagrant@pgbackrest-test:\~$ docker exec -it test-0 perl -v
This behavioral difference causes some interesting issues in our project. We have "fixed" them by appending '' when passing the var to JSON\, but hoping there is a more elegant fix and wondering if this could be a bug.
We have been unable to find a specific issue or release note that explains this behavior.
perl -V output below:
This is perl 5\, version 28\, subversion 2 (v5.28.2) built for x86_64-linux-thread-multi (with 48 registered patches\, see perl -V for more detail)
Copyright 1987-2019\, Larry Wall
Perl may be copied only under the terms of either the Artistic License or the GNU General Public License\, which may be found in the Perl 5 source kit.
Complete documentation for Perl\, including FAQ lists\, should be found on this system using "man perl" or "perldoc perl". If you have access to the Internet\, point your browser at http://www.perl.org/\, the Perl Home Page.
vagrant@pgbackrest-test:\~$ docker exec -it test-0 perl -V Summary of my perl5 (revision 5 version 28 subversion 2) configuration:
Platform: osname=linux osvers=5.0.6-200.fc29.x86_64 archname=x86_64-linux-thread-multi uname='linux buildvm-07.phx2.fedoraproject.org 5.0.6-200.fc29.x86_64 #1 smp wed apr 3 15:09:51 utc 2019 x86_64 x86_64 x86_64 gnulinux ' config_args='-des -Doptimize=none -Dccflags=-O2 -g -pipe -Wall -Werror=format-security -Wp\,-D_FORTIFY_SOURCE=2 -Wp\,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Dldflags=-Wl\,-z\,relro -Wl\,--as-needed -Wl\,-z\,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Dccdlflags=-Wl\,--enable-new-dtags -Wl\,-z\,relro -Wl\,--as-needed -Wl\,-z\,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Dlddlflags=-shared -Wl\,-z\,relro -Wl\,--as-needed -Wl\,-z\,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Dshrpdir=/usr/lib64 -DDEBUGGING=-g -Dversion=5.28.2 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat\, Inc. -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl5 -Dsitearch=/usr/local/lib64/perl5 -Dprivlib=/usr/share/perl5 -Dvendorlib=/usr/share/perl5/vendor_perl -Darchlib=/usr/lib64/perl5 -Dvendorarch=/usr/lib64/perl5/vendor_perl -Darchname=x86_64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Duseshrplib -Dusethreads -Duseithreads -Dusedtrace=/usr/bin/dtrace -Duselargefiles -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/usr/bin -Dusesitecustomize -Duse64bitint' hint=recommended useposix=true d_sigaction=define useithreads=define usemultiplicity=define use64bitint=define use64bitall=define uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='gcc' ccflags ='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp\,-D_FORTIFY_SOURCE=2 -Wp\,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fwrapv -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64' optimize=' -g' cppflags='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp\,-D_FORTIFY_SOURCE=2 -Wp\,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fwrapv -fno-strict-aliasing -I/usr/local/include' ccversion='' gccversion='9.0.1 20190312 (Red Hat 9.0.1-0.10)' gccosandvers='' intsize=4 longsize=8 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=3 ivtype='long' ivsize=8 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='gcc' ldflags ='-Wl\,-z\,relro -Wl\,--as-needed -Wl\,-z\,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -fstack-protector-strong -L/usr/local/lib' libpth=/usr/local/lib64 /lib64 /usr/lib64 /usr/local/lib /usr/lib /lib/../lib64 /usr/lib/../lib64 /lib libs=-lpthread -lresolv -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat perllibs=-lpthread -lresolv -ldl -lm -lcrypt -lutil -lc libc=libc-2.29.so so=so useshrplib=true libperl=libperl.so gnulibc_version='2.29' Dynamic Linking: dlsrc=dl_dlopen.xs dlext=so d_dlsymun=undef ccdlflags='-Wl\,--enable-new-dtags -Wl\,-z\,relro -Wl\,--as-needed -Wl\,-z\,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' cccdlflags='-fPIC' lddlflags='-lpthread -shared -Wl\,-z\,relro -Wl\,--as-needed -Wl\,-z\,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -L/usr/local/lib -fstack-protector-strong'
Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP PERL_OP_PARENT PERL_PRESERVE_IVUV USE_64_BIT_ALL USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API USE_SITECUSTOMIZE Locally applied patches: Fedora Patch1: Removes date check\, Fedora/RHEL specific Fedora Patch2: support for libdir64 Fedora Patch3: use libresolv instead of libbind Fedora Patch4: USE_MM_LD_RUN_PATH Fedora Patch5: Provide MM::maybe_command independently (bug #1129443) Fedora Patch6: Dont run one io test due to random builder failures Fedora Patch8: Define SONAME for libperl.so Fedora Patch9: Install libperl.so to -Dshrpdir value Fedora Patch10: Document Math::BigInt::CalcEmu requires Math::BigInt (CPAN RT#85015) Fedora Patch11: Make *DBM_File desctructors thread-safe (RT#61912) Fedora Patch12: Replace EU::MakeMaker dependency with EU::MM::Utils in IPC::Cmd (bug #1129443) Fedora Patch13: Fix executing arybase::_tie_it() in Safe compartement (RT#131588) Fedora Patch14: Link XS modules to pthread library to fix linking with -z defs Fedora Patch17: Fix printing a warning about a wide character when matching a regular expression while ISO-8859-1 locale is in effect Fedora Patch18: Fix invoking a check for wide characters while ISO-8859-1 locale is in effect Fedora Patch20: Fix build conditions in locale.c Fedora Patch21: Fix a file descriptor leak in in-place edits (RT#133314) Fedora Patch23: Fix a buffer overrun in deprecated S_is_utf8_common() Fedora Patch24: Fix a time race in Time-HiRes/t/itimer.t test Fedora Patch26: Fix Time::Piece to handle objects in overloaded methods correctly Fedora Patch27: Fix an assignment to a lexical variable in multiconcatenation expressions (RT#133441) Fedora Patch28: Fix a spurious warning about uninitialized value in warn (RT#132683) Fedora Patch30: Pass the correct CFLAGS to dtrace Fedora Patch33: Fix PathTools tests to cope with ESTALE error (RT#133534) Fedora Patch34: Fix an undefined behaviour in S_hv_delete_common() Fedora Patch38: Fix compiling regular expressions that contain both compile- and run-time compiled code blocks (RT#133687) Fedora Patch39: Adjust tests to gdbm-1.15 (RT#133295) Fedora Patch44: Fix reporting a line number for non-terminated prototypes (RT#133524) Fedora Patch45: Fix first eof() return value (RT#133721) Fedora Patch49: Prevent long jumps from clobbering local variables (RT#133575) Fedora Patch50: Fix a mismatch with a case-insesitive regular expression on a text with ligatures (RT#133756) Fedora Patch53: Fix setting magic when changing $^R (RT#133782) Fedora Patch54: Fix a race when loading XS modules Fedora Patch56: Fix a leak when compiling a typed hash dereference Fedora Patch58: Fix a buffer overread when handling a scope error in qr/\(?{/ (RT#133879) Fedora Patch59: Fix a buffer overread when parsing a regular expression with an unknown character name (RT#133880) Fedora Patch60: Fix mbstate_t initialization in POSIX::mblen (RT#133928) Fedora Patch61: Fix a memory leak when cloning a regular expression Fedora Patch62: Fix a memory leak when spawning threads in a BEGIN phase Fedora Patch63: Fix a memory leak when assigning a regular expression to a non-copy-on-write string Fedora Patch64: Fix a memory leak when assignig to a localized ${^WARNING_BITS} Fedora Patch65: Fix a memory leak when parsing misindented here-documents Fedora Patch66: Fix a memory leak in package name lookup (RT#133977) Fedora Patch68: Fix a memory leak when deletion in a tied hash dies Fedora Patch69: Fix a crash when matching case insensitively (RT#133892) Fedora Patch70: Fix a memory leak when warning about malformed UTF-8 string Fedora Patch200: Link XS modules to libperl.so with EU::CBuilder on Linux Fedora Patch201: Link XS modules to libperl.so with EU::MM on Linux Built under linux Compiled at Apr 23 2019 08:24:20 @INC: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5
On Wed\, Jul 03\, 2019 at 01:00:49PM -0700\, David Steele (via RT) wrote:
$ ./bug.pl constant in JSON before comparison is "9.1" constant in JSON after comparison is "9.1" var in JSON before comparison is "9.2" var in JSON after comparison is "9.2"
But in "This is perl 5\, version 28\, subversion 2 (v5.28.2) built for x86_64-linux-thread-multi" (Fedora 30) it returns:
$ ./bug.pl constant in JSON before comparison is 9.1 constant in JSON after comparison is 9.1 var in JSON before comparison is "9.2" var in JSON after comparison is 9.2
Its not clear in your report which behaviours you consider buggy in which versions of perl.
Are all the outputs in the first run above "correct"\, while any outputs in the second run which differ from the first are incorrect???
-- The optimist believes that he lives in the best of all possible worlds. As does the pessimist.
The RT System itself - Status changed from 'new' to 'open'
Hi Dave\,
On 7/4/19 3:36 AM\, Dave Mitchell via RT wrote:
On Wed\, Jul 03\, 2019 at 01:00:49PM -0700\, David Steele (via RT) wrote:
$ ./bug.pl constant in JSON before comparison is "9.1" constant in JSON after comparison is "9.1" var in JSON before comparison is "9.2" var in JSON after comparison is "9.2"
But in "This is perl 5\, version 28\, subversion 2 (v5.28.2) built for x86_64-linux-thread-multi" (Fedora 30) it returns:
$ ./bug.pl constant in JSON before comparison is 9.1 constant in JSON after comparison is 9.1 var in JSON before comparison is "9.2" var in JSON after comparison is 9.2
Its not clear in your report which behaviours you consider buggy in which versions of perl.
I consider the new behavior in 5.28 to be buggy -- or at least a significant behavioral change.
Are all the outputs in the first run above "correct"\, while any outputs in the second run which differ from the first are incorrect???
Yes -- the range of Perls (5.10 - 5.26) represent all the versions we currently test on and exhibit the first behavior. The behavior of 5.28 differs and has caused some breakage in our code base.
It appears strings are being converted to numbers after a comparison to a number\, at least from the perspective of JSON. This seems odd at the least and certainly represents a behavior change from the last eleven years of Perl versions.
Regards\, -- -David david@pgbackrest.org
On Thu\, 04 Jul 2019 09:33:40 -0700\, david@pgbackrest.org wrote:
Hi Dave\,
On 7/4/19 3:36 AM\, Dave Mitchell via RT wrote:
On Wed\, Jul 03\, 2019 at 01:00:49PM -0700\, David Steele (via RT) wrote:
$ ./bug.pl constant in JSON before comparison is "9.1" constant in JSON after comparison is "9.1" var in JSON before comparison is "9.2" var in JSON after comparison is "9.2"
But in "This is perl 5\, version 28\, subversion 2 (v5.28.2) built for x86_64-linux-thread-multi" (Fedora 30) it returns:
$ ./bug.pl constant in JSON before comparison is 9.1 constant in JSON after comparison is 9.1 var in JSON before comparison is "9.2" var in JSON after comparison is 9.2
Its not clear in your report which behaviours you consider buggy in which versions of perl.
I consider the new behavior in 5.28 to be buggy -- or at least a significant behavioral change.
Are all the outputs in the first run above "correct"\, while any outputs in the second run which differ from the first are incorrect???
Yes -- the range of Perls (5.10 - 5.26) represent all the versions we currently test on and exhibit the first behavior. The behavior of 5.28 differs and has caused some breakage in our code base.
It appears strings are being converted to numbers after a comparison to a number\, at least from the perspective of JSON. This seems odd at the least and certainly represents a behavior change from the last eleven years of Perl versions.
Regards\,
JSON::PP is maintained on CPAN so that is probably a better place to discuss this. This was an intentional change in JSON::PP 2.92 to detect numbers better in the usual case\, but of course with the lack of typing in Perl\, there is no foolproof solution\, these are all heuristics. The benefit of the new detection heuristic is the opposite direction: numbers don't suddenly become strings because you interpolated them in a string or printed them.
* https://metacpan.org/source/ISHIGAKI/JSON-PP-4.04/Changes#L70-72
-Dan
On Thu\, 04 Jul 2019 10:00:12 -0700\, grinnz@gmail.com wrote:
JSON::PP is maintained on CPAN so that is probably a better place to discuss this. This was an intentional change in JSON::PP 2.92 to detect numbers better in the usual case\, but of course with the lack of typing in Perl\, there is no foolproof solution\, these are all heuristics. The benefit of the new detection heuristic is the opposite direction: numbers don't suddenly become strings because you interpolated them in a string or printed them.
* https://metacpan.org/source/ISHIGAKI/JSON-PP-4.04/Changes#L70-72
As an addendum\, you might consider Cpanel::JSON::XS::Type as a way to declare types for your JSON output and avoid relying on heuristics if it's important.
-Dan
Hi Dan\,
On 7/4/19 1:00 PM\, Dan Book via RT wrote:
On Thu\, 04 Jul 2019 09:33:40 -0700\, david@pgbackrest.org wrote:
It appears strings are being converted to numbers after a comparison to a number\, at least from the perspective of JSON. This seems odd at the least and certainly represents a behavior change from the last eleven years of Perl versions.
Regards\,
JSON::PP is maintained on CPAN so that is probably a better place to discuss this. This was an intentional change in JSON::PP 2.92 to detect numbers better in the usual case\, but of course with the lack of typing in Perl\, there is no foolproof solution\, these are all heuristics. The benefit of the new detection heuristic is the opposite direction: numbers don't suddenly become strings because you interpolated them in a string or printed them.
Hmm\, OK. Still seems odd that the output varies based on whether a numeric comparison has been performed or not\, but I have installed JSON::PP 2.97 on Perl 5.26 and confirmed that the behavior is related to the JSON:PP version.
This drives our unit tests nuts because we have always treated version numbers as strings\, but now they may be output as strings or numbers depending on which code paths are followed in the particular test.
Thanks for pointing me in the right direction.
Regards\, -- -David david@pgbackrest.org
Hi Dan\,
On 7/4/19 1:01 PM\, Dan Book via RT wrote:
On Thu\, 04 Jul 2019 10:00:12 -0700\, grinnz@gmail.com wrote:
JSON::PP is maintained on CPAN so that is probably a better place to discuss this. This was an intentional change in JSON::PP 2.92 to detect numbers better in the usual case\, but of course with the lack of typing in Perl\, there is no foolproof solution\, these are all heuristics. The benefit of the new detection heuristic is the opposite direction: numbers don't suddenly become strings because you interpolated them in a string or printed them.
* https://metacpan.org/source/ISHIGAKI/JSON-PP-4.04/Changes#L70-72
As an addendum\, you might consider Cpanel::JSON::XS::Type as a way to declare types for your JSON output and avoid relying on heuristics if it's important.
Thanks for the tip!
We're currently migrating off Perl so I don't want to introduce a new module at this point\, even assuming it is packaged for all the platforms we support.
Appending '' seems to be working fine in testing so we'll go with that until the migration is complete in a few months.
Regards\, -- -David david@pgbackrest.org
We have completed our migration off of Perl so we are no longer affected by this issue. Feel free to close it.
Migrated from rt.perl.org#134261 (status was 'open')
Searchable as RT134261$