search

Perl / perl5

🐪 The Perl programming language
https://dev.perl.org/perl5/
Other
1.98k stars 558 forks source link

panic: reg_node overrun in unusual regular expression #14891

Closed p5pRT closed 8 years ago

p5pRT commented 9 years ago

Migrated from rt.perl.org#125990 (status was 'resolved')

Searchable as RT125990$

p5pRT commented 9 years ago

From @dcollinsn

Greetings Porters\,

I have compiled bleadperl with the afl-gcc compiler using​:

./Configure -Dusedevel -Dprefix='/usr/local/perl-afl' -Dcc=afl-gcc -Duselongdouble -Duse64bitint -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -Dusethreads -des AFL_HARDEN=1 make && make test

And then fuzzed the resulting binary using​:

AFL_NO_VAR_CHECK=1 afl-fuzz -i in -o out bin/perl @​@​

After reducing testcases using `afl-tmin` and filtering out testcases that are merely iterations of "#!perl -u"\, I have located the following testcase that triggers a panic in the perl interpreter. The testcase is the 62-byte binary file​: (octal bytes)

root@​nagios​:/usr/local/perl-afl/out# od -c allcrash/f2i000000 0000000 / \ N { } 0 0 0 0 0 0 0 0 337 337 337 0000020 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0000040 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0000060 0 0 0 0 0 0 0 0 0 0 375 / i \n 0000076

When executed\, the following bad things happen​:

panic​: reg_node overrun trying to emit 0\, 8833280>=8833280 at test line 1.

The program exits cleanly\, apart from the panic\, so there is no backtrace for GDB to produce\, and Valgrind finds no errors. The next bug I am about to submit is an even more esoteric version of this case which fails somewhat more spectacularly.

**PERL -V**

Summary of my perl5 (revision 5 version 23 subversion 3) configuration​:   Commit id​: 9ae0115f0b854d654461d3c5bbcaa938516d0f4e   Platform​:   osname=linux\, osvers=2.6.32-5-686\, archname=i686-linux-64int-ld-thread-multi   uname='linux nagios 2.6.32-5-686 #1 smp tue may 13 16​:33​:32 utc 2014 i686 gnulinux '   config_args='-Dusedevel -Dprefix=/usr/local/perl-afl -Dcc=afl-gcc -Duselongdouble -Duse64bitint -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -des'   hint=previous\, useposix=true\, d_sigaction=define   useithreads=define\, usemultiplicity=define   use64bitint=define\, use64bitall=undef\, uselongdouble=define   usemymalloc=n\, bincompat5005=undef   Compiler​:   cc='afl-gcc'\, ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'\,   optimize='-g'\,   cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'   ccversion=''\, gccversion='4.4.5'\, gccosandvers=''   intsize=4\, longsize=4\, ptrsize=4\, doublesize=8\, byteorder=12345678\, doublekind=3   d_longlong=define\, longlongsize=8\, d_longdbl=define\, longdblsize=12\, longdblkind=3   ivtype='long long'\, ivsize=8\, nvtype='long double'\, nvsize=12\, Off_t='off_t'\, lseeksize=8   alignbytes=4\, prototype=define   Linker and Libraries​:   ld='afl-gcc'\, ldflags =' -fstack-protector -L/usr/local/lib'   libpth=/usr/local/lib /usr/lib/gcc/i486-linux-gnu/4.4.5/include-fixed /usr/lib /lib/../lib /usr/lib/../lib /lib /usr/lib/i486-linux-gnu /usr/lib64 /usr/local/lib /usr/lib/gcc/i486-linux-gnu/4.4.5/include-fixed /usr/lib /usr/local/lib /usr/lib/gcc/i486-linux-gnu/4.4.5/include-fixed /usr/lib /usr/local/lib /usr/lib/gcc/i486-linux-gnu/4.4.5/include-fixed /usr/lib   libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc   perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc   libc=libc-2.11.3.so\, so=so\, useshrplib=false\, libperl=libperl.a   gnulibc_version='2.11.3'   Dynamic Linking​:   dlsrc=dl_dlopen.xs\, dlext=so\, d_dlsymun=undef\, ccdlflags='-Wl\,-E'   cccdlflags='-fPIC'\, lddlflags='-shared -g -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl)​:   Compile-time options​: HAS_TIMES MULTIPLICITY PERLIO_LAYERS   PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV   PERL_HASH_FUNC_ONE_AT_A_TIME_HARD   PERL_IMPLICIT_CONTEXT PERL_PRESERVE_IVUV   PERL_USE_DEVEL USE_64_BIT_INT USE_ITHREADS   USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE   USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME   USE_LONG_DOUBLE USE_PERLIO USE_PERL_ATOF   USE_REENTRANT_API   Built under linux   Compiled at Sep 2 2015 09​:16​:23   @​INC​:   /usr/local/perl-afl/lib/site_perl/5.23.3/i686-linux-64int-ld   /usr/local/perl-afl/lib/site_perl/5.23.3   /usr/local/perl-afl/lib/5.23.3/i686-linux-64int-ld   /usr/local/perl-afl/lib/5.23.3   /usr/local/perl-afl/lib/site_perl/5.23.2   /usr/local/perl-afl/lib/site_perl   .

p5pRT commented 9 years ago

From @dcollinsn

Greetings Porters\,

I have compiled bleadperl with the afl-gcc compiler using​:

./Configure -Dusedevel -Dprefix='/usr/local/perl-afl' -Dcc=afl-gcc -Duselongdouble -Duse64bitint -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -Dusethreads -des AFL_HARDEN=1 make && make test

And then fuzzed the resulting binary using​:

AFL_NO_VAR_CHECK=1 afl-fuzz -i in -o out bin/perl @​@​

After reducing testcases using `afl-tmin` and filtering out testcases that are merely iterations of "#!perl -u"\, I have located the following testcase that triggers a panic in the perl interpreter. The testcase is the 74-byte binary file​: (octal bytes)

root@​nagios​:/usr/local/perl-afl/out# od -c allcrash/f2i000000 0000000 / 0 0 0 0 0 0 0 0 0 \ N { } 0 0 0000020 0 0 0 0 0 337 337 0 337 337 337 337 337 0 0 0 0000040 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0000060 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0000100 0 0 0 0 0 0 0 375 / i 0000112

Attached as "f2i000000".

I'm reasonably certain that this is a duplicate of another bug\, because I've seen the below sort of backtrace in a different bug\, that also related to a "creative" regular expression. However\, I can't find that bug. I'm also not sure if this is an upstream\, because Perl is trying to croak here\, and does so successfully under very similar circumstances.

When executed\, the following bad things happen​:

root@​nagios​:/usr/local/perl-afl/out# ../bin/perl allcrash/f2i000000 *** glibc detected *** ../bin/perl​: free()​: invalid next size (normal)​: 0x0a3a2260 *** ======= Backtrace​: ========= /lib/i686/cmov/libc.so.6(+0x6af71)[0xb7619f71] /lib/i686/cmov/libc.so.6(+0x6c7c8)[0xb761b7c8] /lib/i686/cmov/libc.so.6(+0x703ba)[0xb761f3ba] /lib/i686/cmov/libc.so.6(realloc+0xdd)[0xb761f97d] ../bin/perl(Perl_safesysrealloc+0xc8)[0x82d4ff8] ../bin/perl(Perl_sv_grow+0x4c4)[0x83dc5b4] ../bin/perl(Perl_sv_catpvn_flags+0x306)[0x83fbd76] ../bin/perl(Perl_sv_vcatpvfn_flags+0x4fea)[0x83ba5ea] ../bin/perl(Perl_sv_vsetpvfn+0xcb)[0x84047fb] ../bin/perl(Perl_vmess+0x13f)[0x82e35af] ../bin/perl(Perl_vcroak+0x98)[0x82ddf58] ../bin/perl(Perl_parse_unicode_opts+0x0)[0x82de1f0] ../bin/perl[0x82076df] ../bin/perl[0x826295c] ../bin/perl(Perl_re_op_compile+0x306a)[0x829873a] ../bin/perl(Perl_pmruntime+0x1380)[0x80d7580] ../bin/perl(Perl_yyparse+0x4593)[0x81d88b3] ../bin/perl[0x811467f] ../bin/perl(perl_parse+0x189f)[0x8117b1f] ../bin/perl(main+0x119)[0x80629d9] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb75c5ca6] ../bin/perl[0x8062821] ======= Memory map​: ======== 08048000-0875c000 r-xp 00000000 08​:01 734611 /usr/local/perl-afl/bin/perl 0875c000-0875e000 rw-p 00713000 08​:01 734611 /usr/local/perl-afl/bin/perl 0a387000-0a3a8000 rw-p 00000000 00​:00 0 [heap] b7300000-b7321000 rw-p 00000000 00​:00 0 b7321000-b7400000 ---p 00000000 00​:00 0 b741a000-b7437000 r-xp 00000000 08​:01 432483 /lib/libgcc_s.so.1 b7437000-b7438000 rw-p 0001c000 08​:01 432483 /lib/libgcc_s.so.1 b7438000-b75ad000 r--p 00000000 08​:01 99452 /usr/lib/locale/locale-archive b75ad000-b75af000 rw-p 00000000 00​:00 0 b75af000-b76ef000 r-xp 00000000 08​:01 449158 /lib/i686/cmov/libc-2.11.3.so b76ef000-b76f0000 ---p 00140000 08​:01 449158 /lib/i686/cmov/libc-2.11.3.so b76f0000-b76f2000 r--p 00140000 08​:01 449158 /lib/i686/cmov/libc-2.11.3.so b76f2000-b76f3000 rw-p 00142000 08​:01 449158 /lib/i686/cmov/libc-2.11.3.so b76f3000-b76f6000 rw-p 00000000 00​:00 0 b76f6000-b76f8000 r-xp 00000000 08​:01 449149 /lib/i686/cmov/libutil-2.11.3.so b76f8000-b76f9000 r--p 00001000 08​:01 449149 /lib/i686/cmov/libutil-2.11.3.so b76f9000-b76fa000 rw-p 00002000 08​:01 449149 /lib/i686/cmov/libutil-2.11.3.so b76fa000-b7703000 r-xp 00000000 08​:01 449143 /lib/i686/cmov/libcrypt-2.11.3.so b7703000-b7704000 r--p 00008000 08​:01 449143 /lib/i686/cmov/libcrypt-2.11.3.so b7704000-b7705000 rw-p 00009000 08​:01 449143 /lib/i686/cmov/libcrypt-2.11.3.so b7705000-b772c000 rw-p 00000000 00​:00 0 b772c000-b7750000 r-xp 00000000 08​:01 449140 /lib/i686/cmov/libm-2.11.3.so b7750000-b7751000 r--p 00023000 08​:01 449140 /lib/i686/cmov/libm-2.11.3.so b7751000-b7752000 rw-p 00024000 08​:01 449140 /lib/i686/cmov/libm-2.11.3.so b7752000-b7754000 r-xp 00000000 08​:01 449139 /lib/i686/cmov/libdl-2.11.3.so b7754000-b7755000 r--p 00001000 08​:01 449139 /lib/i686/cmov/libdl-2.11.3.so b7755000-b7756000 rw-p 00002000 08​:01 449139 /lib/i686/cmov/libdl-2.11.3.so b7756000-b7757000 rw-p 00000000 00​:00 0 b7757000-b776a000 r-xp 00000000 08​:01 449142 /lib/i686/cmov/libnsl-2.11.3.so b776a000-b776b000 r--p 00012000 08​:01 449142 /lib/i686/cmov/libnsl-2.11.3.so b776b000-b776c000 rw-p 00013000 08​:01 449142 /lib/i686/cmov/libnsl-2.11.3.so b776c000-b776e000 rw-p 00000000 00​:00 0 b776e000-b7783000 r-xp 00000000 08​:01 449148 /lib/i686/cmov/libpthread-2.11.3.so b7783000-b7784000 r--p 00014000 08​:01 449148 /lib/i686/cmov/libpthread-2.11.3.so b7784000-b7785000 rw-p 00015000 08​:01 449148 /lib/i686/cmov/libpthread-2.11.3.so b7785000-b7787000 rw-p 00000000 00​:00 0 b7797000-b7799000 rw-p 00000000 00​:00 0 b7799000-b779a000 r-xp 00000000 00​:00 0 [vdso] b779a000-b77b5000 r-xp 00000000 08​:01 432496 /lib/ld-2.11.3.so b77b5000-b77b6000 r--p 0001b000 08​:01 432496 /lib/ld-2.11.3.so b77b6000-b77b7000 rw-p 0001c000 08​:01 432496 /lib/ld-2.11.3.so bf8b7000-bf8cc000 rw-p 00000000 00​:00 0 [stack] Aborted

When Perl is instead configured with PERL_MALLOC_WRAP and DEBUGGING\, or when the testcase is shortened by removing even one zero\, an error such as the following appears\, which I've reported as a separate bug\, [perl #125990]

panic​: reg_node overrun trying to emit 0\, a18b1fc>=a18b1f8 at test line 1.

**GDB**

(the above output\, followed by...) Program received signal SIGABRT\, Aborted. 0xb7fe2424 in __kernel_vsyscall () (gdb) bt #0 0xb7fe2424 in __kernel_vsyscall () #1 0xb7e22781 in *__GI_raise (sig=6)   at ../nptl/sysdeps/unix/sysv/linux/raise.c​:64 #2 0xb7e25bb2 in *__GI_abort () at abort.c​:92 #3 0xb7e58e75 in __libc_message (do_abort=2\,   fmt=0xb7f1dbd8 "*** glibc detected *** %s​: %s​: 0x%s ***\n")   at ../sysdeps/unix/sysv/linux/libc_fatal.c​:189 #4 0xb7e62f71 in malloc_printerr (action=\\,   str=0x6 \<Address 0x6 out of bounds>\, ptr=0x87792a0) at malloc.c​:6267 #5 0xb7e647c8 in _int_free (av=\\, p=\)   at malloc.c​:4795 #6 0xb7e683ba in _int_realloc (av=0x3464\, oldp=0x8779268\, oldsize=16\, nb=48)   at malloc.c​:5353 #7 0xb7e6897d in *__GI___libc_realloc (oldmem=0x8779270\, bytes=44)   at malloc.c​:3822 #8 0x082d4ff8 in Perl_safesysrealloc (where=0x6\, size=0) at util.c​:270 #9 0x083dc5b4 in Perl_sv_grow (my_perl=0x875e008\, sv=0x8771cd8\, newlen=44)   at sv.c​:1621 #10 0x083fbd76 in Perl_sv_catpvn_flags (my_perl=0x875e008\, dsv=0x8771cd8\,   sstr=0x873cd0c "panic​: reg_node overrun trying to emit %d\, %p>=%p"\,   slen=39\, flags=0) at sv.c​:5366 #11 0x083ba5ea in Perl_sv_vcatpvfn_flags (my_perl=0x875e008\, sv=0x8771cd8\,   pat=0x873cd0c "panic​: reg_node overrun trying to emit %d\, %p>=%p"\,   patlen=49\, args=0xbfffeca8\, svargs=0x0\, svmax=0\, maybe_tainted=0x0\, ---Type \ to continue\, or q \ to quit---   flags=\) at sv.c​:11245 #12 0x084047fb in Perl_sv_vsetpvfn (my_perl=0x875e008\, sv=0x8771cd8\,   pat=0x873cd0c "panic​: reg_node overrun trying to emit %d\, %p>=%p"\,   patlen=49\, args=0xbfffeca8\, svargs=0x0\, svmax=0\, maybe_tainted=0x0)   at sv.c​:10580 #13 0x082e35af in Perl_vmess (my_perl=0x875e008\,   pat=0x873cd0c "panic​: reg_node overrun trying to emit %d\, %p>=%p"\,   args=0xbfffeca8) at util.c​:1466 #14 0x082ddf58 in Perl_vcroak (my_perl=0x875e008\, pat=0x0\, args=0x6)   at util.c​:1695 #15 0x082de1f0 in Perl_croak (my_perl=0x875e008\,   pat=0x873cd0c "panic​: reg_node overrun trying to emit %d\, %p>=%p")   at util.c​:1742 #16 0x082076df in S_regnode_guts (my_perl=\\,   pRExC_state=\\, op=100 'd') at regcomp.c​:16400 #17 S_reg_node (my_perl=\\,   pRExC_state=\\, op=100 'd') at regcomp.c​:16429 #18 0x0826295c in S_reg (my_perl=\\,   pRExC_state=0xbfffeff4\, paren=\\, flagp=0xbfffeefc\,   depth=1) at regcomp.c​:10596 #19 0x0829873a in Perl_re_op_compile (my_perl=0x875e008\, patternp=0x0\,   pat_count=0\, expr=0x877919c\, eng=0x873f960\, old_re=0x0\, is_bare_re=0x0\,   orig_rx_flags=4\, pm_flags=4) at regcomp.c​:6902 #20 0x080d7580 in Perl_pmruntime (my_perl=0x875e008\, o=0x87791bc\, ---Type \ to continue\, or q \ to quit---   expr=0x877919c\, repl=0x0\, isreg=true\, floor=0) at op.c​:5579 #21 0x081d88b3 in Perl_yyparse (my_perl=0x875e008\, gramtype=258)   at perly.y​:1038 #22 0x0811467f in S_parse_body (my_perl=0x875e008\, env=\\,   xsinit=\) at perl.c​:2296 #23 0x08117b1f in perl_parse (my_perl=0x875e008\, xsinit=0x8062c90 \<xs_init>\,   argc=2\, argv=0xbffff4d4\, env=0x0) at perl.c​:1626 #24 0x080629d9 in main (argc=2\, argv=0xbffff4d4\, env=0xbffff4e0)   at perlmain.c​:114 (gdb)

**VALGRIND**

Valgrind seems to cause different behavior in this testcase\, allowing Perl to panic rather than throw the glibc error above. Not sure how to get around that.

==9005== Memcheck\, a memory error detector ==9005== Copyright (C) 2002-2010\, and GNU GPL'd\, by Julian Seward et al. ==9005== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for copyright info ==9005== Command​: ../bin/perl allcrash/f2i000000 ==9005== ==9005== Invalid write of size 1 ==9005== at 0x8289A46​: S_regpiece (regcomp.c​:12725) ==9005== by 0x8294A7E​: S_regbranch (regcomp.c​:10754) ==9005== by 0x825DFB0​: S_reg (regcomp.c​:10504) ==9005== by 0x8298739​: Perl_re_op_compile (regcomp.c​:6902) ==9005== by 0x80D757F​: Perl_pmruntime (op.c​:5579) ==9005== by 0x81D88B2​: Perl_yyparse (perly.y​:1038) ==9005== by 0x811467E​: S_parse_body (perl.c​:2296) ==9005== by 0x8117B1E​: perl_parse (perl.c​:1626) ==9005== by 0x80629D8​: main (perlmain.c​:114) ==9005== Address 0x4237164 is 0 bytes after a block of size 116 alloc'd ==9005== at 0x4023F50​: malloc (vg_replace_malloc.c​:236) ==9005== by 0x82D4537​: Perl_safesysmalloc (util.c​:149) ==9005== by 0x8297CC1​: Perl_re_op_compile (regcomp.c​:6752) ==9005== by 0x80D757F​: Perl_pmruntime (op.c​:5579) ==9005== by 0x81D88B2​: Perl_yyparse (perly.y​:1038) ==9005== by 0x811467E​: S_parse_body (perl.c​:2296) ==9005== by 0x8117B1E​: perl_parse (perl.c​:1626) ==9005== by 0x80629D8​: main (perlmain.c​:114) ==9005== panic​: reg_node overrun trying to emit 0\, 4237168>=4237164 at allcrash/f2i000000 line 1. ==9005== ==9005== HEAP SUMMARY​: ==9005== in use at exit​: 89\,869 bytes in 582 blocks ==9005== total heap usage​: 757 allocs\, 175 frees\, 122\,964 bytes allocated ==9005== ==9005== LEAK SUMMARY​: ==9005== definitely lost​: 6\,243 bytes in 15 blocks ==9005== indirectly lost​: 83\,626 bytes in 567 blocks ==9005== possibly lost​: 0 bytes in 0 blocks ==9005== still reachable​: 0 bytes in 0 blocks ==9005== suppressed​: 0 bytes in 0 blocks ==9005== Rerun with --leak-check=full to see details of leaked memory ==9005== ==9005== For counts of detected and suppressed errors\, rerun with​: -v ==9005== ERROR SUMMARY​: 1 errors from 1 contexts (suppressed​: 25 from 8)

**BISECT**

First commit that crashes in this particular way is​: commit 75697d6e4ef98ece405210de48e7529d01b619bf Author​: Karl Williamson \khw@&#8203;cpan\.org Date​: Sun Dec 21 22​:02​:30 2014 -0700

  Empty \N{} in regex pattern should force /d to /u  
  \N{} is for Unicode names\, even if the name is actually omitted.   (Accepting an empty name is\, I believe\, an accident\, and now is   supported only for backwards compatibility.)

**PERL -V**

Summary of my perl5 (revision 5 version 23 subversion 3) configuration​:   Commit id​: 9ae0115f0b854d654461d3c5bbcaa938516d0f4e   Platform​:   osname=linux\, osvers=2.6.32-5-686\, archname=i686-linux-64int-ld-thread-multi   uname='linux nagios 2.6.32-5-686 #1 smp tue may 13 16​:33​:32 utc 2014 i686 gnulinux '   config_args='-Dusedevel -Dprefix=/usr/local/perl-afl -Dcc=afl-gcc -Duselongdouble -Duse64bitint -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -des'   hint=previous\, useposix=true\, d_sigaction=define   useithreads=define\, usemultiplicity=define   use64bitint=define\, use64bitall=undef\, uselongdouble=define   usemymalloc=n\, bincompat5005=undef   Compiler​:   cc='afl-gcc'\, ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'\,   optimize='-g'\,   cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'   ccversion=''\, gccversion='4.4.5'\, gccosandvers=''   intsize=4\, longsize=4\, ptrsize=4\, doublesize=8\, byteorder=12345678\, doublekind=3   d_longlong=define\, longlongsize=8\, d_longdbl=define\, longdblsize=12\, longdblkind=3   ivtype='long long'\, ivsize=8\, nvtype='long double'\, nvsize=12\, Off_t='off_t'\, lseeksize=8   alignbytes=4\, prototype=define   Linker and Libraries​:   ld='afl-gcc'\, ldflags =' -fstack-protector -L/usr/local/lib'   libpth=/usr/local/lib /usr/lib/gcc/i486-linux-gnu/4.4.5/include-fixed /usr/lib /lib/../lib /usr/lib/../lib /lib /usr/lib/i486-linux-gnu /usr/lib64 /usr/local/lib /usr/lib/gcc/i486-linux-gnu/4.4.5/include-fixed /usr/lib /usr/local/lib /usr/lib/gcc/i486-linux-gnu/4.4.5/include-fixed /usr/lib /usr/local/lib /usr/lib/gcc/i486-linux-gnu/4.4.5/include-fixed /usr/lib   libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc   perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc   libc=libc-2.11.3.so\, so=so\, useshrplib=false\, libperl=libperl.a   gnulibc_version='2.11.3'   Dynamic Linking​:   dlsrc=dl_dlopen.xs\, dlext=so\, d_dlsymun=undef\, ccdlflags='-Wl\,-E'   cccdlflags='-fPIC'\, lddlflags='-shared -g -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl)​:   Compile-time options​: HAS_TIMES MULTIPLICITY PERLIO_LAYERS   PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV   PERL_HASH_FUNC_ONE_AT_A_TIME_HARD   PERL_IMPLICIT_CONTEXT PERL_PRESERVE_IVUV   PERL_USE_DEVEL USE_64_BIT_INT USE_ITHREADS   USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE   USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME   USE_LONG_DOUBLE USE_PERLIO USE_PERL_ATOF   USE_REENTRANT_API   Built under linux   Compiled at Sep 2 2015 09​:16​:23   @​INC​:   /usr/local/perl-afl/lib/site_perl/5.23.3/i686-linux-64int-ld   /usr/local/perl-afl/lib/site_perl/5.23.3   /usr/local/perl-afl/lib/5.23.3/i686-linux-64int-ld   /usr/local/perl-afl/lib/5.23.3   /usr/local/perl-afl/lib/site_perl/5.23.2   /usr/local/perl-afl/lib/site_perl   .

p5pRT commented 9 years ago

From @dcollinsn

f2i000000

p5pRT commented 9 years ago

From @tonycoz

On Fri Sep 04 17​:27​:40 2015\, dcollinsn@​gmail.com wrote​:

root@​nagios​:/usr/local/perl-afl/out# od -c allcrash/f2i000000 0000000 / \ N { } 0 0 0 0 0 0 0 0 337 337 337 0000020 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0000040 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0000060 0 0 0 0 0 0 0 0 0 0 375 / i \n 0000076

Could you attach this file please?

Thanks\, Tony

p5pRT commented 9 years ago

The RT System itself - Status changed from 'new' to 'open'

p5pRT commented 9 years ago

From @tonycoz

On Sun Sep 06 21​:20​:19 2015\, tonyc wrote​:

On Fri Sep 04 17​:27​:40 2015\, dcollinsn@​gmail.com wrote​:

root@​nagios​:/usr/local/perl-afl/out# od -c allcrash/f2i000000 0000000 / \ N { } 0 0 0 0 0 0 0 0 337 337 337 0000020 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0000040 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0000060 0 0 0 0 0 0 0 0 0 0 375 / i \n 0000076

Could you attach this file please?

I ended up generating it\, attached as 125990.pl.

The problem is len at line 12988 is different between pass 1 and pass 2\, with values of 54 and 57.

This leads to a simplified reproducer​:

tony@​mars​:.../git/perl$ od -c ../125990b.pl 0000000 / \ N { } 337 337 337 375 / i \n 0000014

attached as 125990b.pl\, which should be simpler to debug to find where the mismatch occurs (which I'll do tomorrow if someone else doesn't beat me to it.)

Tony

p5pRT commented 9 years ago

From @tonycoz

125990.pl

p5pRT commented 9 years ago

From @tonycoz

125990b.pl

p5pRT commented 9 years ago

From @tonycoz

On Mon Sep 07 23​:57​:17 2015\, tonyc wrote​:

On Sun Sep 06 21​:20​:19 2015\, tonyc wrote​:

On Fri Sep 04 17​:27​:40 2015\, dcollinsn@​gmail.com wrote​:

root@​nagios​:/usr/local/perl-afl/out# od -c allcrash/f2i000000 0000000 / \ N { } 0 0 0 0 0 0 0 0 337 337 337 0000020 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0000040 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0000060 0 0 0 0 0 0 0 0 0 0 375 / i \n 0000076

Could you attach this file please?

I ended up generating it\, attached as 125990.pl.

RT seems to have attached the same content for both files. Trying 125990.pl again.

Tony

p5pRT commented 9 years ago

From @tonycoz

125990.pl

p5pRT commented 9 years ago

From @khwilliamson

This is likely from my code\, so I'm taking the ticket to look at it.

On 09/08/2015 12​:58 AM\, Tony Cook via RT wrote​:

On Mon Sep 07 23​:57​:17 2015\, tonyc wrote​:

On Sun Sep 06 21​:20​:19 2015\, tonyc wrote​:

On Fri Sep 04 17​:27​:40 2015\, dcollinsn@​gmail.com wrote​:

root@​nagios​:/usr/local/perl-afl/out# od -c allcrash/f2i000000 0000000 / \ N { } 0 0 0 0 0 0 0 0 337 337 337 0000020 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0000040 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0000060 0 0 0 0 0 0 0 0 0 0 375 / i \n 0000076

Could you attach this file please?

I ended up generating it\, attached as 125990.pl.

RT seems to have attached the same content for both files. Trying 125990.pl again.

Tony

--- via perlbug​: queue​: perl5 status​: open https://rt-archive.perl.org/perl5/Ticket/Display.html?id=125990

p5pRT commented 9 years ago

From @tonycoz

On Fri Sep 04 17​:28​:18 2015\, dcollinsn@​gmail.com wrote​:

Greetings Porters\,

I have compiled bleadperl with the afl-gcc compiler using​:

./Configure -Dusedevel -Dprefix='/usr/local/perl-afl' -Dcc=afl-gcc -Duselongdouble -Duse64bitint -Doptimize=-g -Uversiononly -Uman1dir -Uman3dir -Dusethreads -des AFL_HARDEN=1 make && make test

And then fuzzed the resulting binary using​:

AFL_NO_VAR_CHECK=1 afl-fuzz -i in -o out bin/perl @​@​

After reducing testcases using `afl-tmin` and filtering out testcases that are merely iterations of "#!perl -u"\, I have located the following testcase that triggers a panic in the perl interpreter. The testcase is the 74-byte binary file​: (octal bytes)

root@​nagios​:/usr/local/perl-afl/out# od -c allcrash/f2i000000 0000000 / 0 0 0 0 0 0 0 0 0 \ N { } 0 0 0000020 0 0 0 0 0 337 337 0 337 337 337 337 337 0 0 0 0000040 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0000060 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0000100 0 0 0 0 0 0 0 375 / i 0000112

I strongly suspect this is the same bug as 125990.

The base case of 125990 has enough ß (ss) characters for the generated string to take up the space for one extra regnode over the calculated length\, the case in this ticket overflows even further.

Your case doesn't produce an error (even with valgrind) on my 64-bit system\, but a modified (and shorter version) can be made to​:

tony@​mars​:.../git/perl$ od -c ../125991b.pl 0000000 / \ N { } 337 337 337 337 337 337 337 337 337 337 337 0000020 375 / i 0000023

though it's only errors for me with valgrind.

I'm reasonably certain that this is a duplicate of another bug\, because I've seen the below sort of backtrace in a different bug\, that also related to a "creative" regular expression. However\, I can't find that bug. I'm also not sure if this is an upstream\, because Perl is trying to croak here\, and does so successfully under very similar circumstances.

When executed\, the following bad things happen​:

root@​nagios​:/usr/local/perl-afl/out# ../bin/perl allcrash/f2i000000 *** glibc detected *** ../bin/perl​: free()​: invalid next size (normal)​: 0x0a3a2260 *** ======= Backtrace​: ========= /lib/i686/cmov/libc.so.6(+0x6af71)[0xb7619f71] /lib/i686/cmov/libc.so.6(+0x6c7c8)[0xb761b7c8] /lib/i686/cmov/libc.so.6(+0x703ba)[0xb761f3ba] /lib/i686/cmov/libc.so.6(realloc+0xdd)[0xb761f97d] ../bin/perl(Perl_safesysrealloc+0xc8)[0x82d4ff8] ../bin/perl(Perl_sv_grow+0x4c4)[0x83dc5b4] ../bin/perl(Perl_sv_catpvn_flags+0x306)[0x83fbd76] ...

I expect this is caused by the buffer overflow from the above corrupting the malloc() arena.

**BISECT**

First commit that crashes in this particular way is​: commit 75697d6e4ef98ece405210de48e7529d01b619bf Author​: Karl Williamson \khw@&#8203;cpan\.org Date​: Sun Dec 21 22​:02​:30 2014 -0700

Empty \N{} in regex pattern should force /d to /u

\N{} is for Unicode names\, even if the name is actually omitted. (Accepting an empty name is\, I believe\, an accident\, and now is supported only for backwards compatibility.)

While in this case it's \N{} forcing the regexp to be unicode\, other mechanisms can cause it too​:

tony@​mars​:.../git/perl$ od -c ../125991c.pl 0000000 / \ p A 337 337 337 337 337 337 337 337 337 337 337 375 0000020 / i 0000022

Tony

p5pRT commented 9 years ago

From @tonycoz

125991b.pl

p5pRT commented 9 years ago

From @tonycoz

125991c.pl

p5pRT commented 9 years ago

The RT System itself - Status changed from 'new' to 'open'

p5pRT commented 9 years ago

From @tonycoz

On Tue Sep 08 08​:04​:55 2015\, public@​khwilliamson.com wrote​:

This is likely from my code\, so I'm taking the ticket to look at it.

Ok\, 125991 looks like the same bug (but a larger overflow that corrupts the memory arena).

Tony

p5pRT commented 9 years ago

From @khwilliamson

Thanks for finding this.

I have figured out the cause\, and am now contemplating the best solution. Tony Cook was correct that #125990 and #125991 are from the same cause\, and so I have merged the two tickets.

This bug arises when a pattern is compiled under /id rules\, has a ß in it\, and has something else in it\, such as a \p or a \N{}\, that causes it to switch to /iu rules. The latter will try to fold the ß to 'ss'\, the former does not. Thus the latter takes up 2 bytes and the former takes up 1. The sizing pass is done under /id rules\, and the allocation pass is done under /iu\, so the space allocated is less than the space actually used\, and you get the problem.

Anything under compiled under 'use 5.012' or higher or 'use utf8' is by default compiled with /u rules\, so this bug won't likely appear in such code. One would have to explicitly say /d to override the default; this is unlikely.

Karl Williamson

p5pRT commented 9 years ago

From @khwilliamson

Fixed by commit 512c0f5a351dd399dbb069e222741582044f88cc whose message is below​:

  This is a result of a design flaw that I introduced in earlier releases   when attempting to fix earlier design flaws in dealing with the outlier   character ß\, LATIN SMALL LETTER SHARP S. The uppercase of this letter   is SS\, so that when comparing case-insensitively\, it should match 'ss'\,   and hence\, in Unicode terminology\, it folds to 'ss'. This character is   the only one representable without using UTF-8 whose fold is longer than   1 byte\, and so has to have special treatment. Similarly\, the sequence   'ss' can match caselessly the single byte ß\, and this is the only such   sequence that can match something shorter than it\, unless UTF-8 is   involved. The matter is complicated by the fact that under /di rules\,   the ß and 'ss' don't match each other\, unless the target string is in   UTF-8. The solution I used earlier (and continue to use) was to create   a special regnode EXACTFU_SS under /ui rules\, in which any ß is folded   to 'ss'. But under /di rules\, a regular EXACTF regnode is used\, and any   ß is retained as-is.  
  The problem reported here arises when something during the sizing pass   tells perl to use /ui rules rather than the /di rules that were in   effect at the beginning. Recall that perl uses /d rules\, for backward   compatibility\, unless something overrides them. This can be a 'use'   declaration\, an explicit character set pattern modifier\, or something in   the pattern. This bug happens only with the final case. There are   several Unicode-defined constructs that can occur in patterns; if one is   found\, the perl interpreter infers that Unicode is desired\, and switches   from /d to /u for the whole pattern. Two such constructs are a Unicode   property\, \p{}\, and a Unicode named character\, \N{}. The   problem-reproducing code for this ticket uses the latter.  
  The problem was that the switch from /di to /ui was deferred until AFTER   the sizing pass. (A flag was set when one of these constructs was   encountered to tell the parser to later do the switch.)  
  During the second pass\, the code realizes it is under /ui\, so creates an   EXACTFU_SS node and folds the ß into 'ss'. But the first pass thought   it was under /di\, so it sized for just the ß\, i.e.\, for 1 byte\, so we   exceed the allocated space and do a wild write. This may not cause a   problem if the malloc'd space had rounded-up and there were only a few   of these ß characters.  
  One solution I considered was just keeping a global count of the ß   characters in EXACTF nodes. One could just add these to the space   reserved if /ui rules ended up being used. The problem with this is that   nodes that are near their maximum size without the extra space could   exceed it with\, and thus have to be split into 2 nodes\, and the extra   node would have an unplanned-for header\, taking up more unaccounted-for   space. So that doesn't work. One could also just reserve two bytes for   every ß in an EXACTF node\, thus wasting space unless /ui ends up being   used. But the bigger problem is that the code that splits nodes would   have to be made more complicated. It has to find a suitable splitting   spot\, by searching through the text of the node\, and now it would have   to deal with some of that space not being set.  
  Instead\, I opted to change the code so that when it finds one of these   Unicode-defined constructs\, it switches to /u immediately during the   sizing pass. That means that the parse afterwards knows that it is /u   and allocates the correct space. (We now have to remain in /u for the   remainder of the pass\, so some code had to change that reverted this.)   This fixes the test case in the ticket. But there remains a problem if   the sizing has happened earlier in the parse before the construct that   changes from /d to /u is encountered. Like​:  
  qr/.....ß....\N{}/di  
  The incorrect sizing has already happened by the time the \N{} is   encountered. One could solve this by restarting the parse whenever the   /d goes to /u (under /i\, as this issue isn't a problem except when   folding ß). That slows things down. Instead\, I opted to set a global   flag whenever a ß is found in an EXACTF node. If that flag isn't set at   the time of the /d to /u switch\, there's no need to restart the parse.  
  A 'use utf8' or 'use 5.012' or higher selects /u over /d\, so the problem   did not happen with them\, nor if the pattern has to be converted to   UTF-8\, which restarts the sizing pass\, and it only happens with the   sharp s character. And probably unless there a several ß characters\,   the rounding-up of malloc space\, would cause this to not be an issue.   These explain why this hasn't been reported from the field. -- Karl Williamson

p5pRT commented 9 years ago

@khwilliamson - Status changed from 'open' to 'pending release'

p5pRT commented 8 years ago

From @khwilliamson

Thank you for submitting this report. You have helped make Perl better.  
With the release of Perl 5.24.0 on May 9\, 2016\, this and 149 other issues have been resolved.

Perl 5.24.0 may be downloaded via https://metacpan.org/release/RJBS/perl-5.24.0

p5pRT commented 8 years ago

@khwilliamson - Status changed from 'pending release' to 'resolved'