Re: Regular rarely segmentation fault in libperl.so

p5pRT commented 12 years ago

Migrated from rt.perl.org#107392 (status was 'open')

Searchable as RT107392$

p5pRT commented 12 years ago

From perlover@perlover.com

Sorry\, previous letter had a bad formatted lines Here normal letter:

This is a bug report for perl from perlover@perlover.com\, generated with the help of perlbug 1.39 running under perl 5.10.1.

Hello\, developers of perl!

I regulary every day get a core files and segmentation fault in libperl.so I already recompiled perl from sources (SRPMs Linux Fedora Core 13 64 bit) again after problems but this bug remained. Other programs work find in my server so i think this is problem of perl\, not memory or bad binaries (i recompiled it).

Here some backtraces of perl without debug info (i cannot add debug info - server should work under big loading\, sorry)

1st 'bt' from gdb:

Program terminated with signal 11\, Segmentation fault. #0 Perl_newBINOP (my_perl=0x139c010\, type=136\, flags=0\, first=0x21ab840\, last=0x21ab800) at op.c:3074 3074 op.c: No such file or directory. in op.c Missing separate debuginfos\, use: debuginfo-install perl-5.10.1-123.fc13.x86_64 (gdb) bt #0 Perl_newBINOP (my_perl=0x139c010\, type=136\, flags=0\, first=0x21ab840\, last=0x21ab800) at op.c:3074 #1 0x00007f78f433167d in Perl_yyparse (my_perl=0x139c010) at perly.y:819 #2 0x00007f78f439a0ba in S_doeval (my_perl=0x139c010\, gimme=0\, startop=0x0\, outside=\\, seq=\) at pp_ctl.c:2981 #3 0x00007f78f439bb65 in Perl_pp_require (my_perl=0x139c010) at pp_ctl.c:3573 #4 0x00007f78f43657d6 in Perl_runops_standard (my_perl=0x139c010) at run.c:40 #5 0x00007f78f430af28 in S_run_body (my_perl=0x139c010) at perl.c:2435 #6 perl_run (my_perl=0x139c010) at perl.c:2353 #7 0x0000000000400b9c in main ()

Second core file:

Program terminated with signal 11\, Segmentation fault. #0 Perl_newOP (my_perl=0x1c88010\, type=12\, flags=0) at op.c:3024 3024 op.c: No such file or directory. in op.c Missing separate debuginfos\, use: debuginfo-install perl-5.10.1-123.fc13.x86_64 (gdb) bt #0 Perl_newOP (my_perl=0x1c88010\, type=12\, flags=0) at op.c:3024 #1 0x00007f40fe0cda42 in S_pending_ident (my_perl=0x1c88010) at toke.c:7106 #2 Perl_yylex (my_perl=0x1c88010) at toke.c:3337 #3 0x00007f40fe0d9cf6 in Perl_yyparse (my_perl=0x1c88010) at perly.c:409 #4 0x00007f40fe1440ba in S_doeval (my_perl=0x1c88010\, gimme=0\, startop=0x0\, outside=\\, seq=\) at pp_ctl.c:2981 #5 0x00007f40fe145b65 in Perl_pp_require (my_perl=0x1c88010) at pp_ctl.c:3573 #6 0x00007f40fe10f7d6 in Perl_runops_standard (my_perl=0x1c88010) at run.c:40 #7 0x00007f40fe0b4f28 in S_run_body (my_perl=0x1c88010) at perl.c:2435 #8 perl_run (my_perl=0x1c88010) at perl.c:2353 #9 0x0000000000400b9c in main ()

3rd core file:

Program terminated with signal 11\, Segmentation fault. #0 Perl_newBINOP (my_perl=0xa4f010\, type=136\, flags=0\, first=0x175d7f0\, last=0x175d950) at op.c:3074 3074 op.c: No such file or directory. in op.c Missing separate debuginfos\, use: debuginfo-install perl-5.10.1-123.fc13.x86_64 (gdb) bt #0 Perl_newBINOP (my_perl=0xa4f010\, type=136\, flags=0\, first=0x175d7f0\, last=0x175d950) at op.c:3074 #1 0x00007fca68d146dd in Perl_yyparse (my_perl=0xa4f010) at perly.y:809 #2 0x00007fca68d7d0ba in S_doeval (my_perl=0xa4f010\, gimme=0\, startop=0x0\, outside=\\, seq=\) at pp_ctl.c:2981 #3 0x00007fca68d7eb65 in Perl_pp_require (my_perl=0xa4f010) at pp_ctl.c:3573 #4 0x00007fca68d487d6 in Perl_runops_standard (my_perl=0xa4f010) at run.c:40 #5 0x00007fca68ced42f in Perl_call_sv (my_perl=0xa4f010\, sv=0x1711840\, flags=6) at perl.c:2721 #6 0x00007fca68ced96d in Perl_call_list (my_perl=0xa4f010\, oldscope=21\, paramList=0x1711780) at perl.c:5275 #7 0x00007fca68cd9c39 in S_process_special_blocks (my_perl=0xa4f010\, fullname=\\, gv=0x17118e8\, cv=0x1711840) at op.c:5864 #8 0x00007fca68ce7612 in Perl_newATTRSUB (my_perl=0xa4f010\, floor=363\, o=\\, proto=\\, attrs=0x0\, block=0x1686250) at op.c:5835 #9 0x00007fca68ce6593 in Perl_utilize (my_perl=0xa4f010\, aver=1\, floor=363\, version=\\, idop=0x16edb40\, arg=\<value optimized out>) at op.c:3878 #10 0x00007fca68d14c87 in Perl_yyparse (my_perl=0xa4f010) at perly.y:659 #11 0x00007fca68d7d0ba in S_doeval (my_perl=0xa4f010\, gimme=0\, startop=0x0\, outside=\\, seq=\) at pp_ctl.c:2981 #12 0x00007fca68d7eb65 in Perl_pp_require (my_perl=0xa4f010) at pp_ctl.c:3573 #13 0x00007fca68d487d6 in Perl_runops_standard (my_perl=0xa4f010) at run.c:40 #14 0x00007fca68ced42f in Perl_call_sv (my_perl=0xa4f010\, sv=0x16a6970\, flags=6) at perl.c:2721 #15 0x00007fca68ced96d in Perl_call_list (my_perl=0xa4f010\, oldscope=15\, paramList=0x1390850) at perl.c:5275 #16 0x00007fca68cd9c39 in S_process_special_blocks (my_perl=0xa4f010\, fullname=\\, gv=0xbeb8e0\, cv=0x16a6970) at op.c:5864 #17 0x00007fca68ce7612 in Perl_newATTRSUB (my_perl=0xa4f010\, floor=224\, o=\\, proto=\\, attrs=0x0\, block=0x1369240) at op.c:5835 #18 0x00007fca68ce6593 in Perl_utilize (my_perl=0xa4f010\, aver=1\, floor=224\, version=\\, idop=0x1392090\, arg=\<value optimized out>) at op.c:3878 #19 0x00007fca68d14c87 in Perl_yyparse (my_perl=0xa4f010) at perly.y:659 #20 0x00007fca68d7d0ba in S_doeval (my_perl=0xa4f010\, gimme=0\, startop=0x0\, outside=\\, seq=\) at pp_ctl.c:2981 #21 0x00007fca68d7eb65 in Perl_pp_require (my_perl=0xa4f010) at pp_ctl.c:3573 #22 0x00007fca68d487d6 in Perl_runops_standard (my_perl=0xa4f010) at run.c:40 #23 0x00007fca68cedf28 in S_run_body (my_perl=0xa4f010) at perl.c:2435 #24 perl_run (my_perl=0xa4f010) at perl.c:2353 #25 0x0000000000400b9c in main ()

4th core file:

Program terminated with signal 11\, Segmentation fault. #0 Perl_newSTATEOP (my_perl=0xf6d010\, flags=0\, label=0x0\, o=0x1bb72d0) at op.c:4350 4350 op.c: No such file or directory. in op.c Missing separate debuginfos\, use: debuginfo-install perl-5.10.1-123.fc13.x86_64 (gdb) bt #0 Perl_newSTATEOP (my_perl=0xf6d010\, flags=0\, label=0x0\, o=0x1bb72d0) at op.c:4350 #1 0x00007f09571c61c8 in Perl_yyparse (my_perl=0xf6d010) at perly.y:230 #2 0x00007f09572300ba in S_doeval (my_perl=0xf6d010\, gimme=0\, startop=0x0\, outside=\\, seq=\) at pp_ctl.c:2981 #3 0x00007f0957231b65 in Perl_pp_require (my_perl=0xf6d010) at pp_ctl.c:3573 #4 0x00007f09571fb7d6 in Perl_runops_standard (my_perl=0xf6d010) at run.c:40 #5 0x00007f09571a0f28 in S_run_body (my_perl=0xf6d010) at perl.c:2435 #6 perl_run (my_perl=0xf6d010) at perl.c:2353 #7 0x0000000000400b9c in main ()

And so on... Always core files will happen in one place: op.c file

I don't know fixed this bug or no but i didn't find in Google some bugs like this.

Best regards\, Perlover

Flags: category=core severity=low

This perlbug was built using Perl 5.10.1 in the Fedora build system. It is being executed now by Perl 5.10.1 - Wed Dec 28 17:51:47 YEKT 2011.

Site configuration information for perl 5.10.1:

Configured by Red Hat\, Inc. at Wed Dec 28 17:51:47 YEKT 2011.

Summary of my perl5 (revision 5 version 10 subversion 1) configuration:

Platform: osname=linux\, osvers=2.6.34.9-69.fc13.x86_64\, archname=x86_64-linux-thread-multi uname='linux 213.174.142.19 2.6.34.9-69.fc13.x86_64 #1 smp tue may 3 09:23:03 utc 2011 x86_64 x86_64 x86_64 gnulinux ' config_args='-des -Doptimize=-O2 -g -Dccdlflags=-Wl\,--enable-new-dtags -DDEBUGGING=-g -Dversion=5.10.1 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat\, Inc. -Dprefix=/usr -Dvend hint=recommended\, useposix=true\, d_sigaction=define useithreads=define\, usemultiplicity=define useperlio=define\, d_sfio=undef\, uselargefiles=define\, usesocks=undef use64bitint=define\, use64bitall=define\, uselongdouble=undef usemymalloc=n\, bincompat5005=undef Compiler: cc='gcc'\, ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'\, optimize='-O2 -g'\, cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include' ccversion=''\, gccversion='4.4.5 20101112 (Red Hat 4.4.5-2)'\, gccosandvers='' intsize=4\, longsize=8\, ptrsize=8\, doublesize=8\, byteorder=12345678 d_longlong=define\, longlongsize=8\, d_longdbl=define\, longdblsize=16 ivtype='long'\, ivsize=8\, nvtype='double'\, nvsize=8\, Off_t='off_t'\, lseeksize=8 alignbytes=8\, prototype=define Linker and Libraries: ld='gcc'\, ldflags =' -fstack-protector' libpth=/usr/local/lib64 /lib64 /usr/lib64 libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc libc=\, so=so\, useshrplib=true\, libperl=libperl.so gnulibc_version='2.12.2' Dynamic Linking: dlsrc=dl_dlopen.xs\, dlext=so\, d_dlsymun=undef\, ccdlflags='-Wl\,--enable-new-dtags -Wl\,-rpath\,/usr/lib64/perl5/CORE' cccdlflags='-fPIC'\, lddlflags='-shared -O2 -g -fstack-protector'

Locally applied patches:

@INC for perl 5.10.1: /home/cjperl/local/lib/perl5/x86_64-linux-thread-multi /home/cjperl/local/lib/perl5 /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5 /usr/share/perl5 /usr/lib64/perl5 /usr/share/perl5 /usr/local/lib64/perl5/site_perl/5.10.0/x86_64-linux-thread-multi /usr/local/lib/perl5/site_perl/5.10.0 /usr/lib64/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl /usr/lib/perl5/site_perl .

Environment for perl 5.10.1: HOME=/home/cjperl LANG=ru_RU.utf8 LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/cjperl/local/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cjperl/bin PERL5LIB=/home/cjperl/local/lib/perl5 PERL_BADLANG (unset) SHELL=/bin/bash

p5pRT commented 12 years ago

From @cpansprout

On Mon Jan 02 03:25:46 2012\, Perlover wrote:

Sorry\, previous letter had a bad formatted lines Here normal letter:

This is a bug report for perl from perlover@perlover.com\, generated with the help of perlbug 1.39 running under perl 5.10.1.

-----------------------------------------------------------------

Hello\, developers of perl!

I regulary every day get a core files and segmentation fault in libperl.so I already recompiled perl from sources (SRPMs Linux Fedora Core 13 64 bit) again after problems but this bug remained. Other programs work find in my server so i think this is problem of perl\, not memory or bad binaries (i recompiled it).

Here some backtraces of perl without debug info (i cannot add debug info - server should work under big loading\, sorry)

Are you using any XS modules that create their own ops or fiddle with PL_check?

Do you know what Perl code is being compiled?

Typing

call Perl_warn(my_perl\, "here")

in the debugger (omit my_perl for non-threaded builds) should result in output like this:

here at lib/Foo/Bar.pm line 3123.

Maybe you could send some code snippets surrounding the offending lines.

I see you are using 5.10.1. There have been a ton of bug fixes since then\, so this problem may already be gone in a later perl. Also\, please be aware that 5.10 is not supported any more. It’s not likely to get any new bug fixes.

1st 'bt' from gdb:

Program terminated with signal 11\, Segmentation fault. #0 Perl_newBINOP (my_perl=0x139c010\, type=136\, flags=0\, first=0x21ab840\, last=0x21ab800) at op.c:3074 3074 op.c: No such file or directory. in op.c Missing separate debuginfos\, use: debuginfo-install perl-5.10.1- 123.fc13.x86_64 (gdb) bt #0 Perl_newBINOP (my_perl=0x139c010\, type=136\, flags=0\, first=0x21ab840\, last=0x21ab800) at op.c:3074 #1 0x00007f78f433167d in Perl_yyparse (my_perl=0x139c010) at perly.y:819 #2 0x00007f78f439a0ba in S_doeval (my_perl=0x139c010\, gimme=0\, startop=0x0\, outside=\\, seq=\) at pp_ctl.c:2981 #3 0x00007f78f439bb65 in Perl_pp_require (my_perl=0x139c010) at pp_ctl.c:3573 #4 0x00007f78f43657d6 in Perl_runops_standard (my_perl=0x139c010) at run.c:40 #5 0x00007f78f430af28 in S_run_body (my_perl=0x139c010) at perl.c:2435 #6 perl_run (my_perl=0x139c010) at perl.c:2353 #7 0x0000000000400b9c in main ()

---

Second core file:

Program terminated with signal 11\, Segmentation fault. #0 Perl_newOP (my_perl=0x1c88010\, type=12\, flags=0) at op.c:3024 3024 op.c: No such file or directory. in op.c Missing separate debuginfos\, use: debuginfo-install perl-5.10.1- 123.fc13.x86_64 (gdb) bt #0 Perl_newOP (my_perl=0x1c88010\, type=12\, flags=0) at op.c:3024 #1 0x00007f40fe0cda42 in S_pending_ident (my_perl=0x1c88010) at toke.c:7106 #2 Perl_yylex (my_perl=0x1c88010) at toke.c:3337 #3 0x00007f40fe0d9cf6 in Perl_yyparse (my_perl=0x1c88010) at perly.c:409 #4 0x00007f40fe1440ba in S_doeval (my_perl=0x1c88010\, gimme=0\, startop=0x0\, outside=\\, seq=\) at pp_ctl.c:2981 #5 0x00007f40fe145b65 in Perl_pp_require (my_perl=0x1c88010) at pp_ctl.c:3573 #6 0x00007f40fe10f7d6 in Perl_runops_standard (my_perl=0x1c88010) at run.c:40 #7 0x00007f40fe0b4f28 in S_run_body (my_perl=0x1c88010) at perl.c:2435 #8 perl_run (my_perl=0x1c88010) at perl.c:2353 #9 0x0000000000400b9c in main ()

---

3rd core file:

Program terminated with signal 11\, Segmentation fault. #0 Perl_newBINOP (my_perl=0xa4f010\, type=136\, flags=0\, first=0x175d7f0\, last=0x175d950) at op.c:3074 3074 op.c: No such file or directory. in op.c Missing separate debuginfos\, use: debuginfo-install perl-5.10.1- 123.fc13.x86_64 (gdb) bt #0 Perl_newBINOP (my_perl=0xa4f010\, type=136\, flags=0\, first=0x175d7f0\, last=0x175d950) at op.c:3074 #1 0x00007fca68d146dd in Perl_yyparse (my_perl=0xa4f010) at perly.y:809 #2 0x00007fca68d7d0ba in S_doeval (my_perl=0xa4f010\, gimme=0\, startop=0x0\, outside=\\, seq=\) at pp_ctl.c:2981 #3 0x00007fca68d7eb65 in Perl_pp_require (my_perl=0xa4f010) at pp_ctl.c:3573 #4 0x00007fca68d487d6 in Perl_runops_standard (my_perl=0xa4f010) at run.c:40 #5 0x00007fca68ced42f in Perl_call_sv (my_perl=0xa4f010\, sv=0x1711840\, flags=6) at perl.c:2721 #6 0x00007fca68ced96d in Perl_call_list (my_perl=0xa4f010\, oldscope=21\, paramList=0x1711780) at perl.c:5275 #7 0x00007fca68cd9c39 in S_process_special_blocks (my_perl=0xa4f010\, fullname=\\, gv=0x17118e8\, cv=0x1711840) at op.c:5864 #8 0x00007fca68ce7612 in Perl_newATTRSUB (my_perl=0xa4f010\, floor=363\, o=\\, proto=\\, attrs=0x0\, block=0x1686250) at op.c:5835 #9 0x00007fca68ce6593 in Perl_utilize (my_perl=0xa4f010\, aver=1\, floor=363\, version=\\, idop=0x16edb40\, arg=\<value optimized out>) at op.c:3878 #10 0x00007fca68d14c87 in Perl_yyparse (my_perl=0xa4f010) at perly.y:659 #11 0x00007fca68d7d0ba in S_doeval (my_perl=0xa4f010\, gimme=0\, startop=0x0\, outside=\\, seq=\) at pp_ctl.c:2981 #12 0x00007fca68d7eb65 in Perl_pp_require (my_perl=0xa4f010) at pp_ctl.c:3573 #13 0x00007fca68d487d6 in Perl_runops_standard (my_perl=0xa4f010) at run.c:40 #14 0x00007fca68ced42f in Perl_call_sv (my_perl=0xa4f010\, sv=0x16a6970\, flags=6) at perl.c:2721 #15 0x00007fca68ced96d in Perl_call_list (my_perl=0xa4f010\, oldscope=15\, paramList=0x1390850) at perl.c:5275 #16 0x00007fca68cd9c39 in S_process_special_blocks (my_perl=0xa4f010\, fullname=\\, gv=0xbeb8e0\, cv=0x16a6970) at op.c:5864 #17 0x00007fca68ce7612 in Perl_newATTRSUB (my_perl=0xa4f010\, floor=224\, o=\\, proto=\\, attrs=0x0\, block=0x1369240) at op.c:5835 #18 0x00007fca68ce6593 in Perl_utilize (my_perl=0xa4f010\, aver=1\, floor=224\, version=\\, idop=0x1392090\, arg=\<value optimized out>) at op.c:3878 #19 0x00007fca68d14c87 in Perl_yyparse (my_perl=0xa4f010) at perly.y:659 #20 0x00007fca68d7d0ba in S_doeval (my_perl=0xa4f010\, gimme=0\, startop=0x0\, outside=\\, seq=\) at pp_ctl.c:2981 #21 0x00007fca68d7eb65 in Perl_pp_require (my_perl=0xa4f010) at pp_ctl.c:3573 #22 0x00007fca68d487d6 in Perl_runops_standard (my_perl=0xa4f010) at run.c:40 #23 0x00007fca68cedf28 in S_run_body (my_perl=0xa4f010) at perl.c:2435 #24 perl_run (my_perl=0xa4f010) at perl.c:2353 #25 0x0000000000400b9c in main ()

---

4th core file:

Program terminated with signal 11\, Segmentation fault. #0 Perl_newSTATEOP (my_perl=0xf6d010\, flags=0\, label=0x0\, o=0x1bb72d0) at op.c:4350 4350 op.c: No such file or directory. in op.c Missing separate debuginfos\, use: debuginfo-install perl-5.10.1- 123.fc13.x86_64 (gdb) bt #0 Perl_newSTATEOP (my_perl=0xf6d010\, flags=0\, label=0x0\, o=0x1bb72d0) at op.c:4350 #1 0x00007f09571c61c8 in Perl_yyparse (my_perl=0xf6d010) at perly.y:230 #2 0x00007f09572300ba in S_doeval (my_perl=0xf6d010\, gimme=0\, startop=0x0\, outside=\\, seq=\) at pp_ctl.c:2981 #3 0x00007f0957231b65 in Perl_pp_require (my_perl=0xf6d010) at pp_ctl.c:3573 #4 0x00007f09571fb7d6 in Perl_runops_standard (my_perl=0xf6d010) at run.c:40 #5 0x00007f09571a0f28 in S_run_body (my_perl=0xf6d010) at perl.c:2435 #6 perl_run (my_perl=0xf6d010) at perl.c:2353 #7 0x0000000000400b9c in main ()

And so on... Always core files will happen in one place: op.c file

I don't know fixed this bug or no but i didn't find in Google some bugs like this.

--

Father Chrysostomos

p5pRT commented 12 years ago

The RT System itself - Status changed from 'new' to 'open'

p5pRT commented 12 years ago

From perlover@perlover.com

I think this but may be related with bug # 72406 ( https://rt-archive.perl.org/perl5/Public/Bug/Display.html?id=72406 ) I got same segmentation fault if i do: perl -le 'do{print("foobar");}until(1)}'

p5pRT commented 12 years ago

From @iabyn

On Mon\, Jan 02\, 2012 at 03:25:47AM -0800\, Perlover wrote:

I regulary every day get a core files and segmentation fault in libperl.so I already recompiled perl from sources (SRPMs Linux Fedora Core 13 64 bit) again after problems but this bug remained. Other programs work find in my server so i think this is problem of perl\, not memory or bad binaries (i recompiled it).

Here some backtraces of perl without debug info (i cannot add debug info - server should work under big loading\, sorry)

[snip]

All the segfaults occur where the code is doing the first dereference of a newly-malloced OP. Interestingly\, perl doesn't check the return value of malloc for a new OP\, so it won't detect malloc errors (i.e. out of memory) in those situations.

So I reckon that the most likely thing is that your perl process is running out of memory (either because your server doesn't have enough\, or because your perl processes are run within a ulimit-restricted environment). Which perl then fails to report cleanly.

-- In the 70's we wore flares because we didn't know any better. What possible excuse does the current generation have?

p5pRT commented 12 years ago

From mark@exonetric.com

On 4 Jan 2012\, at 13:08\, Dave Mitchell wrote:

On Mon\, Jan 02\, 2012 at 03:25:47AM -0800\, Perlover wrote:

I regulary every day get a core files and segmentation fault in libperl.so I already recompiled perl from sources (SRPMs Linux Fedora Core 13 64 bit) again after problems but this bug remained. Other programs work find in my server so i think this is problem of perl\, not memory or bad binaries (i recompiled it).

Here some backtraces of perl without debug info (i cannot add debug info - server should work under big loading\, sorry)

[snip]

All the segfaults occur where the code is doing the first dereference of a newly-malloced OP. Interestingly\, perl doesn't check the return value of malloc for a new OP\, so it won't detect malloc errors (i.e. out of memory) in those situations.

So I reckon that the most likely thing is that your perl process is running out of memory (either because your server doesn't have enough\, or because your perl processes are run within a ulimit-restricted environment). Which perl then fails to report cleanly.

Now\, this is an interesting report for us as we have recently implemented address-space limits (as data segment limits are no good with the system malloc). We routinely see segfaults as well and we *know* it's do with running out of address space after a malloc.

I was a little surprised that such an easy to check condition was resulting in a segfault. This is under perl5.10.0 on 64-bit amd64 linux 2.6.26-something.

- Mark

p5pRT commented 12 years ago

From perlover@perlover.com

Good day!

[snip]

All the segfaults occur where the code is doing the first dereference of a newly-malloced OP. Interestingly\, perl doesn't check the return value of malloc for a new OP\, so it won't detect malloc errors (i.e. out of memory) in those situations.

Yes\, it may be These scripts are run from not root This user has a soft limits as:

core file size (blocks\, -c) 0 data seg size (kbytes\, -d) unlimited scheduling priority (-e) 0 file size (blocks\, -f) unlimited pending signals (-i) 128522 max locked memory (kbytes\, -l) 64 max memory size (kbytes\, -m) unlimited open files (-n) 1024 pipe size (512 bytes\, -p) 8 POSIX message queues (bytes\, -q) 819200 real-time priority (-r) 0 stack size (kbytes\, -s) 10240 cpu time (seconds\, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes\, -v) unlimited file locks (-x) unlimited

Here "max memory size" is unlimited But "stack size" is 10Mb Can this (stack size) affect to new OP or malloc? I think malloc limited by "max memory size" not stack size

So I reckon that the most likely thing is that your perl process is running out of memory (either because your server doesn't have enough\, or because your perl processes are run within a ulimit-restricted environment). Which perl then fails to report cleanly.

ulimits i dumped above

Best regards\, Perlover

p5pRT commented 12 years ago

From @iabyn

On Wed\, Jan 04\, 2012 at 02:59:46PM +0100\, Perlover wrote:

All the segfaults occur where the code is doing the first dereference of a newly-malloced OP. Interestingly\, perl doesn't check the return value of malloc for a new OP\, so it won't detect malloc errors (i.e. out of memory) in those situations.

Yes\, it may be These scripts are run from not root This user has a soft limits as:

core file size (blocks\, -c) 0 data seg size (kbytes\, -d) unlimited scheduling priority (-e) 0 file size (blocks\, -f) unlimited pending signals (-i) 128522 max locked memory (kbytes\, -l) 64 max memory size (kbytes\, -m) unlimited open files (-n) 1024 pipe size (512 bytes\, -p) 8 POSIX message queues (bytes\, -q) 819200 real-time priority (-r) 0 stack size (kbytes\, -s) 10240 cpu time (seconds\, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes\, -v) unlimited file locks (-x) unlimited

Here "max memory size" is unlimited But "stack size" is 10Mb Can this (stack size) affect to new OP or malloc? I think malloc limited by "max memory size" not stack size

Yeah\, it shouldn't be affected by stack size (that would just trigger a SEGV at random places where it tries to call a function).

Could you apply this crude patch to your 5.10.1 perl? It checks for a zero return from malloc()\, outputs a message to stderr\, and aborts. With it we might at least see whether this is where the problem lies.

Inline Patch

```diff --- op.h- 2012-01-08 15:54:03.944453254 +0000 +++ op.h 2012-01-08 16:33:07.592199391 +0000 @@ -640,9 +640,14 @@ (var = (OP *) Perl_Slab_Alloc(aTHX_ size)) #define FreeOp(p) Perl_Slab_Free(aTHX_ p) #else -#define NewOp(m, var, c, type) \ +#define NewOp(m, var, c, type) STMT_START { \ (var = (MEM_WRAP_CHECK_(c,type) \ - (type*)PerlMemShared_calloc(c, sizeof(type)))) + (type*)PerlMemShared_calloc(c, sizeof(type)))); \ + if (!var) { \ + write(2, STR_WITH_LEN("Zero malloc in NewOp\n")); \ + abort(); \ + } \ + } STMT_END; #define NewOpSz(m, var, size) \ (var = (OP*)PerlMemShared_calloc(1, size)) #define FreeOp(p) PerlMemShared_free(p) -- ```

A walk of a thousand miles begins with a single step... then continues for another 1\,999\,999 or so.

p5pRT commented 12 years ago

From @iabyn

On Tue\, Jan 17\, 2012 at 11:14:26AM +0100\, Perlover wrote:

I think it is good that happened because it signals to us that your abort() was executed and so problem in out of memory of malloc

What do you think?

Yeah\, I think it's definitely a lack of memory problem at your end\, so there's not much we can do to fix that. What we do need to do at this end though\, is to fix that bit of perl code so that it detects and reports out of memory errors.

-- The warp engines start playing up a bit\, but seem to sort themselves out after a while without any intervention from boy genius Wesley Crusher. -- Things That Never Happen in "Star Trek" #17

p5pRT commented 12 years ago

From perlover@perlover.com

Good day\, Dave and everybody in perl bug list!

I have two core files after your patches My STDERR is empty after segfaults and it looks like segfault from your abort()

Here backtrace from gdb:

#0 0x00007f07e7db28f5 in raise () from /lib64/libc.so.6 Missing separate debuginfos\, use: debuginfo-install perl-5.10.1-123.fc13.x86_64 (gdb) bt #0 0x00007f07e7db28f5 in raise () from /lib64/libc.so.6 #1 0x00007f07e7db40d5 in abort () from /lib64/libc.so.6 #2 0x00007f07e904a363 in Perl_newSTATEOP (my_perl=0x239e010\, flags=0\, label=0x0\, o=0x32b5080) at op.c:4344 #3 0x00007f07e907d458 in Perl_yyparse (my_perl=0x239e010) at perly.y:230 #4 0x00007f07e90e734a in S_doeval (my_perl=0x239e010\, gimme=0\, startop=0x0\, outside=\\, seq=\) at pp_ctl.c:2981 #5 0x00007f07e90e8df5 in Perl_pp_require (my_perl=0x239e010) at pp_ctl.c:3573 #6 0x00007f07e90b2a66 in Perl_runops_standard (my_perl=0x239e010) at run.c:40 #7 0x00007f07e90576bf in Perl_call_sv (my_perl=0x239e010\, sv=0x2cd1288\, flags=6) at perl.c:2721 #8 0x00007f07e9057bfd in Perl_call_list (my_perl=0x239e010\, oldscope=14\, paramList=0x25a1238) at perl.c:5275 #9 0x00007f07e9043c39 in S_process_special_blocks (my_perl=0x239e010\, fullname=\\, gv=0x2cc50a8\, cv=0x2cd1288) at op.c:5864 #10 0x00007f07e9051872 in Perl_newATTRSUB (my_perl=0x239e010\, floor=217\, o=\\, proto=\\, attrs=0x0\, block=0x2d6f890) at op.c:5835 #11 0x00007f07e90507f3 in Perl_utilize (my_perl=0x239e010\, aver=1\, floor=217\, version=\\, idop=0x2c74d60\, arg=\<value optimized out>) at op.c:3878 #12 0x00007f07e907ef17 in Perl_yyparse (my_perl=0x239e010) at perly.y:659 #13 0x00007f07e90e734a in S_doeval (my_perl=0x239e010\, gimme=0\, startop=0x0\, outside=\\, seq=\) at pp_ctl.c:2981 #14 0x00007f07e90e8df5 in Perl_pp_require (my_perl=0x239e010) at pp_ctl.c:3573 #15 0x00007f07e90b2a66 in Perl_runops_standard (my_perl=0x239e010) at run.c:40 #16 0x00007f07e90581b8 in S_run_body (my_perl=0x239e010) at perl.c:2435 #17 perl_run (my_perl=0x239e010) at perl.c:2353 #18 0x0000000000400b9c in main ()

And second corefile: #0 0x00007f0f7645a8f5 in raise () from /lib64/libc.so.6 Missing separate debuginfos\, use: debuginfo-install perl-5.10.1-123.fc13.x86_64 (gdb) bt #0 0x00007f0f7645a8f5 in raise () from /lib64/libc.so.6 #1 0x00007f0f7645c0d5 in abort () from /lib64/libc.so.6 #2 0x00007f0f776f1b6a in Perl_newOP (my_perl=0x17c5010\, type=12\, flags=0) at op.c:3023 #3 0x00007f0f77718cd2 in S_pending_ident (my_perl=0x17c5010) at toke.c:7106 #4 Perl_yylex (my_perl=0x17c5010) at toke.c:3337 #5 0x00007f0f77724f86 in Perl_yyparse (my_perl=0x17c5010) at perly.c:409 #6 0x00007f0f7778f34a in S_doeval (my_perl=0x17c5010\, gimme=0\, startop=0x0\, outside=\\, seq=\) at pp_ctl.c:2981 #7 0x00007f0f77790df5 in Perl_pp_require (my_perl=0x17c5010) at pp_ctl.c:3573 #8 0x00007f0f7775aa66 in Perl_runops_standard (my_perl=0x17c5010) at run.c:40 #9 0x00007f0f776ff6bf in Perl_call_sv (my_perl=0x17c5010\, sv=0x1961880\, flags=6) at perl.c:2721 #10 0x00007f0f776ffbfd in Perl_call_list (my_perl=0x17c5010\, oldscope=15\, paramList=0x20f7fe0) at perl.c:5275 #11 0x00007f0f776ebc39 in S_process_special_blocks (my_perl=0x17c5010\, fullname=\\, gv=0x19618e0\, cv=0x1961880) at op.c:5864 #12 0x00007f0f776f9872 in Perl_newATTRSUB (my_perl=0x17c5010\, floor=224\, o=\\, proto=\\, attrs=0x0\, block=0x21982a0) at op.c:5835 #13 0x00007f0f776f87f3 in Perl_utilize (my_perl=0x17c5010\, aver=1\, floor=224\, version=\\, idop=0x22888e0\, arg=\<value optimized out>) at op.c:3878 #14 0x00007f0f77726f17 in Perl_yyparse (my_perl=0x17c5010) at perly.y:659 #15 0x00007f0f7778f34a in S_doeval (my_perl=0x17c5010\, gimme=0\, startop=0x0\, outside=\\, seq=\) at pp_ctl.c:2981 #16 0x00007f0f77790df5 in Perl_pp_require (my_perl=0x17c5010) at pp_ctl.c:3573 #17 0x00007f0f7775aa66 in Perl_runops_standard (my_perl=0x17c5010) at run.c:40 #18 0x00007f0f777001b8 in S_run_body (my_perl=0x17c5010) at perl.c:2435 #19 perl_run (my_perl=0x17c5010) at perl.c:2353 #20 0x0000000000400b9c in main ()

I think it is good that happened because it signals to us that your abort() was executed and so problem in out of memory of malloc

What do you think?

Perlover

2012/1/9 Perlover \perlover@perlover\.com:

I patched and installed perl I will let know to you about "Zero malloc in NewOp" messages in STDERR if its will be

Best regards :)

2012/1/8 Dave Mitchell \davem@iabyn\.com:

On Wed\, Jan 04\, 2012 at 02:59:46PM +0100\, Perlover wrote:

All the segfaults occur where the code is doing the first dereference of a newly-malloced OP. Interestingly\, perl doesn't check the return value of malloc for a new OP\, so it won't detect malloc errors (i.e. out of memory) in those situations.

Yes\, it may be These scripts are run from not root This user has a soft limits as:

core file size (blocks\, -c) 0 data seg size (kbytes\, -d) unlimited scheduling priority (-e) 0 file size (blocks\, -f) unlimited pending signals (-i) 128522 max locked memory (kbytes\, -l) 64 max memory size (kbytes\, -m) unlimited open files (-n) 1024 pipe size (512 bytes\, -p) 8 POSIX message queues (bytes\, -q) 819200 real-time priority (-r) 0 stack size (kbytes\, -s) 10240 cpu time (seconds\, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes\, -v) unlimited file locks (-x) unlimited

Here "max memory size" is unlimited But "stack size" is 10Mb Can this (stack size) affect to new OP or malloc? I think malloc limited by "max memory size" not stack size

Yeah\, it shouldn't be affected by stack size (that would just trigger a SEGV at random places where it tries to call a function).

Could you apply this crude patch to your 5.10.1 perl? It checks for a zero return from malloc()\, outputs a message to stderr\, and aborts. With it we might at least see whether this is where the problem lies.

--- op.h- 2012-01-08 15:54:03.944453254 +0000 +++ op.h 2012-01-08 16:33:07.592199391 +0000 @@ -640\,9 +640\,14 @@ (var = (OP *) Perl_Slab_Alloc(aTHX_ size)) #define FreeOp(p) Perl_Slab_Free(aTHX_ p) #else -#define NewOp(m\, var\, c\, type) \ +#define NewOp(m\, var\, c\, type) STMT_START { \ (var = (MEM_WRAP_CHECK_(c\,type) \ - (type*)PerlMemShared_calloc(c\, sizeof(type)))) + (type*)PerlMemShared_calloc(c\, sizeof(type)))); \ + if (!var) { \ + write(2\, STR_WITH_LEN("Zero malloc in NewOp\n")); \ + abort(); \ + } \ + } STMT_END; #define NewOpSz(m\, var\, size) \ (var = (OP*)PerlMemShared_calloc(1\, size)) #define FreeOp(p) PerlMemShared_free(p)

-- A walk of a thousand miles begins with a single step... then continues for another 1\,999\,999 or so.

p5pRT commented 11 years ago

From @jkeenan

On Tue Jan 17 03:48:20 2012\, davem wrote:

On Tue\, Jan 17\, 2012 at 11:14:26AM +0100\, Perlover wrote:

I think it is good that happened because it signals to us that your abort() was executed and so problem in out of memory of malloc

What do you think?

Yeah\, I think it's definitely a lack of memory problem at your end\, so there's not much we can do to fix that. What we do need to do at this end though\, is to fix that bit of perl code so that it detects and reports out of memory errors.

Dave\, do you have any suggestions as to how we could fix this failure to detect out-of-memory problem?

Thank you very much. Jim Keenan

p5pRT commented 11 years ago

From @iabyn

On Sun\, Feb 03\, 2013 at 10:48:41AM -0800\, James E Keenan via RT wrote:

On Tue Jan 17 03:48:20 2012\, davem wrote:

On Tue\, Jan 17\, 2012 at 11:14:26AM +0100\, Perlover wrote:

I think it is good that happened because it signals to us that your abort() was executed and so problem in out of memory of malloc

What do you think?

Yeah\, I think it's definitely a lack of memory problem at your end\, so there's not much we can do to fix that. What we do need to do at this end though\, is to fix that bit of perl code so that it detects and reports out of memory errors.

Dave\, do you have any suggestions as to how we could fix this failure to detect out-of-memory problem?

Not without looking at the code (and I'd look at the code if/when I was in the process of fixing the issue)

-- The Enterprise successfully ferries an alien VIP from one place to another without serious incident. -- Things That Never Happen in "Star Trek" #7

Perl / perl5