Perl / perl5

🐪 The Perl programming language
https://dev.perl.org/perl5/
Other
1.96k stars 555 forks source link

perl crash related to threads and lock() #8103

Closed p5pRT closed 19 years ago

p5pRT commented 19 years ago

Migrated from rt.perl.org#37133 (status was 'resolved')

Searchable as RT37133$

p5pRT commented 19 years ago

From sgromoll@grad.physics.sunysb.edu

This is a bug report for perl from sgromoll@​grad.physics.sunysb.edu\, generated with the help of perlbug 1.35 running under perl v5.8.4.


[Please enter your report here]

The simple program below causes a crash in perl on the following platforms​: (5.8.7) 2.6.8-2-mckinley-smp #1 SMP ia64 GNU/Linux (5.8.4) 2.6.8.1.20040829 #1 SMP i686 GNU/Linux (5.8.4) 2.4.25 #6 SMP i686 GNU/Linux (5.8.4) 2.4.27.20040829 #5 SMP i686 GNU/Linux

(perlbug was run on the 2.4.25 machine\, but sent from another machine)

the same program does not appear to crash on the following platforms​: (5.8.4) 2.6.10-5-386 #1 i686 GNU/Linux (5.8.0) 2.4.21-4.ELsmp #1 SMP i686 i686 i386 GNU/Linux

When perl crashes it gives one of the following errors\, depending on the platform​:

== panic​: destruct destroyed thread 0@​a\uffffl0@​l0@​ during global destruction. Segmentation fault == (in cleanup) panic​: destruct destroyed thread 828c0e8 (no reference) at ./test.pl line 23. panic​: MUTEX_DESTROY (16) [threads.xs​:152] during global destruction. Segmentation fault

the program also sometimes generates a segmentation fault (but no error) when CTRL-C is pressed while it is running.

Sometimes the thread reference is a valid number\, but sometimes it is printed out as random ascii characters (as in the first message above). the program usually crashes within a minute or two. on the 2.6.10 machine where it did not crash\, i let the program run for 20 minutes or more.

Here is the program​:

==== #!/usr/bin/perl -w

use warnings; use strict; use threads; use threads​::shared;

my $var : shared = 0;

sub go {   lock($var); }

while(1) {   my $thr1 = threads->new(\&go);   $thr1->detach; }

[please do not change anything below this line]



flags​:   category=core   severity=high


site configuration information for perl v5.8.4​:

configured by debian project at thu feb 3 01​:11​:27 est 2005.

summary of my perl5 (revision 5 version 8 subversion 4) configuration​:   platform​:   osname=linux\, osvers=2.4.27-ti1211\, archname=i386-linux-thread-multi   uname='linux kosh 2.4.27-ti1211 #1 sun sep 19 18​:17​:45 est 2004 i686 gnulinux '   config_args='-dusethreads -duselargefiles -dccflags=-ddebian -dcccdlflags=-fpic -darchname=i386-linux -dprefix=/usr -dprivlib=/usr/share/perl/5.8 -darchlib=/usr/lib/perl/5.8 -dvendorprefix=/usr -dvendorlib=/usr/share/perl5 -dvendorarch=/usr/lib/perl5 -dsiteprefix=/usr/local -dsitelib=/usr/local/share/perl/5.8.4 -dsitearch=/usr/local/lib/perl/5.8.4 -dman1dir=/usr/share/man/man1 -dman3dir=/usr/share/man/man3 -dsiteman1dir=/usr/local/man/man1 -dsiteman3dir=/usr/local/man/man3 -dman1ext=1 -dman3ext=3perl -dpager=/usr/bin/sensible-pager -uafs -ud_csh -uusesfio -uusenm -duseshrplib -dlibperl=libperl.so.5.8.4 -dd_dosuid -des'   hint=recommended\, useposix=true\, d_sigaction=define   usethreads=define use5005threads=undef useithreads=define usemultiplicity=define   useperlio=define d_sfio=undef uselargefiles=define usesocks=undef   use64bitint=undef use64bitall=undef uselongdouble=undef   usemymalloc=n\, bincompat5005=undef   compiler​:   cc='cc'\, ccflags ='-d_reentrant -d_gnu_source -dthreads_have_pids -ddebian -fno-strict-aliasing -i/usr/local/include -d_largefile_source -d_file_offset_bits=64'\,   optimize='-o2'\,   cppflags='-d_reentrant -d_gnu_source -dthreads_have_pids -ddebian -fno-strict-aliasing -i/usr/local/include'   ccversion=''\, gccversion='3.3.5 (debian 1​:3.3.5-8)'\, gccosandvers=''   intsize=4\, longsize=4\, ptrsize=4\, doublesize=8\, byteorder=1234   d_longlong=define\, longlongsize=8\, d_longdbl=define\, longdblsize=12   ivtype='long'\, ivsize=4\, nvtype='double'\, nvsize=8\, off_t='off_t'\, lseeksize=8   alignbytes=4\, prototype=define   linker and libraries​:   ld='cc'\, ldflags =' -l/usr/local/lib'   libpth=/usr/local/lib /lib /usr/lib   libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt   perllibs=-ldl -lm -lpthread -lc -lcrypt   libc=/lib/libc-2.3.2.so\, so=so\, useshrplib=true\, libperl=libperl.so.5.8.4   gnulibc_version='2.3.2'   dynamic linking​:   dlsrc=dl_dlopen.xs\, dlext=so\, d_dlsymun=undef\, ccdlflags='-wl\,-e'   cccdlflags='-fpic'\, lddlflags='-shared -l/usr/local/lib'

locally applied patches​:  


@​inc for perl v5.8.4​:   /usr/local/sw/perl   /etc/perl   /usr/local/lib/perl/5.8.4   /usr/local/share/perl/5.8.4   /usr/lib/perl5   /usr/share/perl5   /usr/lib/perl/5.8   /usr/share/perl/5.8   /usr/local/lib/site_perl   .


environment for perl v5.8.4​:   home=/home/stefan   lang (unset)   language (unset)   ld_library_path=/home/stefan/physics/galaxyanalysis/lib​:/usr/local/sw/root-i686/lib​:/usr/local/sw/intel_cc_80/lib   logdir (unset)   path=/home/stefan/physics/galaxyanalysis/test​:/usr/local/sw/bin​:/usr/local/sw/bin-i686​:/usr/local/bin​:/bin​:/usr/bin​:/usr/x11r6/bin​:/home/stefan/bin​:.​:/usr/local/sw/root-i686/bin​:/usr/local/sw/intel_cc_80/bin   perl5lib=/usr/local/sw/perl   perl_badlang (unset)   shell=/usr/bin/tcsh

p5pRT commented 19 years ago

From @nwc10

On Sat\, Sep 10\, 2005 at 01​:58​:33PM -0700\, sgromoll @​ grad. physics. sunysb. edu wrote​:

When perl crashes it gives one of the following errors\, depending on the platform​:

== panic​: destruct destroyed thread 0@​a\uffffl0@​l0@​ during global destruction. Segmentation fault

I get this. So I an it under valgrind. I got something stranger​:

$ PERL_CORE=1 valgrind --db-attach=yes --db-command='gdb %f %p' --num-callers=20 --tool=memcheck ./perl -Ilib bug37133==22320== Memcheck\, a memory error detector for x86-linux.==22320== Copyright (C) 2002-2004\, and GNU GPL'd\, by Julian Seward.==22320== Using valgrind-2.1.1\, a program supervision framework for x86-linux.==22320== Copyright (C) 2000-2004\, and GNU GPL'd\, by Julian Seward.==22320== For more details\, rerun with​: -v==22320== vg_alloc_ThreadState​: no free slots available Increase VG_N_THREADS\, rebuild and try again.

valgrind​: the `impossible' happened​:   VG_N_THREADS is too low Basic block ctr is approximately 123300000 ==22320== at 0xB802A070​: vgPlain_core_panic (in /usr/lib/valgrind/stage2) ==22320== by 0xB802A06F​: panic (in /usr/lib/valgrind/stage2) ==22320== by 0xB802A084​: vgPlain_core_panic (in /usr/lib/valgrind/stage2) ==22320== by 0xB800CB7D​: vg_alloc_ThreadState (in /usr/lib/valgrind/stage2) ==22320== by 0xB800F514​: do__apply_in_new_thread (in /usr/lib/valgrind/stage2) ==22320== by 0xB8011A03​: do_client_request (in /usr/lib/valgrind/stage2) ==22320== by 0xB800DF33​: vgPlain_scheduler (in /usr/lib/valgrind/stage2) ==22320== by 0xB802567B​: main (in /usr/lib/valgrind/stage2)

sched status​:

Thread 1​: status = Runnable\, associated_mx = 0x0\, associated_cv = 0x0 ==22320== at 0x3C0A3061​: pthread_create (vg_libpthread.c​:927) ==22320== by 0x3C0260D4​: Perl_ithread_create (threads.xs​:521) ==22320== by 0x3C0268F3​: XS_threads_new (threads.xs​:686) ==22320== by 0x80FA247​: Perl_pp_entersub (pp_hot.c​:2788) ==22320== by 0x80D1C7A​: Perl_runops_debug (dump.c​:1597) ==22320== by 0x80673A4​: S_run_body (perl.c​:2274) ==22320== by 0x8066DAA​: perl_run (perl.c​:2201) ==22320== by 0x806061D​: main (perlmain.c​:103)

Thread 2​: status = WaitJoiner\, associated_mx = 0x0\, associated_cv = 0x0 ==22320== at 0x3C0A2BD0​: thread_exit_wrapper (vg_libpthread.c​:732) ==22320== by 0x3C0A2D7E​: thread_wrapper (vg_libpthread.c​:838) ==22320== by 0xB800F4C8​: do__quit (in /usr/lib/valgrind/stage2)

Thread 3​: status = WaitMX\, associated_mx = 0x3C02946C\, associated_cv = 0x0 ==22320== at 0x3C0A38AB​: pthread_mutex_lock (vg_libpthread.c​:1160) ==22320== by 0x3C025819​: Perl_ithread_run (threads.xs​:326) ==22320== by 0x3C0A2D78​: thread_wrapper (vg_libpthread.c​:837) ==22320== by 0xB800F4C8​: do__quit (in /usr/lib/valgrind/stage2)

Thread 4​: status = WaitMX\, associated_mx = 0x3C02946C\, associated_cv = 0x0 ==22320== at 0x3C0A38AB​: pthread_mutex_lock (vg_libpthread.c​:1160) ==22320== by 0x3C025819​: Perl_ithread_run (threads.xs​:326) ==22320== by 0x3C0A2D78​: thread_wrapper (vg_libpthread.c​:837) ==22320== by 0xB800F4C8​: do__quit (in /usr/lib/valgrind/stage2)

Thread 5​: status = WaitMX\, associated_mx = 0x3C02946C\, associated_cv = 0x0 ==22320== at 0x3C0A38AB​: pthread_mutex_lock (vg_libpthread.c​:1160) ==22320== by 0x3C0249AD​: Perl_ithread_destruct (threads.xs​:105) ==22320== by 0x3C0257CB​: Perl_ithread_run (threads.xs​:322) ==22320== by 0x3C0A2D78​: thread_wrapper (vg_libpthread.c​:837) ==22320== by 0xB800F4C8​: do__quit (in /usr/lib/valgrind/stage2)

[and so on]

Thread 99​: status = WaitMX\, associated_mx = 0x3C02946C\, associated_cv = 0x0 ==22320== at 0x3C0A38AB​: pthread_mutex_lock (vg_libpthread.c​:1160) ==22320== by 0x3C0249AD​: Perl_ithread_destruct (threads.xs​:105) ==22320== by 0x3C0257CB​: Perl_ithread_run (threads.xs​:322) ==22320== by 0x3C0A2D78​: thread_wrapper (vg_libpthread.c​:837) ==22320== by 0xB800F4C8​: do__quit (in /usr/lib/valgrind/stage2)

Note​: see also the FAQ.txt in the source distribution. It contains workarounds to several common problems.

If that doesn't help\, please report this bug to​: valgrind.kde.org

In the bug report\, send all the above text\, the valgrind version\, and what Linux distro you are using. Thanks.

Am I right in thinking that this suggests that the threads implementation has a deadlock of some sort here?

Even taking out the "obvious" deadlock doesn't make the bug go away​:

==== //depot/perl/ext/threads/threads.xs#86 - /home/nick/p4perl/perl/ext/threads/threads.xs ====

Inline Patch ```diff --- /tmp/tmp.23174.0 2005-09-12 12:07:37.000000000 +0100 +++ /home/nick/p4perl/perl/ext/threads/threads.xs 2005-09-12 12:03:23.000000000 +0100 @@ -96,6 +96,7 @@ Perl_ithread_destruct (pTHX_ ithread* th PerlInterpreter *freeperl = NULL; MUTEX_LOCK(&thread->mutex); if (!thread->next) { + MUTEX_UNLOCK(&thread->mutex); Perl_croak(aTHX_ "panic: destruct destroyed thread %p (%s)",thread, why); } if (thread->count != 0) { ```

The alternative crash mode I can create is:

(gdb) p thread->prev $3 = (struct ithread_s *) 0x0

which unsurprisingly causes this to SEGV​:

  thread->next->prev = thread->prev;

So\, also\, somehow it's looking like a thread can get freed up without correctly tidying up the linked list.

Thoughts?

Nicholas Clark

p5pRT commented 19 years ago

The RT System itself - Status changed from 'new' to 'open'

p5pRT commented 19 years ago

From @smpeters

[nicholas - Mon Sep 12 04​:07​:05 2005]​:

On Sat\, Sep 10\, 2005 at 01​:58​:33PM -0700\, sgromoll @​ grad. physics. sunysb. edu wrote​:

When perl crashes it gives one of the following errors\, depending on the platform​:

== panic​: destruct destroyed thread 0@​a\uffffl0@​l0@​ during global destruction. Segmentation fault

I get this. So I an it under valgrind. I got something stranger​:

$ PERL_CORE=1 valgrind --db-attach=yes --db-command='gdb %f %p' --num-callers=20 --tool=memcheck ./perl -Ilib bug37133==22320== Memcheck\, a memory error detector for x86-linux.==22320== Copyright (C) 2002-2004\, and GNU GPL'd\, by Julian Seward.==22320== Using valgrind-2.1.1\, a program supervision framework for x86- linux.==22320== Copyright (C) 2000-2004\, and GNU GPL'd\, by Julian Seward.==22320== For more details\, rerun with​: -v==22320== vg_alloc_ThreadState​: no free slots available Increase VG_N_THREADS\, rebuild and try again.

valgrind​: the `impossible' happened​: VG_N_THREADS is too low Basic block ctr is approximately 123300000 ==22320== at 0xB802A070​: vgPlain_core_panic (in /usr/lib/valgrind/stage2) ==22320== by 0xB802A06F​: panic (in /usr/lib/valgrind/stage2) ==22320== by 0xB802A084​: vgPlain_core_panic (in /usr/lib/valgrind/stage2) ==22320== by 0xB800CB7D​: vg_alloc_ThreadState (in /usr/lib/valgrind/stage2) ==22320== by 0xB800F514​: do__apply_in_new_thread (in /usr/lib/valgrind/stage2) ==22320== by 0xB8011A03​: do_client_request (in /usr/lib/valgrind/stage2) ==22320== by 0xB800DF33​: vgPlain_scheduler (in /usr/lib/valgrind/stage2) ==22320== by 0xB802567B​: main (in /usr/lib/valgrind/stage2)

sched status​:

Thread 1​: status = Runnable\, associated_mx = 0x0\, associated_cv = 0x0 ==22320== at 0x3C0A3061​: pthread_create (vg_libpthread.c​:927) ==22320== by 0x3C0260D4​: Perl_ithread_create (threads.xs​:521) ==22320== by 0x3C0268F3​: XS_threads_new (threads.xs​:686) ==22320== by 0x80FA247​: Perl_pp_entersub (pp_hot.c​:2788) ==22320== by 0x80D1C7A​: Perl_runops_debug (dump.c​:1597) ==22320== by 0x80673A4​: S_run_body (perl.c​:2274) ==22320== by 0x8066DAA​: perl_run (perl.c​:2201) ==22320== by 0x806061D​: main (perlmain.c​:103)

Thread 2​: status = WaitJoiner\, associated_mx = 0x0\, associated_cv = 0x0 ==22320== at 0x3C0A2BD0​: thread_exit_wrapper (vg_libpthread.c​:732) ==22320== by 0x3C0A2D7E​: thread_wrapper (vg_libpthread.c​:838) ==22320== by 0xB800F4C8​: do__quit (in /usr/lib/valgrind/stage2)

Thread 3​: status = WaitMX\, associated_mx = 0x3C02946C\, associated_cv = 0x0 ==22320== at 0x3C0A38AB​: pthread_mutex_lock (vg_libpthread.c​:1160) ==22320== by 0x3C025819​: Perl_ithread_run (threads.xs​:326) ==22320== by 0x3C0A2D78​: thread_wrapper (vg_libpthread.c​:837) ==22320== by 0xB800F4C8​: do__quit (in /usr/lib/valgrind/stage2)

Thread 4​: status = WaitMX\, associated_mx = 0x3C02946C\, associated_cv = 0x0 ==22320== at 0x3C0A38AB​: pthread_mutex_lock (vg_libpthread.c​:1160) ==22320== by 0x3C025819​: Perl_ithread_run (threads.xs​:326) ==22320== by 0x3C0A2D78​: thread_wrapper (vg_libpthread.c​:837) ==22320== by 0xB800F4C8​: do__quit (in /usr/lib/valgrind/stage2)

Thread 5​: status = WaitMX\, associated_mx = 0x3C02946C\, associated_cv = 0x0 ==22320== at 0x3C0A38AB​: pthread_mutex_lock (vg_libpthread.c​:1160) ==22320== by 0x3C0249AD​: Perl_ithread_destruct (threads.xs​:105) ==22320== by 0x3C0257CB​: Perl_ithread_run (threads.xs​:322) ==22320== by 0x3C0A2D78​: thread_wrapper (vg_libpthread.c​:837) ==22320== by 0xB800F4C8​: do__quit (in /usr/lib/valgrind/stage2)

[and so on]

Thread 99​: status = WaitMX\, associated_mx = 0x3C02946C\, associated_cv = 0x0 ==22320== at 0x3C0A38AB​: pthread_mutex_lock (vg_libpthread.c​:1160) ==22320== by 0x3C0249AD​: Perl_ithread_destruct (threads.xs​:105) ==22320== by 0x3C0257CB​: Perl_ithread_run (threads.xs​:322) ==22320== by 0x3C0A2D78​: thread_wrapper (vg_libpthread.c​:837) ==22320== by 0xB800F4C8​: do__quit (in /usr/lib/valgrind/stage2)

Note​: see also the FAQ.txt in the source distribution. It contains workarounds to several common problems.

If that doesn't help\, please report this bug to​: valgrind.kde.org

In the bug report\, send all the above text\, the valgrind version\, and what Linux distro you are using. Thanks.

Am I right in thinking that this suggests that the threads implementation has a deadlock of some sort here?

Even taking out the "obvious" deadlock doesn't make the bug go away​:

==== //depot/perl/ext/threads/threads.xs#86 - /home/nick/p4perl/perl/ext/threads/threads.xs ==== --- /tmp/tmp.23174.0 2005-09-12 12​:07​:37.000000000 +0100 +++ /home/nick/p4perl/perl/ext/threads/threads.xs 2005-09-12 12​:03​:23.000000000 +0100 @​@​ -96\,6 +96\,7 @​@​ Perl_ithread_destruct (pTHX_ ithread* th PerlInterpreter *freeperl = NULL; MUTEX_LOCK(&thread->mutex); if (!thread->next) { + MUTEX_UNLOCK(&thread->mutex); Perl_croak(aTHX_ "panic​: destruct destroyed thread %p (%s)"\,thread\, why); } if (thread->count != 0) {

The alternative crash mode I can create is​:

(gdb) p thread->prev $3 = (struct ithread_s *) 0x0

which unsurprisingly causes this to SEGV​:

    thread\->next\->prev = thread\->prev;

So\, also\, somehow it's looking like a thread can get freed up without correctly tidying up the linked list.

The patch above was applied as change #25442.

p5pRT commented 19 years ago

@smpeters - Status changed from 'open' to 'resolved'