search

epics-base / pva2pva

PV Access gateway/proxy and EPICS Process Database integration
https://epics-base.github.io/pva2pva/
Other
4 stars 13 forks source link

softIocPVA dies sending high-volume monitors #10

Closed anjohnson closed 5 years ago

anjohnson commented 6 years ago

Tom Fors found a crash while doing some performance testing.

spam.db is

record(calc, "X1") {
    field(INPA, "X1.VAL CP")
    field(CALC, "A+1")
}

There are two separate errors reported a few seconds apart, not sure if the second is a consequence of the first, I don't always see the exception before the seg-fault. Here's how to trigger it:

tux% bin/linux-x86_64/softIocPVA -d ../db/spam.db 
Starting iocInit
############################################################################
## EPICS R7.0.1-rc1-DEV
## EPICS Base built Dec  8 2017
############################################################################
iocRun: All initialization complete
epics>

This runs quite happily, but after starting pvget -m X1 the IOC eventually dies like this:

2017-12-12T16:46:54.684 an exception caught while processing a send message: std::bad_alloc at ../../src/remote/codec.cpp:902.
Segmentation fault

Running the IOC under GDB it takes a bit longer and the errors are clearly separated in time:

iocRun: All initialization complete
[New Thread 0x7fffe6a73700 (LWP 26039)]
[New Thread 0x7fffe6af4700 (LWP 26038)]
[New Thread 0x7fffe6972700 (LWP 26040)]
[New Thread 0x7fffe6871700 (LWP 26041)]
2017-12-12T16:47:49.872 Using dynamically assigned TCP port 44243.
[New Thread 0x7fffe6770700 (LWP 26042)]
[New Thread 0x7fffe66ef700 (LWP 26043)]
[New Thread 0x7fffe666e700 (LWP 26044)]
[New Thread 0x7fffe65ed700 (LWP 26045)]
epics> [New Thread 0x7fffe62ea700 (LWP 26055)]
[New Thread 0x7fffe6269700 (LWP 26056)]
2017-12-12T16:48:33.004 an exception caught while processing a send message: std::bad_alloc at ../../src/remote/codec.cpp:902.
[Thread 0x7fffe6269700 (LWP 26056) exited]
[Thread 0x7fffe62ea700 (LWP 26055) exited]
[New Thread 0x7fffe62ea700 (LWP 26090)]
[New Thread 0x7fffe6269700 (LWP 26091)]
*** Error in `/local/anj/base-7.0/bin/linux-x86_64/softIocPVA': double free or corruption (fasttop): 0x00007fffac000df0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7c619)[0x7ffff5baa619]
/lib64/libstdc++.so.6(_ZNSs6assignERKSs+0x9e)[0x7ffff64c809e]
/local/anj/base-7.0/lib/linux-x86_64/libqsrv.so.1.0(+0x140d5)[0x7ffff7b9d0d5]
/local/anj/base-7.0/lib/linux-x86_64/libqsrv.so.1.0(+0x143c4)[0x7ffff7b9d3c4]
/local/anj/base-7.0/lib/linux-x86_64/libqsrv.so.1.0(+0x1b30e)[0x7ffff7ba430e]
/local/anj/base-7.0/lib/linux-x86_64/libqsrv.so.1.0(+0x30715)[0x7ffff7bb9715]
/local/anj/base-7.0/lib/linux-x86_64/libdbCore.so.3.17.0(+0x34244)[0x7ffff6c05244]
/local/anj/base-7.0/lib/linux-x86_64/libCom.so.3.17.0(+0x383cc)[0x7ffff67493cc]
/lib64/libpthread.so.0(+0x7e25)[0x7ffff5919e25]
/lib64/libc.so.6(clone+0x6d)[0x7ffff5c2634d]
======= Memory map: ========
…
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe6972700 (LWP 26040)]
0x00007ffff5b631f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-196.el7_4.2.x86_64 libgcc-4.8.5-16.el7_4.1.x86_64 libstdc++-4.8.5-16.el7_4.1.x86_64 ncurses-libs-5.9-14.20130511.el7_4.x86_64 readline-6.2-10.el7.x86_64
(gdb) bt
#0  0x00007ffff5b631f7 in raise () from /lib64/libc.so.6
#1  0x00007ffff5b648e8 in abort () from /lib64/libc.so.6
#2  0x00007ffff5ba2f47 in __libc_message () from /lib64/libc.so.6
#3  0x00007ffff5baa619 in _int_free () from /lib64/libc.so.6
#4  0x00007ffff64c809e in std::string::assign(std::string const&) ()
   from /lib64/libstdc++.so.6
#5  0x00007ffff7b9d0d5 in operator= (__str="NO_ALARM", this=0x7fff98001470)
    at /usr/include/c++/4.8.2/bits/basic_string.h:547
#6  store (val="NO_ALARM", this=0x7fff98001470)
    at /local/anj/base-7.0/include/pv/pvData.h:389
#7  put (v="NO_ALARM", this=0x7fff98001410)
    at /local/anj/base-7.0/include/pv/pvData.h:426
#8  (anonymous namespace)::mapStatus (code=0, status=0x7fff98001370, 
    message=0x7fff98001410) at ../pvif.cpp:226
#9  0x00007ffff7b9d3c4 in (anonymous namespace)::putTime (pv=..., 
    dbe=dbe@entry=5, pfl=pfl@entry=0x7ffff7e1f8d8) at ../pvif.cpp:288
#10 0x00007ffff7ba430e in putAll<(anonymous namespace)::pvScalar, (anonymous namespace)::metaDOUBLE> (pfl=0x7ffff7e1f8d8, dbe=5, pv=...) at ../pvif.cpp:528
#11 (anonymous namespace)::PVIFScalarNumeric<(anonymous namespace)::pvScalar, (anonymous namespace)::metaDOUBLE>::put (this=0x7fff980026c0, mask=..., dbe=5, 
    pfl=0x7ffff7e1f8d8) at ../pvif.cpp:595
#12 0x00007ffff7bb9715 in pdb_single_event (user_arg=0x7fff98000e60, 
    chan=<optimized out>, eventsRemaining=<optimized out>, pfl=0x7ffff7e1f8d8)
    at ../pdbsingle.cpp:53
#13 0x00007ffff6c05244 in event_read (ev_que=0x7fffd802d4b8)
    at ../../../src/ioc/db/dbEvent.c:970
#14 event_task (pParm=0x7fffd802d4b8) at ../../../src/ioc/db/dbEvent.c:1049
#15 0x00007ffff67493cc in start_routine (arg=0x6d7840)
    at ../../src/osi/os/posix/osdThread.c:403
#16 0x00007ffff5919e25 in start_thread () from /lib64/libpthread.so.0
#17 0x00007ffff5c2634d in clone () from /lib64/libc.so.6
anjohnson commented 6 years ago

Tom says he doesn't see the crash at all under Ubuntu 16.04, and I can't replicate it on my 32-bit RHEL-6.9 box either. My results above and Tom's crash were from RHEL-7.4 (in his case a VM, mine was native, both 64-bit).

RHEL-7.4 comes with

gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)

RHEL-6.9 has

gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18)

While the linux-x86 IOC hasn't died, stopping the pvget client (which was running on my RHEL-7.4 machine) with ^C did cause this on the IOC, but it kept on running:

Unhandled exception in pdb_single_event(): epicsMutex::invalidMutex()

Running the client on a different box doesn't make any difference, the RHEL-7.4 IOC still dies. Thus we might be seeing a RHEL bug.

anjohnson commented 6 years ago

Not seeing it running the IOC on macOS either, although it does report the same Unhandled exception on killing the client.

anjohnson commented 6 years ago

Tested on RHEL-7.2, failed with a slightly different error message:

apsgpfs01% bin/linux-x86_64/softIocPVA -d ~/phoebus/db/spam.db 
Starting iocInit
############################################################################
## EPICS R7.0.1-rc1-DEV
## EPICS Base built Dec 13 2017
############################################################################
iocRun: All initialization complete
epics> 2017-12-13T11:19:22.287 an exception caught while processing a send message: std::bad_alloc at ../../src/remote/codec.cpp:902.
*** Error in `bin/linux-x86_64/softIocPVA': double free or corruption (fasttop): 0x00007f1064000950 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7d053)[0x7f10ad888053]
/lib64/libstdc++.so.6(_ZNSs6assignERKSs+0x9e)[0x7f10ae1a403e]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libqsrv.so.1.0(+0x1ddc5)[0x7f10af909dc5]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libqsrv.so.1.0(+0x1e0b4)[0x7f10af90a0b4]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libqsrv.so.1.0(+0x2507e)[0x7f10af91107e]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libqsrv.so.1.0(+0x3a455)[0x7f10af926455]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libdbCore.so.3.17.0(+0x39a94)[0x7f10ae902a94]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libCom.so.3.17.0(+0x400ec)[0x7f10ae42e0ec]
/lib64/libpthread.so.0(+0x7dc5)[0x7f10ad5f6dc5]
/lib64/libc.so.6(clone+0x6d)[0x7f10ad901ced]

This wasn't run under gdb so it doesn't show the detailed back-trace, but the library list is identical to on RHEL-7.2. On this system gcc is

gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4)

anjohnson commented 6 years ago

Also crashed on 64-bit RHEL-6.9 with a similar glibc double-free error:

tux-vm-rhel6% bin/linux-x86_64/softIocPVA -d ~/db/spam.db 
Starting iocInit
############################################################################
## EPICS R7.0.1-rc1-DEV
## EPICS Base built Dec 13 2017
############################################################################
iocRun: All initialization complete
epics> *** glibc detected *** bin/linux-x86_64/softIocPVA: double free or corruption (fasttop): 0x00007f9574000950 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3ed2c75dee]
/lib64/libc.so.6[0x3ed2c78c3d]
/usr/lib64/libstdc++.so.6(_ZNSs6assignERKSs+0x85)[0x3ed7c9d565]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libqsrv.so.1.0(_ZN5epics6pvData13PVScalarValueISsE3putERKSs+0x81)[0x7f95c573ac51]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libqsrv.so.1.0(+0x12c1e)[0x7f95c5730c1e]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libqsrv.so.1.0(+0x12d54)[0x7f95c5730d54]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libqsrv.so.1.0(+0x194f1)[0x7f95c57374f1]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libqsrv.so.1.0(+0x2e931)[0x7f95c574c931]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libdbCore.so.3.17.0(+0x338f6)[0x7f95c47ae8f6]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libCom.so.3.17.0(+0x394dc)[0x7f95c43004dc]
/lib64/libpthread.so.0[0x3ed3807aa1]
/lib64/libc.so.6(clone+0x6d)[0x3ed2ce8bcd]

This reports the exact same gcc version I showed above for the 32-bit system which didn't crash (Red Hat 4.4.7-18).

I also ran that same 6.9-compiled binary on my 7.4 system and got more bad_alloc exceptions this time:

tux% bin/linux-x86_64/softIocPVA -d ~/db/spam.db 
Starting iocInit
############################################################################
## EPICS R7.0.1-rc1-DEV
## EPICS Base built Dec 13 2017
############################################################################
iocRun: All initialization complete
epics> 2017-12-13T11:43:13.457 an exception caught while processing a send message: std::bad_alloc at ../../src/remote/codec.cpp:902.
2017-12-13T11:43:14.179 an exception caught while processing a send message: std::bad_alloc at ../../src/remote/codec.cpp:902.
2017-12-13T11:43:14.637 an exception caught while processing a send message: std::bad_alloc at ../../src/remote/codec.cpp:902.
2017-12-13T11:43:18.320 an exception caught while processing a send message: std::bad_alloc at ../../src/remote/codec.cpp:902.
*** Error in `bin/linux-x86_64/softIocPVA': double free or corruption (fasttop): 0x00007f9c04000bc0 ***
mdavidsaver commented 6 years ago

I've not been able to replicate this w/ Debian 9/amd64. Can you run softIocPVA with valgrind? Triggering the double free with valgrind should give some useful info.

Also, can you ensure that you're building from scratch? As in remove all source and $(TOP)/../RELEASE.local then rebuild. I can't recall ever seeing std::bad_alloc except from a large std::vector, so my first thought is that this is some ABI mis-match.

anjohnson commented 6 years ago

I was able to trigger the segfault on my test build from the 7.0.1.1 tarfile, which definitely started from scratch, but I didn't get the double free message that time.

Under valgrind the count goes slower, but despite letting it count for well over 3 times as many loops I didn't get a segfault. Valgrind did report a few things when I finally shut down the IOC, although none of them seem obviously problematic. Here's the output:

epics> dbCa: chan_count = 1 at shutdown
==5425== Thread 21 PDB-event:
==5425== Invalid read of size 8
==5425==    at 0x62D601C: taskwdRemove (taskwd.c:221)
==5425==    by 0x5DEB442: event_task (dbEvent.c:1071)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425==  Address 0x7f00750 is 16 bytes inside a block of size 48 alloc'd
==5425==    at 0x4C2B975: calloc (vg_replace_malloc.c:711)
==5425==    by 0x62D5E4A: newNode (taskwd.c:406)
==5425==    by 0x62D5EDF: allocNode (taskwd.c:418)
==5425==    by 0x62D5EDF: taskwdInsert (taskwd.c:186)
==5425==    by 0x5DF1FA1: callbackTask (callback.c:160)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425== 
==5425== Invalid read of size 8
==5425==    at 0x62D6010: taskwdRemove (taskwd.c:237)
==5425==    by 0x5DEB442: event_task (dbEvent.c:1071)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425==  Address 0x7f00740 is 0 bytes inside a block of size 48 alloc'd
==5425==    at 0x4C2B975: calloc (vg_replace_malloc.c:711)
==5425==    by 0x62D5E4A: newNode (taskwd.c:406)
==5425==    by 0x62D5EDF: allocNode (taskwd.c:418)
==5425==    by 0x62D5EDF: taskwdInsert (taskwd.c:186)
==5425==    by 0x5DF1FA1: callbackTask (callback.c:160)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425== 
==5425== Invalid read of size 8
==5425==    at 0x62B678B: ellDelete (ellLib.c:79)
==5425==    by 0x62D6030: taskwdRemove (taskwd.c:222)
==5425==    by 0x5DEB442: event_task (dbEvent.c:1071)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425==  Address 0x7fb5a58 is 8 bytes inside a block of size 48 alloc'd
==5425==    at 0x4C2B975: calloc (vg_replace_malloc.c:711)
==5425==    by 0x62D5E4A: newNode (taskwd.c:406)
==5425==    by 0x62D5EDF: allocNode (taskwd.c:418)
==5425==    by 0x62D5EDF: taskwdInsert (taskwd.c:186)
==5425==    by 0x5DEB0BF: event_task (dbEvent.c:1017)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425== 
==5425== Invalid read of size 8
==5425==    at 0x62B6776: ellDelete (ellLib.c:86)
==5425==    by 0x62D6030: taskwdRemove (taskwd.c:222)
==5425==    by 0x5DEB442: event_task (dbEvent.c:1071)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425==  Address 0x7fb5a50 is 0 bytes inside a block of size 48 alloc'd
==5425==    at 0x4C2B975: calloc (vg_replace_malloc.c:711)
==5425==    by 0x62D5E4A: newNode (taskwd.c:406)
==5425==    by 0x62D5EDF: allocNode (taskwd.c:418)
==5425==    by 0x62D5EDF: taskwdInsert (taskwd.c:186)
==5425==    by 0x5DEB0BF: event_task (dbEvent.c:1017)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425== 
==5425== Invalid read of size 8
==5425==    at 0x62B6779: ellDelete (ellLib.c:86)
==5425==    by 0x62D6030: taskwdRemove (taskwd.c:222)
==5425==    by 0x5DEB442: event_task (dbEvent.c:1071)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425==  Address 0x7fb5a58 is 8 bytes inside a block of size 48 alloc'd
==5425==    at 0x4C2B975: calloc (vg_replace_malloc.c:711)
==5425==    by 0x62D5E4A: newNode (taskwd.c:406)
==5425==    by 0x62D5EDF: allocNode (taskwd.c:418)
==5425==    by 0x62D5EDF: taskwdInsert (taskwd.c:186)
==5425==    by 0x5DEB0BF: event_task (dbEvent.c:1017)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425== 
==5425== Invalid write of size 8
==5425==    at 0x62B677D: ellDelete (ellLib.c:86)
==5425==    by 0x62D6030: taskwdRemove (taskwd.c:222)
==5425==    by 0x5DEB442: event_task (dbEvent.c:1071)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425==  Address 0x7f90230 is 0 bytes inside a block of size 48 alloc'd
==5425==    at 0x4C2B975: calloc (vg_replace_malloc.c:711)
==5425==    by 0x62D5E4A: newNode (taskwd.c:406)
==5425==    by 0x62D5EDF: allocNode (taskwd.c:418)
==5425==    by 0x62D5EDF: taskwdInsert (taskwd.c:186)
==5425==    by 0x5DEB0BF: event_task (dbEvent.c:1017)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425== 
==5425== Illegal memory pool address
==5425==    at 0x62D608E: freeNode (taskwd.c:425)
==5425==    by 0x62D608E: taskwdRemove (taskwd.c:224)
==5425==    by 0x5DEB442: event_task (dbEvent.c:1071)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425==  Address 0x64ff1d0 is 0 bytes inside data symbol "fList"
==5425== 
==5425== Illegal memory pool address
==5425==    at 0x62D60E5: freeNode (taskwd.c:427)
==5425==    by 0x62D60E5: taskwdRemove (taskwd.c:224)
==5425==    by 0x5DEB442: event_task (dbEvent.c:1071)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425==  Address 0x64ff1d0 is 0 bytes inside data symbol "fList"
==5425== 
==5425== Invalid write of size 8
==5425==    at 0x62B66C4: ellAdd (ellLib.c:27)
==5425==    by 0x62D6127: freeNode (taskwd.c:428)
==5425==    by 0x62D6127: taskwdRemove (taskwd.c:224)
==5425==    by 0x5DEB442: event_task (dbEvent.c:1071)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425==  Address 0x7fb5a50 is 0 bytes inside a block of size 48 alloc'd
==5425==    at 0x4C2B975: calloc (vg_replace_malloc.c:711)
==5425==    by 0x62D5E4A: newNode (taskwd.c:406)
==5425==    by 0x62D5EDF: allocNode (taskwd.c:418)
==5425==    by 0x62D5EDF: taskwdInsert (taskwd.c:186)
==5425==    by 0x5DEB0BF: event_task (dbEvent.c:1017)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425== 
==5425== Invalid write of size 8
==5425==    at 0x62B66CB: ellAdd (ellLib.c:28)
==5425==    by 0x62D6127: freeNode (taskwd.c:428)
==5425==    by 0x62D6127: taskwdRemove (taskwd.c:224)
==5425==    by 0x5DEB442: event_task (dbEvent.c:1071)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425==  Address 0x7fb5a58 is 8 bytes inside a block of size 48 alloc'd
==5425==    at 0x4C2B975: calloc (vg_replace_malloc.c:711)
==5425==    by 0x62D5E4A: newNode (taskwd.c:406)
==5425==    by 0x62D5EDF: allocNode (taskwd.c:418)
==5425==    by 0x62D5EDF: taskwdInsert (taskwd.c:186)
==5425==    by 0x5DEB0BF: event_task (dbEvent.c:1017)
==5425==    by 0x62CD3CB: start_routine (osdThread.c:403)
==5425==    by 0x70EAE24: start_thread (in /usr/lib64/libpthread-2.17.so)
==5425==    by 0x6E1834C: clone (in /usr/lib64/libc-2.17.so)
==5425== 
==5425== 
==5425== HEAP SUMMARY:
==5425==     in use at exit: 1,131,797 bytes in 12,464 blocks
==5425==   total heap usage: 68,907,242 allocs, 14,439,043 frees, 2,629,460,283 bytes allocated
==5425== 
==5425== LEAK SUMMARY:
==5425==    definitely lost: 842 bytes in 23 blocks
==5425==    indirectly lost: 0 bytes in 0 blocks
==5425==      possibly lost: 12,538 bytes in 20 blocks
==5425==    still reachable: 948,641 bytes in 14,864 blocks
==5425==                       of which reachable via heuristic:
==5425==                         stdstring          : 836 bytes in 21 blocks
==5425==         suppressed: 0 bytes in 0 blocks
==5425== Rerun with --leak-check=full to see details of leaked memory
==5425== 
==5425== For counts of detected and suppressed errors, rerun with: -v
==5425== Use --track-origins=yes to see where uninitialised values come from
==5425== ERROR SUMMARY: 49 errors from 11 contexts (suppressed: 0 from 0)

I ran it again for a few seconds and shut down as before, and there was little difference in the valgrind output, here's the final summary it gave for comparison:

==13115== HEAP SUMMARY:
==13115==     in use at exit: 948,425 bytes in 12,411 blocks
==13115==   total heap usage: 122,325 allocs, 3,592 frees, 7,209,735 bytes allocated
==13115== 
==13115== LEAK SUMMARY:
==13115==    definitely lost: 738 bytes in 21 blocks
==13115==    indirectly lost: 0 bytes in 0 blocks
==13115==      possibly lost: 11,962 bytes in 19 blocks
==13115==    still reachable: 763,733 bytes in 14,814 blocks
==13115==                       of which reachable via heuristic:
==13115==                         stdstring          : 836 bytes in 21 blocks
==13115==         suppressed: 0 bytes in 0 blocks
==13115== Rerun with --leak-check=full to see details of leaked memory
==13115== 
==13115== For counts of detected and suppressed errors, rerun with: -v
==13115== ERROR SUMMARY: 37 errors from 10 contexts (suppressed: 0 from 0)

I would put this low on the priority list for now.

mdavidsaver commented 6 years ago

Right, so you've reminded me that I meant to fix the valgrind hints in taskwd.c as there is a problem. These are false positives.

So we'll have to go with plan B, which is analyzing a core dump w/ gdb.

mdavidsaver commented 6 years ago

I was able to trigger the segfault ...

If you can trigger this again in GDB, run "thread apply all backtrace" and attach the resulting output. I'm wondering if this is a case of insufficient locking of concurrent writers, though I can't immediately think of what the other writer would be.

anjohnson commented 6 years ago 
tux% gdb bin/linux-x86_64/softIocPVA
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/phoebus3/ANJ/epics/base/7.0/bin/linux-x86_64/softIocPVA...done.
(gdb) run -d ~/db/spam.db
Starting program: /home/phoebus3/ANJ/epics/base/7.0/bin/linux-x86_64/softIocPVA -d ~/db/spam.db
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff4e7f700 (LWP 10002)]
[Thread 0x7ffff4e7f700 (LWP 10002) exited]
Detaching after fork from child process 10003.
[New Thread 0x7ffff7fa3700 (LWP 10004)]
Starting iocInit
############################################################################
## EPICS R7.0.1.2-DEV
## EPICS Base built Jan  4 2018
############################################################################
[New Thread 0x7ffff7f22700 (LWP 10005)]
[New Thread 0x7ffff467e700 (LWP 10006)]
[New Thread 0x7ffff457d700 (LWP 10007)]
[New Thread 0x7ffff437c700 (LWP 10008)]
[New Thread 0x7fffe7fff700 (LWP 10009)]
[New Thread 0x7fffe7dfe700 (LWP 10010)]
[New Thread 0x7fffe7bfd700 (LWP 10011)]
[New Thread 0x7fffe79fc700 (LWP 10012)]
[New Thread 0x7fffe77fb700 (LWP 10013)]
[New Thread 0x7fffe75fa700 (LWP 10014)]
[New Thread 0x7fffe73f9700 (LWP 10015)]
[New Thread 0x7fffe71f8700 (LWP 10016)]
[New Thread 0x7fffe6ff7700 (LWP 10017)]
[New Thread 0x7fffe6df6700 (LWP 10018)]
[New Thread 0x7ffff417b700 (LWP 10020)]
[New Thread 0x7fffe6bf5700 (LWP 10021)]
[New Thread 0x7ffff7ea1700 (LWP 10022)]
iocRun: All initialization complete
[New Thread 0x7fffe6af4700 (LWP 10023)]
[New Thread 0x7fffe6a73700 (LWP 10024)]
[New Thread 0x7fffe6972700 (LWP 10025)]
[New Thread 0x7fffe6871700 (LWP 10026)]
2018-01-30T10:10:49.390 Using dynamically assigned TCP port 33618.
[New Thread 0x7fffe6770700 (LWP 10027)]
[New Thread 0x7fffe66ef700 (LWP 10028)]
[New Thread 0x7fffe666e700 (LWP 10029)]
[New Thread 0x7fffe65ed700 (LWP 10030)]
[New Thread 0x7fffe62ea700 (LWP 10032)]
[New Thread 0x7fffe6269700 (LWP 10034)]
epics> *** Error in `/home/phoebus3/ANJ/epics/base/7.0/bin/linux-x86_64/softIocPVA': double free or corruption (fasttop): 0x00007fffac000950 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7c619)[0x7ffff5bbb619]
/lib64/libstdc++.so.6(_ZNSs6assignERKSs+0x9e)[0x7ffff64d909e]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvData.so.7.0.1(+0xb46bd)[0x7ffff715f6bd]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvData.so.7.0.1(_ZN5epics6pvData13PVScalarValueISsE13copyUncheckedERKNS0_8PVScalarE+0x40)[0x7ffff715f750]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvData.so.7.0.1(_ZN5epics6pvData7PVField13copyUncheckedERKS1_+0x59)[0x7ffff713c489]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvData.so.7.0.1(_ZN5epics6pvData11PVStructure13copyUncheckedERKS1_+0x68)[0x7ffff713e5c8]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvData.so.7.0.1(_ZN5epics6pvData7PVField13copyUncheckedERKS1_+0x130)[0x7ffff713c560]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvData.so.7.0.1(_ZN5epics6pvData11PVStructure13copyUncheckedERKS1_+0x68)[0x7ffff713e5c8]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libqsrv.so.1.0(+0x34219)[0x7ffff7bbd219]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libqsrv.so.1.0(+0x30e94)[0x7ffff7bb9e94]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libqsrv.so.1.0(+0x3400a)[0x7ffff7bbd00a]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvAccess.so.6.0.1(+0xc11c2)[0x7ffff74c91c2]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvAccess.so.6.0.1(_ZN5epics8pvAccess6detail13AbstractCodec13processSenderERKNSt3tr110shared_ptrINS0_15TransportSenderEEE+0x84)[0x7ffff7488fe4]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvAccess.so.6.0.1(_ZN5epics8pvAccess6detail13AbstractCodec16processSendQueueEv+0x9a)[0x7ffff748a02a]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvAccess.so.6.0.1(+0x823f0)[0x7ffff748a3f0]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libCom.so.3.17.0(+0x33039)[0x7ffff6755039]
/home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libCom.so.3.17.0(+0x3941c)[0x7ffff675b41c]
/lib64/libpthread.so.0(+0x7e25)[0x7ffff592ae25]
/lib64/libc.so.6(clone+0x6d)[0x7ffff5c3734d]
======= Memory map: ========
00400000-00407000 r-xp 00000000 00:28 20950552                           /home/phoebus3/ANJ/epics/base/7.0/bin/linux-x86_64/softIocPVA
00606000-00607000 r--p 00006000 00:28 20950552                           /home/phoebus3/ANJ/epics/base/7.0/bin/linux-x86_64/softIocPVA
00607000-00608000 rw-p 00007000 00:28 20950552                           /home/phoebus3/ANJ/epics/base/7.0/bin/linux-x86_64/softIocPVA
00608000-0077c000 rw-p 00000000 00:00 0                                  [heap]
7fff91ad7000-7fff98000000 r--p 00000000 09:7f 201329009                  /usr/lib/locale/locale-archive
7fff98000000-7fff98021000 rw-p 00000000 00:00 0 
7fff98021000-7fff9c000000 ---p 00000000 00:00 0 
7fff9c000000-7fff9c021000 rw-p 00000000 00:00 0 
7fff9c021000-7fffa0000000 ---p 00000000 00:00 0 
7fffa0000000-7fffa0021000 rw-p 00000000 00:00 0 
7fffa0021000-7fffa4000000 ---p 00000000 00:00 0 
7fffa4000000-7fffa4021000 rw-p 00000000 00:00 0 
7fffa4021000-7fffa8000000 ---p 00000000 00:00 0 
7fffa8000000-7fffa8021000 rw-p 00000000 00:00 0 
7fffa8021000-7fffac000000 ---p 00000000 00:00 0 
7fffac000000-7fffac021000 rw-p 00000000 00:00 0 
7fffac021000-7fffb0000000 ---p 00000000 00:00 0 
7fffb0000000-7fffb0022000 rw-p 00000000 00:00 0 
7fffb0022000-7fffb4000000 ---p 00000000 00:00 0 
7fffb4000000-7fffb4021000 rw-p 00000000 00:00 0 
7fffb4021000-7fffb8000000 ---p 00000000 00:00 0 
7fffb8000000-7fffb8021000 rw-p 00000000 00:00 0 
7fffb8021000-7fffbc000000 ---p 00000000 00:00 0 
7fffbc000000-7fffbc021000 rw-p 00000000 00:00 0 
7fffbc021000-7fffc0000000 ---p 00000000 00:00 0 
7fffc0000000-7fffc0021000 rw-p 00000000 00:00 0 
7fffc0021000-7fffc4000000 ---p 00000000 00:00 0 
7fffc4000000-7fffc4021000 rw-p 00000000 00:00 0 
7fffc4021000-7fffc8000000 ---p 00000000 00:00 0 
7fffc8000000-7fffc8021000 rw-p 00000000 00:00 0 
7fffc8021000-7fffcc000000 ---p 00000000 00:00 0 
7fffcc000000-7fffcc021000 rw-p 00000000 00:00 0 
7fffcc021000-7fffd0000000 ---p 00000000 00:00 0 
7fffd0000000-7fffd0021000 rw-p 00000000 00:00 0 
7fffd0021000-7fffd4000000 ---p 00000000 00:00 0 
7fffd4000000-7fffd4021000 rw-p 00000000 00:00 0 
7fffd4021000-7fffd8000000 ---p 00000000 00:00 0 
7fffd8000000-7fffd803c000 rw-p 00000000 00:00 0 
7fffd803c000-7fffdc000000 ---p 00000000 00:00 0 
7fffdc000000-7fffdc021000 rw-p 00000000 00:00 0 
7fffdc021000-7fffe0000000 ---p 00000000 00:00 0 
7fffe0000000-7fffe0021000 rw-p 00000000 00:00 0 
7fffe0021000-7fffe4000000 ---p 00000000 00:00 0 
7fffe61e9000-7fffe61ea000 ---p 00000000 00:00 0 
7fffe61ea000-7fffe626a000 rw-p 00000000 00:00 0 
7fffe626a000-7fffe626b000 ---p 00000000 00:00 0 
7fffe626b000-7fffe656d000 rw-p 00000000 00:00 0 
7fffe656d000-7fffe656e000 ---p 00000000 00:00 0 
7fffe656e000-7fffe65ee000 rw-p 00000000 00:00 0 
7fffe65ee000-7fffe65ef000 ---p 00000000 00:00 0 
7fffe65ef000-7fffe666f000 rw-p 00000000 00:00 0 
7fffe666f000-7fffe6670000 ---p 00000000 00:00 0 
7fffe6670000-7fffe66f0000 rw-p 00000000 00:00 0 
7fffe66f0000-7fffe66f1000 ---p 00000000 00:00 0 
7fffe66f1000-7fffe6771000 rw-p 00000000 00:00 0 
7fffe6771000-7fffe6772000 ---p 00000000 00:00 0 
7fffe6772000-7fffe6872000 rw-p 00000000 00:00 0 
7fffe6872000-7fffe6873000 ---p 00000000 00:00 0 
7fffe6873000-7fffe6973000 rw-p 00000000 00:00 0 
7fffe6973000-7fffe6974000 ---p 00000000 00:00 0 
7fffe6974000-7fffe6a74000 rw-p 00000000 00:00 0 
7fffe6a74000-7fffe6a75000 ---p 00000000 00:00 0 
7fffe6a75000-7fffe6af5000 rw-p 00000000 00:00 0 
7fffe6af5000-7fffe6af6000 ---p 00000000 00:00 0 
7fffe6af6000-7fffe6bf6000 rw-p 00000000 00:00 0 
7fffe6bf6000-7fffe6bf7000 ---p 00000000 00:00 0 
7fffe6bf7000-7fffe6df7000 rw-p 00000000 00:00 0 
7fffe6df7000-7fffe6df8000 ---p 00000000 00:00 0 
7fffe6df8000-7fffe6ff8000 rw-p 00000000 00:00 0 
7fffe6ff8000-7fffe6ff9000 ---p 00000000 00:00 0 
7fffe6ff9000-7fffe71f9000 rw-p 00000000 00:00 0 
7fffe71f9000-7fffe71fa000 ---p 00000000 00:00 0 
7fffe71fa000-7fffe73fa000 rw-p 00000000 00:00 0 
7fffe73fa000-7fffe73fb000 ---p 00000000 00:00 0 
7fffe73fb000-7fffe75fb000 rw-p 00000000 00:00 0 
7fffe75fb000-7fffe75fc000 ---p 00000000 00:00 0 
7fffe75fc000-7fffe77fc000 rw-p 00000000 00:00 0 
7fffe77fc000-7fffe77fd000 ---p 00000000 00:00 0 
7fffe77fd000-7fffe79fd000 rw-p 00000000 00:00 0 
7fffe79fd000-7fffe79fe000 ---p 00000000 00:00 0 
7fffe79fe000-7fffe7bfe000 rw-p 00000000 00:00 0 
7fffe7bfe000-7fffe7bff000 ---p 00000000 00:00 0 
7fffe7bff000-7fffe7dff000 rw-p 00000000 00:00 0 
7fffe7dff000-7fffe7e00000 ---p 00000000 00:00 0 
7fffe7e00000-7fffe8000000 rw-p 00000000 00:00 0 
7fffe8000000-7fffe8021000 rw-p 00000000 00:00 0 
7fffe8021000-7fffec000000 ---p 00000000 00:00 0 
7fffec000000-7fffec021000 rw-p 00000000 00:00 0 
7fffec021000-7ffff0000000 ---p 00000000 00:00 0 
7ffff0000000-7ffff0021000 rw-p 00000000 00:00 0 
7ffff0021000-7ffff4000000 ---p 00000000 00:00 0 
7ffff407b000-7ffff407c000 ---p 00000000 00:00 0 
7ffff407c000-7ffff417c000 rw-p 00000000 00:00 0 
7ffff417c000-7ffff417d000 ---p 00000000 00:00 0 
7ffff417d000-7ffff437d000 rw-p 00000000 00:00 0 
7ffff437d000-7ffff437e000 ---p 00000000 00:00 0 
7ffff437e000-7ffff457e000 rw-p 00000000 00:00 0 
7ffff457e000-7ffff457f000 ---p 00000000 00:00 0 
7ffff457f000-7ffff467f000 rw-p 00000000 00:00 0 
7ffff467f000-7ffff4680000 ---p 00000000 00:00 0 
7ffff4680000-7ffff4e80000 rw-p 00000000 00:00 0 
7ffff4e80000-7ffff4e82000 r-xp 00000000 09:7f 135947940                  /usr/lib64/libdl-2.17.so
7ffff4e82000-7ffff5082000 ---p 00002000 09:7f 135947940                  /usr/lib64/libdl-2.17.so
7ffff5082000-7ffff5083000 r--p 00002000 09:7f 135947940                  /usr/lib64/libdl-2.17.so
7ffff5083000-7ffff5084000 rw-p 00003000 09:7f 135947940                  /usr/lib64/libdl-2.17.so
7ffff5084000-7ffff508b000 r-xp 00000000 09:7f 135947966                  /usr/lib64/librt-2.17.so
7ffff508b000-7ffff528a000 ---p 00007000 09:7f 135947966                  /usr/lib64/librt-2.17.so
7ffff528a000-7ffff528b000 r--p 00006000 09:7f 135947966                  /usr/lib64/librt-2.17.so
7ffff528b000-7ffff528c000 rw-p 00007000 09:7f 135947966                  /usr/lib64/librt-2.17.so
7ffff528c000-7ffff52b1000 r-xp 00000000 09:7f 134328334                  /usr/lib64/libtinfo.so.5.9
7ffff52b1000-7ffff54b1000 ---p 00025000 09:7f 134328334                  /usr/lib64/libtinfo.so.5.9
7ffff54b1000-7ffff54b5000 r--p 00025000 09:7f 134328334                  /usr/lib64/libtinfo.so.5.9
7ffff54b5000-7ffff54b6000 rw-p 00029000 09:7f 134328334                  /usr/lib64/libtinfo.so.5.9
7ffff54b6000-7ffff54dc000 r-xp 00000000 09:7f 134328324                  /usr/lib64/libncurses.so.5.9
7ffff54dc000-7ffff56db000 ---p 00026000 09:7f 134328324                  /usr/lib64/libncurses.so.5.9
7ffff56db000-7ffff56dc000 r--p 00025000 09:7f 134328324                  /usr/lib64/libncurses.so.5.9
7ffff56dc000-7ffff56dd000 rw-p 00026000 09:7f 134328324                  /usr/lib64/libncurses.so.5.9
7ffff56dd000-7ffff5719000 r-xp 00000000 09:7f 134370020                  /usr/lib64/libreadline.so.6.2
7ffff5719000-7ffff5919000 ---p 0003c000 09:7f 134370020                  /usr/lib64/libreadline.so.6.2
7ffff5919000-7ffff591b000 r--p 0003c000 09:7f 134370020                  /usr/lib64/libreadline.so.6.2
7ffff591b000-7ffff5921000 rw-p 0003e000 09:7f 134370020                  /usr/lib64/libreadline.so.6.2
7ffff5921000-7ffff5923000 rw-p 00000000 00:00 0 
7ffff5923000-7ffff593a000 r-xp 00000000 09:7f 134501194                  /usr/lib64/libpthread-2.17.so
7ffff593a000-7ffff5b39000 ---p 00017000 09:7f 134501194                  /usr/lib64/libpthread-2.17.so
7ffff5b39000-7ffff5b3a000 r--p 00016000 09:7f 134501194                  /usr/lib64/libpthread-2.17.so
7ffff5b3a000-7ffff5b3b000 rw-p 00017000 09:7f 134501194                  /usr/lib64/libpthread-2.17.so
7ffff5b3b000-7ffff5b3f000 rw-p 00000000 00:00 0 
7ffff5b3f000-7ffff5cf7000 r-xp 00000000 09:7f 134321202                  /usr/lib64/libc-2.17.so
7ffff5cf7000-7ffff5ef7000 ---p 001b8000 09:7f 134321202                  /usr/lib64/libc-2.17.so
7ffff5ef7000-7ffff5efb000 r--p 001b8000 09:7f 134321202                  /usr/lib64/libc-2.17.so
7ffff5efb000-7ffff5efd000 rw-p 001bc000 09:7f 134321202                  /usr/lib64/libc-2.17.so
7ffff5efd000-7ffff5f02000 rw-p 00000000 00:00 0 
7ffff5f02000-7ffff5f17000 r-xp 00000000 09:7f 134763721                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7ffff5f17000-7ffff6116000 ---p 00015000 09:7f 134763721                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7ffff6116000-7ffff6117000 r--p 00014000 09:7f 134763721                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7ffff6117000-7ffff6118000 rw-p 00015000 09:7f 134763721                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7ffff6118000-7ffff6219000 r-xp 00000000 09:7f 135947942                  /usr/lib64/libm-2.17.so
7ffff6219000-7ffff6418000 ---p 00101000 09:7f 135947942                  /usr/lib64/libm-2.17.so
7ffff6418000-7ffff6419000 r--p 00100000 09:7f 135947942                  /usr/lib64/libm-2.17.so
7ffff6419000-7ffff641a000 rw-p 00101000 09:7f 135947942                  /usr/lib64/libm-2.17.so
7ffff641a000-7ffff6503000 r-xp 00000000 09:7f 134709262                  /usr/lib64/libstdc++.so.6.0.19
7ffff6503000-7ffff6703000 ---p 000e9000 09:7f 134709262                  /usr/lib64/libstdc++.so.6.0.19
7ffff6703000-7ffff670b000 r--p 000e9000 09:7f 134709262                  /usr/lib64/libstdc++.so.6.0.19
7ffff670b000-7ffff670d000 rw-p 000f1000 09:7f 134709262                  /usr/lib64/libstdc++.so.6.0.19
7ffff670d000-7ffff6722000 rw-p 00000000 00:00 0 
7ffff6722000-7ffff6785000 r-xp 00000000 00:28 24637444                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libCom.so.3.17.0
7ffff6785000-7ffff6985000 ---p 00063000 00:28 24637444                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libCom.so.3.17.0
7ffff6985000-7ffff6988000 r--p 00063000 00:28 24637444                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libCom.so.3.17.0
7ffff6988000-7ffff698a000 rw-p 00066000 00:28 24637444                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libCom.so.3.17.0
7ffff698a000-7ffff698c000 rw-p 00000000 00:00 0 
7ffff698c000-7ffff69d5000 r-xp 00000000 00:28 24637447                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libca.so.4.13.1
7ffff69d5000-7ffff6bd4000 ---p 00049000 00:28 24637447                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libca.so.4.13.1
7ffff6bd4000-7ffff6bd8000 r--p 00048000 00:28 24637447                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libca.so.4.13.1
7ffff6bd8000-7ffff6bd9000 rw-p 0004c000 00:28 24637447                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libca.so.4.13.1
7ffff6bd9000-7ffff6c54000 r-xp 00000000 00:28 30088032                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libdbCore.so.3.17.0
7ffff6c54000-7ffff6e53000 ---p 0007b000 00:28 30088032                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libdbCore.so.3.17.0
7ffff6e53000-7ffff6e56000 r--p 0007a000 00:28 30088032                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libdbCore.so.3.17.0
7ffff6e56000-7ffff6e5a000 rw-p 0007d000 00:28 30088032                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libdbCore.so.3.17.0
7ffff6e5a000-7ffff6e5d000 rw-p 00000000 00:00 0 
7ffff6e5d000-7ffff6ea7000 r-xp 00000000 00:28 30088038                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libdbRecStd.so.3.17.0
7ffff6ea7000-7ffff70a6000 ---p 0004a000 00:28 30088038                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libdbRecStd.so.3.17.0
7ffff70a6000-7ffff70a7000 r--p 00049000 00:28 30088038                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libdbRecStd.so.3.17.0
7ffff70a7000-7ffff70ab000 rw-p 0004a000 00:28 30088038                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libdbRecStd.so.3.17.0
7ffff70ab000-7ffff71f5000 r-xp 00000000 00:28 24637450                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvData.so.7.0.1
7ffff71f5000-7ffff73f5000 ---p 0014a000 00:28 24637450                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvData.so.7.0.1
7ffff73f5000-7ffff7406000 r--p 0014a000 00:28 24637450                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvData.so.7.0.1
7ffff7406000-7ffff7408000 rw-p 0015b000 00:28 24637450                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvData.so.7.0.1
7ffff7408000-7ffff7536000 r-xp 00000000 00:28 30088041                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvAccess.so.6.0.1
7ffff7536000-7ffff7735000 ---p 0012e000 00:28 30088041                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvAccess.so.6.0.1
7ffff7735000-7ffff7745000 r--p 0012d000 00:28 30088041                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvAccess.so.6.0.1
7ffff7745000-7ffff7747000 rw-p 0013d000 00:28 30088041                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvAccess.so.6.0.1
7ffff7747000-7ffff777e000 r-xp 00000000 00:28 17482350                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvAccessCA.so.6.0.1
7ffff777e000-7ffff797d000 ---p 00037000 00:28 17482350                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvAccessCA.so.6.0.1
7ffff797d000-7ffff7980000 r--p 00036000 00:28 17482350                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvAccessCA.so.6.0.1
7ffff7980000-7ffff7981000 rw-p 00039000 00:28 17482350                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvAccessCA.so.6.0.1
7ffff7981000-7ffff7988000 r-xp 00000000 00:28 17482347                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvAccessIOC.so.6.0.1
7ffff7988000-7ffff7b87000 ---p 00007000 00:28 17482347                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvAccessIOC.so.6.0.1
7ffff7b87000-7ffff7b88000 r--p 00006000 00:28 17482347                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvAccessIOC.so.6.0.1
7ffff7b88000-7ffff7b89000 rw-p 00007000 00:28 17482347                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libpvAccessIOC.so.6.0.1
7ffff7b89000-7ffff7bd7000 r-xp 00000000 00:28 18978332                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libqsrv.so.1.0
7ffff7bd7000-7ffff7dd7000 ---p 0004e000 00:28 18978332                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libqsrv.so.1.0
7ffff7dd7000-7ffff7dda000 r--p 0004e000 00:28 18978332                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libqsrv.so.1.0
7ffff7dda000-7ffff7ddb000 rw-p 00051000 00:28 18978332                   /home/phoebus3/ANJ/epics/base/7.0/lib/linux-x86_64/libqsrv.so.1.0
7ffff7ddb000-7ffff7dfc000 r-xp 00000000 09:7f 134321196                  /usr/lib64/ld-2.17.so
7ffff7e00000-7ffff7e21000 rw-p 00000000 00:00 0 
7ffff7e21000-7ffff7e22000 ---p 00000000 00:00 0 
7ffff7e22000-7ffff7ea2000 rw-p 00000000 00:00 0 
7ffff7ea2000-7ffff7ea3000 ---p 00000000 00:00 0 
7ffff7ea3000-7ffff7f23000 rw-p 00000000 00:00 0 
7ffff7f23000-7ffff7f24000 ---p 00000000 00:00 0 
7ffff7f24000-7ffff7fad000 rw-p 00000000 00:00 0 
7ffff7fb8000-7ffff7fb9000 rw-p 00000000 00:00 0 
7ffff7fb9000-7ffff7fc0000 r--s 00000000 09:7f 4059318                    /usr/lib64/gconv/gconv-modules.cache
7ffff7fc0000-7ffff7fc1000 rw-p 00000000 00:00 0 
7ffff7fc1000-7ffff7ff6000 r--s 00000000 09:7f 134370021                  /var/db/nscd/passwd
7ffff7ff6000-7ffff7ffa000 rw-p 00000000 00:00 0 
7ffff7ffa000-7ffff7ffc000 r-xp 00000000 00:00 0                          [vdso]
7ffff7ffc000-7ffff7ffd000 r--p 00021000 09:7f 134321196                  /usr/lib64/ld-2.17.so
7ffff7ffd000-7ffff7ffe000 rw-p 00022000 09:7f 134321196                  /usr/lib64/ld-2.17.so
7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0 
7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe6269700 (LWP 10034)]
0x00007ffff5b741f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-196.el7_4.2.x86_64 libgcc-4.8.5-16.el7_4.1.x86_64 libstdc++-4.8.5-16.el7_4.1.x86_64 ncurses-libs-5.9-14.20130511.el7_4.x86_64 readline-6.2-10.el7.x86_64
(gdb) thread apply all backtrace

Thread 30 (Thread 0x7fffe6269700 (LWP 10034)):
#0  0x00007ffff5b741f7 in raise () from /lib64/libc.so.6
#1  0x00007ffff5b758e8 in abort () from /lib64/libc.so.6
#2  0x00007ffff5bb3f47 in __libc_message () from /lib64/libc.so.6
#3  0x00007ffff5bbb619 in _int_free () from /lib64/libc.so.6
#4  0x00007ffff64d909e in std::string::assign(std::string const&) ()
   from /lib64/libstdc++.so.6
#5  0x00007ffff715f6bd in operator= (__str="NO_ALARM", this=0x7fff9c007690)
    at /usr/include/c++/4.8.2/bits/basic_string.h:547
#6  epics::pvData::detail::ScalarStorageOps<std::string>::store (
    this=this@entry=0x7fff9c007690, val="NO_ALARM") at ../pv/pvData.h:389
#7  0x00007ffff715f750 in put (v="NO_ALARM", this=0x7fff9c007630)
    at ../pv/pvData.h:426
#8  epics::pvData::PVScalarValue<std::string>::copyUnchecked (
    this=0x7fff9c007630, from=...) at ../pv/pvData.h:483
#9  0x00007ffff713c489 in epics::pvData::PVField::copyUnchecked (
    this=0x7fff9c007630, from=...) at ../../src/factory/PVField.cpp:249
#10 0x00007ffff713e5c8 in epics::pvData::PVStructure::copyUnchecked (
    this=this@entry=0x7fff9c007410, from=...)
    at ../../src/factory/PVStructure.cpp:362
#11 0x00007ffff713c560 in epics::pvData::PVField::copyUnchecked (
    this=0x7fff9c007410, from=...) at ../../src/factory/PVField.cpp:263
#12 0x00007ffff713e5c8 in epics::pvData::PVStructure::copyUnchecked (
    this=<optimized out>, from=...) at ../../src/factory/PVStructure.cpp:362
#13 0x00007ffff7bbd219 in p_postone (this=0x7fff9c0069f0)
    at ../../common/pvahelper.h:234
#14 BaseMonitor::post (this=this@entry=0x7fff9c0069f0)
    at ../../common/pvahelper.h:164
#15 0x00007ffff7bb9e94 in PDBSingleMonitor::requestUpdate (
    this=0x7fff9c0069f0) at ../pdbsingle.cpp:463
#16 0x00007ffff7bbd00a in BaseMonitor::release (this=0x7fff9c0069f0, 
    elem=...) at ../../common/pvahelper.h:320
#17 0x00007ffff74c91c2 in epics::pvAccess::ServerMonitorRequesterImpl::send
    (this=0x7fff9c006810, buffer=0x7fffa4000930, control=0x7fffa40008c0)
    at ../../src/server/responseHandlers.cpp:2109
#18 0x00007ffff7488fe4 in epics::pvAccess::detail::AbstractCodec::processSender (this=this@entry=0x7fffa40008c0, 
    sender=std::tr1::shared_ptr (count 4, weak 2) 0x7fff9c006868)
    at ../../src/remote/codec.cpp:888
#19 0x00007ffff748a02a in epics::pvAccess::detail::AbstractCodec::processSendQueue (this=0x7fffa40008c0) at ../../src/remote/codec.cpp:850
#20 0x00007ffff748a385 in epics::pvAccess::detail::AbstractCodec::processWrite (this=this@entry=0x7fffa40008c0) at ../../src/remote/codec.cpp:752
#21 0x00007ffff748a3f0 in epics::pvAccess::detail::BlockingTCPTransportCodec::sendThread (this=0x7fffa40008c0) at ../../src/remote/codec.cpp:1116
#22 0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x7fffa4000a38)
    at ../../src/osi/epicsThread.cpp:83
#23 0x00007ffff675b41c in start_routine (arg=0x7fffa40057e0)
    at ../../src/osi/os/posix/osdThread.c:403
#24 0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#25 0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7fffe62ea700 (LWP 10032)):
#0  0x00007ffff5c381fb in recv () from /lib64/libc.so.6
#1  0x00007ffff7485d6a in epics::pvAccess::detail::BlockingTCPTransportCodec::read (this=0x7fffa40008c0, dst=0x7fffa4000908)
    at ../../src/remote/codec.cpp:1308
#2  0x00007ffff7486ba7 in epics::pvAccess::detail::AbstractCodec::readToBuffer (this=this@entry=0x7fffa40008c0, requiredBytes=requiredBytes@entry=8, 
    persistent=persistent@entry=false) at ../../src/remote/codec.cpp:377
#3  0x00007ffff7486d9c in epics::pvAccess::detail::AbstractCodec::processReadNormal (this=0x7fffa40008c0) at ../../src/remote/codec.cpp:173
#4  0x00007ffff74871f5 in epics::pvAccess::detail::AbstractCodec::processRead (this=this@entry=0x7fffa40008c0) at ../../src/remote/codec.cpp:121
#5  0x00007ffff7488668 in epics::pvAccess::detail::BlockingTCPTransportCodec:---Type <return> to continue, or q <return> to quit---
:receiveThread (this=0x7fffa40008c0) at ../../src/remote/codec.cpp:1091
#6  0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x7fffa40009f8)
    at ../../src/osi/epicsThread.cpp:83
#7  0x00007ffff675b41c in start_routine (arg=0x7fffa4005300)
    at ../../src/osi/os/posix/osdThread.c:403
#8  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7fffe65ed700 (LWP 10030)):
#0  0x00007ffff5c38263 in recvfrom () from /lib64/libc.so.6
#1  0x00007ffff7472874 in epics::pvAccess::BlockingUDPTransport::run (
    this=0x73a060) at ../../src/remote/blockingUDPTransport.cpp:221
#2  0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x75a5a0)
    at ../../src/osi/epicsThread.cpp:83
#3  0x00007ffff675b41c in start_routine (arg=0x75a5e0)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7fffe666e700 (LWP 10029)):
#0  0x00007ffff5c38263 in recvfrom () from /lib64/libc.so.6
#1  0x00007ffff7472874 in epics::pvAccess::BlockingUDPTransport::run (
    this=0x6f9210) at ../../src/remote/blockingUDPTransport.cpp:221
#2  0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x6f9830)
    at ../../src/osi/epicsThread.cpp:83
#3  0x00007ffff675b41c in start_routine (arg=0x739c90)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7fffe66ef700 (LWP 10028)):
#0  0x00007ffff5c38263 in recvfrom () from /lib64/libc.so.6
#1  0x00007ffff7472874 in epics::pvAccess::BlockingUDPTransport::run (
    this=0x6f8eb0) at ../../src/remote/blockingUDPTransport.cpp:221
#2  0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x6f97f0)
    at ../../src/osi/epicsThread.cpp:83
#3  0x00007ffff675b41c in start_routine (arg=0x6f96a0)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7fffe6770700 (LWP 10027)):
#0  0x00007ffff5c38263 in recvfrom () from /lib64/libc.so.6
#1  0x00007ffff7472874 in epics::pvAccess::BlockingUDPTransport::run (
    this=0x6d8760) at ../../src/remote/blockingUDPTransport.cpp:221
#2  0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x6f89b0)
    at ../../src/osi/epicsThread.cpp:83
#3  0x00007ffff675b41c in start_routine (arg=0x6f8b10)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7fffe6871700 (LWP 10026)):
#0  0x00007ffff5c37ffd in accept () from /lib64/libc.so.6
#1  0x00007ffff6759223 in epicsSocketAccept (sock=<optimized out>, 
    pAddr=<optimized out>, addrlen=<optimized out>)
    at ../../src/osi/os/posix/osdSock.c:98
#2  0x00007ffff7480376 in epics::pvAccess::BlockingTCPAcceptor::run (
    this=0x6d76a0) at ../../src/remote/blockingTCPAcceptor.cpp:177
#3  0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x6d76f0)
    at ../../src/osi/epicsThread.cpp:83
#4  0x00007ffff675b41c in start_routine (arg=0x6d7880)
    at ../../src/osi/os/posix/osdThread.c:403
#5  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 23 (Thread 0x7fffe6972700 (LWP 10025)):
#0  0x00007ffff593142d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007ffff592cde6 in _L_lock_870 () from /lib64/libpthread.so.0
#2  0x00007ffff592ccdf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007ffff675cda6 in mutexLock (id=0x7fff9c006b40)
    at ../../src/osi/os/posix/osdMutex.c:46
#4  epicsMutexOsdLock (pmutex=0x7fff9c006b40)
    at ../../src/osi/os/posix/osdMutex.c:130
#5  0x00007ffff67551cc in epicsMutex::lock (this=<optimized out>, 
    this@entry=0x7fff9c006a08) at ../../src/osi/epicsMutex.cpp:273
#6  0x00007ffff7bb956e in epicsGuard (mutexIn=..., this=<synthetic pointer>)
    at /home/phoebus3/ANJ/epics/base/7.0/include/epicsGuard.h:70
#7  post (updated=..., this=0x7fff9c0069f0) at ../../common/pvahelper.h:205
#8  pdb_single_event (user_arg=<optimized out>, chan=<optimized out>, 
    eventsRemaining=<optimized out>, pfl=<optimized out>)
    at ../pdbsingle.cpp:61
#9  0x00007ffff6c0cee6 in event_read (ev_que=0x7fffd802d4b8)
    at ../../../src/ioc/db/dbEvent.c:970
#10 event_task (pParm=0x7fffd802d4b8) at ../../../src/ioc/db/dbEvent.c:1049
#11 0x00007ffff675b41c in start_routine (arg=0x6d6cc0)
    at ../../src/osi/os/posix/osdThread.c:403
#12 0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#13 0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7fffe6a73700 (LWP 10024)):
#0  0x00007ffff592f006 in pthread_cond_signal@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d408 in epicsEventTrigger (pevent=0x6cac90)
    at ../../src/osi/os/posix/osdEvent.c:88
#2  0x00007ffff6755be9 in epicsEventMustTrigger (id=<optimized out>)
    at ../../src/osi/epicsEvent.cpp:118
#3  0x00007ffff6c098bd in scanOnceCallback (precord=<optimized out>, 
    cb=<optimized out>, usr=<optimized out>)
    at ../../../src/ioc/db/dbScan.c:675
#4  0x00007ffff6c1622f in scanLinkOnce (prec=<optimized out>, pca=0x6b7c80)
    at ../../../src/ioc/db/dbCa.c:734
#5  0x00007ffff6c164ac in eventCallback (arg=...)
    at ../../../src/ioc/db/dbCa.c:896
#6  0x00007ffff69bd388 in oldSubscription::current (this=<optimized out>, 
    guard=..., type=20, count=1, pData=0x7fffa8000900)
    at ../oldSubscription.cpp:68
#7  0x00007ffff6c1790f in dbContext::callStateNotify (this=0x7fffd8000c80, 
    dbch=0x7fffd80134e8, type=20, count=0, pfl=<optimized out>, 
    notifyIn=...) at ../../../src/ioc/db/dbContext.cpp:189
#8  0x00007ffff6c0cee6 in event_read (ev_que=0x7fffd802dd40)
    at ../../../src/ioc/db/dbEvent.c:970
#9  event_task (pParm=0x7fffd802dd40) at ../../../src/ioc/db/dbEvent.c:1049
#10 0x00007ffff675b41c in start_routine (arg=0x7fffd802e820)
    at ../../src/osi/os/posix/osdThread.c:403
#11 0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#12 0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7fffe6af4700 (LWP 10023)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6d5f30, 
    timeout=14.999930813000001) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff710e500 in epics::pvData::Event::wait (this=<optimized out>, 
    timeOut=<optimized out>) at ../../src/misc/event.cpp:60
#3  0x00007ffff711074a in epics::pvData::Timer::run (this=0x6d5bc0)
    at ../../src/misc/timer.cpp:138
#4  0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x6d5bf0)
    at ../../src/osi/epicsThread.cpp:83
#5  0x00007ffff675b41c in start_routine (arg=0x6d60f0)
    at ../../src/osi/os/posix/osdThread.c:403
#6  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#7  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7ffff7ea1700 (LWP 10022)):
#0  0x00007ffff5bfe1ad in nanosleep () from /lib64/libc.so.6
#1  0x00007ffff675abd8 in epicsThreadSleep (seconds=<optimized out>)
    at ../../src/osi/os/posix/osdThread.c:715
#2  0x00007ffff6c3759b in rsrv_online_notify_task (pParm=<optimized out>)
    at ../../../src/ioc/rsrv/online_notify.c:98
#3  0x00007ffff675b41c in start_routine (arg=0x6d5640)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7fffe6bf5700 (LWP 10021)):
#0  0x00007ffff5c38263 in recvfrom () from /lib64/libc.so.6
#1  0x00007ffff6c370e6 in cast_server (pParm=<optimized out>)
    at ../../../src/ioc/rsrv/cast_server.c:167
#2  0x00007ffff675b41c in start_routine (arg=0x6d52e0)
    at ../../src/osi/os/posix/osdThread.c:403
#3  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#4  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7ffff417b700 (LWP 10020)):
#0  0x00007ffff5c37ffd in accept () from /lib64/libc.so.6
#1  0x00007ffff6759223 in epicsSocketAccept (sock=<optimized out>, 
    pAddr=<optimized out>, addrlen=<optimized out>)
    at ../../src/osi/os/posix/osdSock.c:98
#2  0x00007ffff6c305e5 in req_server (pParm=<optimized out>)
    at ../../../src/ioc/rsrv/caservertask.c:93
#3  0x00007ffff675b41c in start_routine (arg=0x6d4f80)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7fffe6df6700 (LWP 10018)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6c4e00, 
    timeout=0.099932652999999996) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff6c0b2f5 in periodicTask (arg=0x6c4d30)
    at ../../../src/ioc/db/dbScan.c:796
#3  0x00007ffff675b41c in start_routine (arg=0x6cc4a0)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7fffe6ff7700 (LWP 10017)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6c4cc0, 
    timeout=0.19993502499999999) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff6c0b2f5 in periodicTask (arg=0x6c4c30)
    at ../../../src/ioc/db/dbScan.c:796
#3  0x00007ffff675b41c in start_routine (arg=0x6cc140)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7fffe71f8700 (LWP 10016)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6c4bc0, 
    timeout=0.49993812999999998) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff6c0b2f5 in periodicTask (arg=0x6b84f0)
    at ../../../src/ioc/db/dbScan.c:796
#3  0x00007ffff675b41c in start_routine (arg=0x6cbde0)
---Type <return> to continue, or q <return> to quit---
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fffe73f9700 (LWP 10015)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6b8480, 
    timeout=0.99993643899999995) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff6c0b2f5 in periodicTask (arg=0x6b83b0)
    at ../../../src/ioc/db/dbScan.c:796
#3  0x00007ffff675b41c in start_routine (arg=0x6cba80)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7fffe75fa700 (LWP 10014)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6b8340, 
    timeout=1.999932045) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff6c0b2f5 in periodicTask (arg=0x6b8270)
    at ../../../src/ioc/db/dbScan.c:796
#3  0x00007ffff675b41c in start_routine (arg=0x6cb720)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fffe77fb700 (LWP 10013)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6b8200, 
    timeout=4.9999249060000004) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff6c0b2f5 in periodicTask (arg=0x6b8130)
    at ../../../src/ioc/db/dbScan.c:796
#3  0x00007ffff675b41c in start_routine (arg=0x6cb3c0)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fffe79fc700 (LWP 10012)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6b80c0, 
    timeout=9.9999941450000005) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff6c0b2f5 in periodicTask (arg=0x6b7ff0)
    at ../../../src/ioc/db/dbScan.c:796
#3  0x00007ffff675b41c in start_routine (arg=0x6cb060)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fffe7bfd700 (LWP 10011)):
#0  0x00007ffff593142d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007ffff5933c6c in _L_cond_lock_789 () from /lib64/libpthread.so.0
#2  0x00007ffff5933b01 in __pthread_mutex_cond_lock ()
   from /lib64/libpthread.so.0
#3  0x00007ffff592e9d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#4  0x00007ffff675d2d6 in epicsEventWait (pevent=0x6cac90)
    at ../../src/osi/os/posix/osdEvent.c:103
#5  0x00007ffff6755bb9 in epicsEventMustWait (id=<optimized out>)
    at ../../src/osi/epicsEvent.cpp:125
#6  0x00007ffff6c0b19c in onceTask (arg=<optimized out>)
    at ../../../src/ioc/db/dbScan.c:687
#7  0x00007ffff675b41c in start_routine (arg=0x6cad00)
---Type <return> to continue, or q <return> to quit---
    at ../../src/osi/os/posix/osdThread.c:403
#8  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fffe7dfe700 (LWP 10010)):
#0  0x00007ffff592e945 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d2d6 in epicsEventWait (pevent=0x6b7400)
    at ../../src/osi/os/posix/osdEvent.c:103
#2  0x00007ffff6755bb9 in epicsEventMustWait (id=<optimized out>)
    at ../../src/osi/epicsEvent.cpp:125
#3  0x00007ffff6c1526c in dbCaTask (arg=<optimized out>)
    at ../../../src/ioc/db/dbCa.c:1038
#4  0x00007ffff675b41c in start_routine (arg=0x6b74e0)
    at ../../src/osi/os/posix/osdThread.c:403
#5  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fffe7fff700 (LWP 10009)):
#0  0x00007ffff592e945 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d2d6 in epicsEventWait (pevent=0x6b6e70)
    at ../../src/osi/os/posix/osdEvent.c:103
#2  0x00007ffff6755bb9 in epicsEventMustWait (id=<optimized out>)
    at ../../src/osi/epicsEvent.cpp:125
#3  0x00007ffff6c12cf5 in callbackTask (arg=<optimized out>)
    at ../../../src/ioc/db/callback.c:166
#4  0x00007ffff675b41c in start_routine (arg=0x6b6f40)
    at ../../src/osi/os/posix/osdThread.c:403
#5  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7ffff437c700 (LWP 10008)):
#0  0x00007ffff592e945 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d2d6 in epicsEventWait (pevent=0x6b6a40)
    at ../../src/osi/os/posix/osdEvent.c:103
#2  0x00007ffff6755bb9 in epicsEventMustWait (id=<optimized out>)
    at ../../src/osi/epicsEvent.cpp:125
#3  0x00007ffff6c12cf5 in callbackTask (arg=<optimized out>)
    at ../../../src/ioc/db/callback.c:166
#4  0x00007ffff675b41c in start_routine (arg=0x6b6b10)
    at ../../src/osi/os/posix/osdThread.c:403
#5  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7ffff457d700 (LWP 10007)):
#0  0x00007ffff592e945 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d2d6 in epicsEventWait (pevent=0x6b6610)
    at ../../src/osi/os/posix/osdEvent.c:103
#2  0x00007ffff6755bb9 in epicsEventMustWait (id=<optimized out>)
    at ../../src/osi/epicsEvent.cpp:125
#3  0x00007ffff6c12cf5 in callbackTask (arg=<optimized out>)
    at ../../../src/ioc/db/callback.c:166
#4  0x00007ffff675b41c in start_routine (arg=0x6b66e0)
    at ../../src/osi/os/posix/osdThread.c:403
#5  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7ffff467e700 (LWP 10006)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6b6070, 
    timeout=1.7976931348623157e+308)
    at ../../src/osi/os/posix/osdEvent.c:130
---Type <return> to continue, or q <return> to quit---
#2  0x00007ffff6755dac in epicsEvent::wait (this=<optimized out>, 
    timeOut=<optimized out>) at ../../src/osi/epicsEvent.cpp:75
#3  0x00007ffff67666f3 in timerQueueActive::run (this=0x6b5d80)
    at ../../src/timer/timerQueueActive.cpp:95
#4  0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x6b5e50)
    at ../../src/osi/epicsThread.cpp:83
#5  0x00007ffff675b41c in start_routine (arg=0x6b62a0)
    at ../../src/osi/os/posix/osdThread.c:403
#6  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#7  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7ffff7f22700 (LWP 10005)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6b5760, 
    timeout=6) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff6763844 in twdTask (arg=<optimized out>)
    at ../../src/taskwd/taskwd.c:126
#3  0x00007ffff675b41c in start_routine (arg=0x6b5840)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7ffff7fa3700 (LWP 10004)):
#0  0x00007ffff592e945 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d2d6 in epicsEventWait (pevent=0x6a01d0)
    at ../../src/osi/os/posix/osdEvent.c:103
#2  0x00007ffff6755bb9 in epicsEventMustWait (id=<optimized out>)
    at ../../src/osi/epicsEvent.cpp:125
#3  0x00007ffff6743d47 in errlogThread () at ../../src/error/errlog.c:561
#4  0x00007ffff675b41c in start_routine (arg=0x6b52d0)
    at ../../src/osi/os/posix/osdThread.c:403
#5  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7ffff7fa5740 (LWP 9993)):
#0  0x00007ffff5c287fd in read () from /lib64/libc.so.6
#1  0x00007ffff5707097 in rl_getc () from /lib64/libreadline.so.6
#2  0x00007ffff57078e7 in rl_read_key () from /lib64/libreadline.so.6
#3  0x00007ffff56f2b2f in readline_internal_char ()
   from /lib64/libreadline.so.6
#4  0x00007ffff56f31e5 in readline () from /lib64/libreadline.so.6
#5  0x00007ffff6759f1a in osdReadline (context=0x6d7c90, 
    prompt=0x7ffff67702ee "epics> ")
    at ../../src/osi/os/default/gnuReadline.c:67
#6  epicsReadline (prompt=0x7ffff67702ee "epics> ", context=0x6d7c90)
    at ../../src/osi/epicsReadline.c:86
#7  0x00007ffff674ae28 in iocshBody (pathname=<optimized out>, 
    commandLine=0x0, macros=0x0) at ../../src/iocsh/iocsh.cpp:618
#8  0x000000000040476f in main (argc=0, argv=0x7fffffffdd50)
    at ../softMain.cpp:231
(gdb)
mdavidsaver commented 6 years ago

Ok, I think I see it. There is a data race between the monitor update callback and the PVA server over a shared PVStructure.

mdavidsaver commented 6 years ago

I've pushed up a candidate fix to the master branch of https://github.com/mdavidsaver/pva2pva. Please update and see if you can replicate.

mdavidsaver commented 6 years ago

Never mind. This change introduces a potential deadlock.

mdavidsaver commented 6 years ago

I've pushed up another candidate fix. This one shouldn't deadlock. @anjohnson Please update and see if you can replicate this crash (I still haven't been able to).

anjohnson commented 6 years ago

If anything this seemed to make it worse, it crashed much faster. Here's the back-trace:

tux% gdb bin/linux-x86_64/softIocPVA 
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/phoebus3/ANJ/epics/base/7.0/bin/linux-x86_64/softIocPVA...done.
(gdb) run -d ~/db/spam.db
Starting program: /home/phoebus3/ANJ/epics/base/7.0/bin/linux-x86_64/softIocPVA -d ~/db/spam.db
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff4e7f700 (LWP 1774)]
[Thread 0x7ffff4e7f700 (LWP 1774) exited]
Detaching after fork from child process 1775.
[New Thread 0x7ffff7fa3700 (LWP 1776)]
Starting iocInit
############################################################################
## EPICS R7.0.1.2-DEV
## EPICS Base built Jan  4 2018
############################################################################
[New Thread 0x7ffff7f22700 (LWP 1777)]
[New Thread 0x7ffff467e700 (LWP 1778)]
[New Thread 0x7ffff457d700 (LWP 1779)]
[New Thread 0x7ffff437c700 (LWP 1780)]
[New Thread 0x7fffe7fff700 (LWP 1781)]
[New Thread 0x7fffe7dfe700 (LWP 1782)]
[New Thread 0x7fffe7bfd700 (LWP 1783)]
[New Thread 0x7fffe79fc700 (LWP 1784)]
[New Thread 0x7fffe77fb700 (LWP 1785)]
[New Thread 0x7fffe75fa700 (LWP 1786)]
[New Thread 0x7fffe73f9700 (LWP 1787)]
[New Thread 0x7fffe71f8700 (LWP 1788)]
[New Thread 0x7fffe6ff7700 (LWP 1789)]
[New Thread 0x7fffe6df6700 (LWP 1790)]
[New Thread 0x7ffff417b700 (LWP 1792)]
[New Thread 0x7fffe6bf5700 (LWP 1796)]
[New Thread 0x7ffff7ea1700 (LWP 1799)]
iocRun: All initialization complete
[New Thread 0x7fffe6af4700 (LWP 1800)]
[New Thread 0x7fffe6a73700 (LWP 1801)]
[New Thread 0x7fffe6972700 (LWP 1802)]
[New Thread 0x7fffe6871700 (LWP 1803)]
[New Thread 0x7fffe6770700 (LWP 1804)]
[New Thread 0x7fffe66ef700 (LWP 1805)]
[New Thread 0x7fffe666e700 (LWP 1806)]
[New Thread 0x7fffe65ed700 (LWP 1807)]
[New Thread 0x7fffe62ea700 (LWP 1810)]
[New Thread 0x7fffe6269700 (LWP 1811)]
epics> 
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe6269700 (LWP 1811)]
0x00007ffff5bba5aa in malloc_consolidate () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-196.el7_4.2.x86_64 libgcc-4.8.5-16.el7_4.1.x86_64 libstdc++-4.8.5-16.el7_4.1.x86_64 ncurses-libs-5.9-14.20130511.el7_4.x86_64 readline-6.2-10.el7.x86_64
(gdb) thread apply all backtrace

Thread 30 (Thread 0x7fffe6269700 (LWP 1811)):
#0  0x00007ffff5bba5aa in malloc_consolidate () from /lib64/libc.so.6
#1  0x00007ffff5bbb4fe in _int_free () from /lib64/libc.so.6
#2  0x00007ffff7bbbd0a in deallocate (this=0x7fff9c006a80, __p=<optimized out>)
    at /usr/include/c++/4.8.2/ext/new_allocator.h:110
#3  _M_deallocate_node (this=0x7fff9c006a80, __p=<optimized out>)
    at /usr/include/c++/4.8.2/bits/stl_deque.h:539
#4  _M_pop_front_aux (this=0x7fff9c006a80)
    at /usr/include/c++/4.8.2/bits/deque.tcc:523
#5  pop_front (this=0x7fff9c006a80)
    at /usr/include/c++/4.8.2/bits/stl_deque.h:1430
#6  BaseMonitor::poll (this=0x7fff9c0069f0) at ../../common/pvahelper.h:305
#7  0x00007ffff74c9112 in epics::pvAccess::ServerMonitorRequesterImpl::send (
    this=0x7fff9c006810, buffer=0x7fffa4000930, control=0x7fffa40008c0)
    at ../../src/server/responseHandlers.cpp:2091
#8  0x00007ffff7488fe4 in epics::pvAccess::detail::AbstractCodec::processSender
    (this=this@entry=0x7fffa40008c0, 
    sender=std::tr1::shared_ptr (count 4, weak 2) 0x7fff9c006868)
    at ../../src/remote/codec.cpp:888
#9  0x00007ffff748a02a in epics::pvAccess::detail::AbstractCodec::processSendQueue (this=0x7fffa40008c0) at ../../src/remote/codec.cpp:850
#10 0x00007ffff748a385 in epics::pvAccess::detail::AbstractCodec::processWrite
    (this=this@entry=0x7fffa40008c0) at ../../src/remote/codec.cpp:752
#11 0x00007ffff748a3f0 in epics::pvAccess::detail::BlockingTCPTransportCodec::sendThread (this=0x7fffa40008c0) at ../../src/remote/codec.cpp:1116
#12 0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x7fffa4000a38)
    at ../../src/osi/epicsThread.cpp:83
#13 0x00007ffff675b41c in start_routine (arg=0x7fffa40057e0)
    at ../../src/osi/os/posix/osdThread.c:403
#14 0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#15 0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7fffe62ea700 (LWP 1810)):
#0  0x00007ffff5c381fb in recv () from /lib64/libc.so.6
#1  0x00007ffff7485d6a in epics::pvAccess::detail::BlockingTCPTransportCodec::read (this=0x7fffa40008c0, dst=0x7fffa4000908) at ../../src/remote/codec.cpp:1308
#2  0x00007ffff7486ba7 in epics::pvAccess::detail::AbstractCodec::readToBuffer
    (this=this@entry=0x7fffa40008c0, requiredBytes=requiredBytes@entry=8, 
    persistent=persistent@entry=false) at ../../src/remote/codec.cpp:377
#3  0x00007ffff7486d9c in epics::pvAccess::detail::AbstractCodec::processReadNormal (this=0x7fffa40008c0) at ../../src/remote/codec.cpp:173
#4  0x00007ffff74871f5 in epics::pvAccess::detail::AbstractCodec::processRead (
    this=this@entry=0x7fffa40008c0) at ../../src/remote/codec.cpp:121
#5  0x00007ffff7488668 in epics::pvAccess::detail::BlockingTCPTransportCodec::receiveThread (this=0x7fffa40008c0) at ../../src/remote/codec.cpp:1091
#6  0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x7fffa40009f8)
    at ../../src/osi/epicsThread.cpp:83
#7  0x00007ffff675b41c in start_routine (arg=0x7fffa4005300)
    at ../../src/osi/os/posix/osdThread.c:403
#8  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7fffe65ed700 (LWP 1807)):
#0  0x00007ffff5c38263 in recvfrom () from /lib64/libc.so.6
#1  0x00007ffff7472874 in epics::pvAccess::BlockingUDPTransport::run (
    this=0x739650) at ../../src/remote/blockingUDPTransport.cpp:221
#2  0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x759b90)
    at ../../src/osi/epicsThread.cpp:83
#3  0x00007ffff675b41c in start_routine (arg=0x759bd0)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7fffe666e700 (LWP 1806)):
#0  0x00007ffff5c38263 in recvfrom () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
#1  0x00007ffff7472874 in epics::pvAccess::BlockingUDPTransport::run (
    this=0x6f8800) at ../../src/remote/blockingUDPTransport.cpp:221
#2  0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x6f8e20)
    at ../../src/osi/epicsThread.cpp:83
#3  0x00007ffff675b41c in start_routine (arg=0x739280)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7fffe66ef700 (LWP 1805)):
#0  0x00007ffff5c38263 in recvfrom () from /lib64/libc.so.6
#1  0x00007ffff7472874 in epics::pvAccess::BlockingUDPTransport::run (
    this=0x6f84a0) at ../../src/remote/blockingUDPTransport.cpp:221
#2  0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x6f8de0)
    at ../../src/osi/epicsThread.cpp:83
#3  0x00007ffff675b41c in start_routine (arg=0x6f8c90)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7fffe6770700 (LWP 1804)):
#0  0x00007ffff5c38263 in recvfrom () from /lib64/libc.so.6
#1  0x00007ffff7472874 in epics::pvAccess::BlockingUDPTransport::run (
    this=0x6d7c20) at ../../src/remote/blockingUDPTransport.cpp:221
#2  0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x6f7fa0)
    at ../../src/osi/epicsThread.cpp:83
#3  0x00007ffff675b41c in start_routine (arg=0x6f8100)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7fffe6871700 (LWP 1803)):
#0  0x00007ffff5c37ffd in accept () from /lib64/libc.so.6
#1  0x00007ffff6759223 in epicsSocketAccept (sock=<optimized out>, 
    pAddr=<optimized out>, addrlen=<optimized out>)
    at ../../src/osi/os/posix/osdSock.c:98
#2  0x00007ffff7480376 in epics::pvAccess::BlockingTCPAcceptor::run (
    this=0x6d76a0) at ../../src/remote/blockingTCPAcceptor.cpp:177
#3  0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x6d76f0)
    at ../../src/osi/epicsThread.cpp:83
#4  0x00007ffff675b41c in start_routine (arg=0x6d7880)
    at ../../src/osi/os/posix/osdThread.c:403
#5  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7fffe6972700 (LWP 1802)):
#0  0x00007ffff5c44eec in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007ffff5bc205d in _L_lock_14730 () from /lib64/libc.so.6
#2  0x00007ffff5bbf163 in malloc () from /lib64/libc.so.6
#3  0x00007ffff64791bd in operator new(unsigned long) ()
   from /lib64/libstdc++.so.6
#4  0x00007ffff64d7cf9 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) () from /lib64/libstdc++.so.6
#5  0x00007ffff7141bd1 in std::string::_S_construct<char const*> (
    __beg=0x7ffff6772aaa "NO_ALARM", __end=<optimized out>, __a=...)
    at /usr/include/c++/4.8.2/bits/basic_string.tcc:138
#6  0x00007ffff64d99b8 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) ()
   from /lib64/libstdc++.so.6
#7  0x00007ffff7b9ce17 in (anonymous namespace)::mapStatus (code=0, status=
    0x7fff9c0048b0, message=0x7fff9c004950) at ../pvif.cpp:226
#8  0x00007ffff7b9d124 in (anonymous namespace)::putTime (pv=..., 
    dbe=dbe@entry=5, pfl=pfl@entry=0x7ffff7e1fed8) at ../pvif.cpp:295
#9  0x00007ffff7ba40de in putAll<(anonymous namespace)::pvScalar, (anonymous namespace)::metaDOUBLE> (pfl=0x7ffff7e1fed8, dbe=5, pv=...) at ../pvif.cpp:528
#10 (anonymous namespace)::PVIFScalarNumeric<(anonymous namespace)::pvScalar, (a---Type <return> to continue, or q <return> to quit---
nonymous namespace)::metaDOUBLE>::put (this=0x7fff9c005c80, mask=..., dbe=5, 
    pfl=0x7ffff7e1fed8) at ../pvif.cpp:596
#11 0x00007ffff7bb93f5 in pdb_single_event (user_arg=0x7fff9c000bc0, 
    chan=<optimized out>, eventsRemaining=<optimized out>, pfl=0x7ffff7e1fed8)
    at ../pdbsingle.cpp:58
#12 0x00007ffff6c0cee6 in event_read (ev_que=0x7fffd802d4b8)
    at ../../../src/ioc/db/dbEvent.c:970
#13 event_task (pParm=0x7fffd802d4b8) at ../../../src/ioc/db/dbEvent.c:1049
#14 0x00007ffff675b41c in start_routine (arg=0x6d6cc0)
    at ../../src/osi/os/posix/osdThread.c:403
#15 0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#16 0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7fffe6a73700 (LWP 1801)):
#0  0x00007ffff592e945 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d2d6 in epicsEventWait (pevent=0x7fffd802e660)
    at ../../src/osi/os/posix/osdEvent.c:103
#2  0x00007ffff6755bb9 in epicsEventMustWait (id=<optimized out>)
    at ../../src/osi/epicsEvent.cpp:125
#3  0x00007ffff6c0cd46 in event_task (pParm=0x7fffd802dd40)
    at ../../../src/ioc/db/dbEvent.c:1022
#4  0x00007ffff675b41c in start_routine (arg=0x7fffd802e820)
    at ../../src/osi/os/posix/osdThread.c:403
#5  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7fffe6af4700 (LWP 1800)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6d5f30, 
    timeout=14.999917832) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff710e500 in epics::pvData::Event::wait (this=<optimized out>, 
    timeOut=<optimized out>) at ../../src/misc/event.cpp:60
#3  0x00007ffff711074a in epics::pvData::Timer::run (this=0x6d5bc0)
    at ../../src/misc/timer.cpp:138
#4  0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x6d5bf0)
    at ../../src/osi/epicsThread.cpp:83
#5  0x00007ffff675b41c in start_routine (arg=0x6d60f0)
    at ../../src/osi/os/posix/osdThread.c:403
#6  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#7  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7ffff7ea1700 (LWP 1799)):
#0  0x00007ffff5bfe1ad in nanosleep () from /lib64/libc.so.6
#1  0x00007ffff675abd8 in epicsThreadSleep (seconds=<optimized out>)
    at ../../src/osi/os/posix/osdThread.c:715
#2  0x00007ffff6c3759b in rsrv_online_notify_task (pParm=<optimized out>)
    at ../../../src/ioc/rsrv/online_notify.c:98
#3  0x00007ffff675b41c in start_routine (arg=0x6d5640)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7fffe6bf5700 (LWP 1796)):
#0  0x00007ffff5c38263 in recvfrom () from /lib64/libc.so.6
#1  0x00007ffff6c370e6 in cast_server (pParm=<optimized out>)
    at ../../../src/ioc/rsrv/cast_server.c:167
#2  0x00007ffff675b41c in start_routine (arg=0x6d52e0)
    at ../../src/osi/os/posix/osdThread.c:403
#3  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#4  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7ffff417b700 (LWP 1792)):
#0  0x00007ffff5c37ffd in accept () from /lib64/libc.so.6
#1  0x00007ffff6759223 in epicsSocketAccept (sock=<optimized out>, 
---Type <return> to continue, or q <return> to quit---
    pAddr=<optimized out>, addrlen=<optimized out>)
    at ../../src/osi/os/posix/osdSock.c:98
#2  0x00007ffff6c305e5 in req_server (pParm=<optimized out>)
    at ../../../src/ioc/rsrv/caservertask.c:93
#3  0x00007ffff675b41c in start_routine (arg=0x6d4f80)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7fffe6df6700 (LWP 1790)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6c4e00, 
    timeout=0.099938610999999997) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff6c0b2f5 in periodicTask (arg=0x6c4d30)
    at ../../../src/ioc/db/dbScan.c:796
#3  0x00007ffff675b41c in start_routine (arg=0x6cc4a0)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7fffe6ff7700 (LWP 1789)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6c4cc0, 
    timeout=0.199932689) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff6c0b2f5 in periodicTask (arg=0x6c4c30)
    at ../../../src/ioc/db/dbScan.c:796
#3  0x00007ffff675b41c in start_routine (arg=0x6cc140)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7fffe71f8700 (LWP 1788)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6c4bc0, 
    timeout=0.499934936) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff6c0b2f5 in periodicTask (arg=0x6b84f0)
    at ../../../src/ioc/db/dbScan.c:796
#3  0x00007ffff675b41c in start_routine (arg=0x6cbde0)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fffe73f9700 (LWP 1787)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6b8480, 
    timeout=0.99993597700000003) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff6c0b2f5 in periodicTask (arg=0x6b83b0)
    at ../../../src/ioc/db/dbScan.c:796
#3  0x00007ffff675b41c in start_routine (arg=0x6cba80)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7fffe75fa700 (LWP 1786)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6b8340, 
    timeout=1.9999980100000001) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff6c0b2f5 in periodicTask (arg=0x6b8270)
    at ../../../src/ioc/db/dbScan.c:796
#3  0x00007ffff675b41c in start_routine (arg=0x6cb720)
    at ../../src/osi/os/posix/osdThread.c:403
---Type <return> to continue, or q <return> to quit---
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fffe77fb700 (LWP 1785)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6b8200, 
    timeout=4.9999979049999999) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff6c0b2f5 in periodicTask (arg=0x6b8130)
    at ../../../src/ioc/db/dbScan.c:796
#3  0x00007ffff675b41c in start_routine (arg=0x6cb3c0)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fffe79fc700 (LWP 1784)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6b80c0, 
    timeout=9.9999928730000001) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff6c0b2f5 in periodicTask (arg=0x6b7ff0)
    at ../../../src/ioc/db/dbScan.c:796
#3  0x00007ffff675b41c in start_routine (arg=0x6cb060)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fffe7bfd700 (LWP 1783)):
#0  0x00007ffff593142d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007ffff592cde6 in _L_lock_870 () from /lib64/libpthread.so.0
#2  0x00007ffff592ccdf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007ffff675cda6 in mutexLock (id=0x6b79c0)
    at ../../src/osi/os/posix/osdMutex.c:46
#4  epicsMutexOsdLock (pmutex=0x6b79c0)
    at ../../src/osi/os/posix/osdMutex.c:130
#5  0x00007ffff6bf5499 in dbScanLock (precord=<optimized out>)
    at ../../../src/ioc/db/dbLock.c:199
#6  0x00007ffff6c0b1de in onceTask (arg=<optimized out>)
    at ../../../src/ioc/db/dbScan.c:699
#7  0x00007ffff675b41c in start_routine (arg=0x6cad00)
    at ../../src/osi/os/posix/osdThread.c:403
#8  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fffe7dfe700 (LWP 1782)):
#0  0x00007ffff592e945 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d2d6 in epicsEventWait (pevent=0x6b7400)
    at ../../src/osi/os/posix/osdEvent.c:103
#2  0x00007ffff6755bb9 in epicsEventMustWait (id=<optimized out>)
    at ../../src/osi/epicsEvent.cpp:125
#3  0x00007ffff6c1526c in dbCaTask (arg=<optimized out>)
    at ../../../src/ioc/db/dbCa.c:1038
#4  0x00007ffff675b41c in start_routine (arg=0x6b74e0)
    at ../../src/osi/os/posix/osdThread.c:403
#5  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fffe7fff700 (LWP 1781)):
#0  0x00007ffff592e945 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d2d6 in epicsEventWait (pevent=0x6b6e70)
    at ../../src/osi/os/posix/osdEvent.c:103
#2  0x00007ffff6755bb9 in epicsEventMustWait (id=<optimized out>)
    at ../../src/osi/epicsEvent.cpp:125
#3  0x00007ffff6c12cf5 in callbackTask (arg=<optimized out>)
---Type <return> to continue, or q <return> to quit---
    at ../../../src/ioc/db/callback.c:166
#4  0x00007ffff675b41c in start_routine (arg=0x6b6f40)
    at ../../src/osi/os/posix/osdThread.c:403
#5  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7ffff437c700 (LWP 1780)):
#0  0x00007ffff592e945 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d2d6 in epicsEventWait (pevent=0x6b6a40)
    at ../../src/osi/os/posix/osdEvent.c:103
#2  0x00007ffff6755bb9 in epicsEventMustWait (id=<optimized out>)
    at ../../src/osi/epicsEvent.cpp:125
#3  0x00007ffff6c12cf5 in callbackTask (arg=<optimized out>)
    at ../../../src/ioc/db/callback.c:166
#4  0x00007ffff675b41c in start_routine (arg=0x6b6b10)
    at ../../src/osi/os/posix/osdThread.c:403
#5  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7ffff457d700 (LWP 1779)):
#0  0x00007ffff592e945 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d2d6 in epicsEventWait (pevent=0x6b6610)
    at ../../src/osi/os/posix/osdEvent.c:103
#2  0x00007ffff6755bb9 in epicsEventMustWait (id=<optimized out>)
    at ../../src/osi/epicsEvent.cpp:125
#3  0x00007ffff6c12cf5 in callbackTask (arg=<optimized out>)
    at ../../../src/ioc/db/callback.c:166
#4  0x00007ffff675b41c in start_routine (arg=0x6b66e0)
    at ../../src/osi/os/posix/osdThread.c:403
#5  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7ffff467e700 (LWP 1778)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6b6070, 
    timeout=1.7976931348623157e+308) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff6755dac in epicsEvent::wait (this=<optimized out>, 
    timeOut=<optimized out>) at ../../src/osi/epicsEvent.cpp:75
#3  0x00007ffff67666f3 in timerQueueActive::run (this=0x6b5d80)
    at ../../src/timer/timerQueueActive.cpp:95
#4  0x00007ffff6755039 in epicsThreadCallEntryPoint (pPvt=0x6b5e50)
    at ../../src/osi/epicsThread.cpp:83
#5  0x00007ffff675b41c in start_routine (arg=0x6b62a0)
    at ../../src/osi/os/posix/osdThread.c:403
#6  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#7  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7ffff7f22700 (LWP 1777)):
#0  0x00007ffff592ecf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d279 in epicsEventWaitWithTimeout (pevent=0x6b5760, 
    timeout=6) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff6763844 in twdTask (arg=<optimized out>)
    at ../../src/taskwd/taskwd.c:126
#3  0x00007ffff675b41c in start_routine (arg=0x6b5840)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7ffff7fa3700 (LWP 1776)):
#0  0x00007ffff592e945 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff675d2d6 in epicsEventWait (pevent=0x6a01d0)
---Type <return> to continue, or q <return> to quit---
    at ../../src/osi/os/posix/osdEvent.c:103
#2  0x00007ffff6755bb9 in epicsEventMustWait (id=<optimized out>)
    at ../../src/osi/epicsEvent.cpp:125
#3  0x00007ffff6743d47 in errlogThread () at ../../src/error/errlog.c:561
#4  0x00007ffff675b41c in start_routine (arg=0x6b52d0)
    at ../../src/osi/os/posix/osdThread.c:403
#5  0x00007ffff592ae25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ffff5c3734d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7ffff7fa5740 (LWP 1766)):
#0  0x00007ffff5c287fd in read () from /lib64/libc.so.6
#1  0x00007ffff5707097 in rl_getc () from /lib64/libreadline.so.6
#2  0x00007ffff57078e7 in rl_read_key () from /lib64/libreadline.so.6
#3  0x00007ffff56f2b2f in readline_internal_char ()
   from /lib64/libreadline.so.6
#4  0x00007ffff56f31e5 in readline () from /lib64/libreadline.so.6
#5  0x00007ffff6759f1a in osdReadline (context=0x6d5b20, 
    prompt=0x7ffff67702ee "epics> ")
    at ../../src/osi/os/default/gnuReadline.c:67
#6  epicsReadline (prompt=0x7ffff67702ee "epics> ", context=0x6d5b20)
    at ../../src/osi/epicsReadline.c:86
#7  0x00007ffff674ae28 in iocshBody (pathname=<optimized out>, 
    commandLine=0x0, macros=0x0) at ../../src/iocsh/iocsh.cpp:618
#8  0x000000000040476f in main (argc=0, argv=0x7fffffffdd50)
    at ../softMain.cpp:231
(gdb)

I pulled and built pva2pva, but haven't necessarily rebuilt all of the other modules, would that matter?

Note that I'm not getting a deadlock, it's a seg-fault, but it does occur inside the memory allocator. It's possible this is a RHEL GLIBC bug, but I have no idea how to go about proving or diagnosing that.

mdavidsaver commented 6 years ago

@anjohnson From the line numbers it looks like you're building from the main repo, not from my repo (https://github.com/mdavidsaver/pva2pva). So you haven't picked up the fix mdavidsaver/pva2pva@b6b6ac330546c4537ec50b8c70789383f098ca4b

bhill-slac commented 6 years ago

I've run over 10M iterations of the X1 record on RHEL5, RHEL6, and RHEL7 w/o any failures. Thanks Michael! P.S. My version of pvAccessCPP was 6.0.0 plus local build support plus Michael's fixes for pva provider registration and shutdown PVA server via epicsAtExit(). For pva2pva I cherry-picked 6e77fc8, 1635e75, and b6b6ac3 from his github master branch.

mdavidsaver commented 6 years ago

So at least I haven't made things worse. I've pushed the fixes to the main repo.

anjohnson commented 6 years ago

Looks good, no longer crashing with the tip of the master branch of pva2pva.

Thanks!

mdavidsaver commented 6 years ago

Reopening until fix is released

mdavidsaver commented 5 years ago

Released with Base 7.0.2