search

epics-base / pvAccessCPP

pvAccessCPP is an EPICS V4 C++ module
https://epics-base.github.io/pvAccessCPP/
Other
10 stars 22 forks source link

pvput segfaults for bo over ca #114

Closed brunoseivam closed 5 years ago

brunoseivam commented 6 years ago

Maybe it is related to #93 ?

$ cat test.db 
record(bo, "TEST") {
    field(ZNAM, "Zero")
    field(ONAM, "One")
    field(VAL,  "0")
    field(PINI, "YES")
}

$ ~/epics/base/bin/linux-x86_64/softIoc -d test.db 
Starting iocInit
############################################################################
## EPICS R7.0.1.2-DEV
## EPICS Base built Jul 10 2018
############################################################################
cas warning: Configured TCP port was unavailable.
cas warning: Using dynamically assigned TCP port 45739,
cas warning: but now two or more servers share the same UDP port.
cas warning: Depending on your IP kernel this server may not be
cas warning: reachable with UDP unicast (a host's IP in EPICS_CA_ADDR_LIST)
iocRun: All initialization complete
epics> dbl
TEST
epics>

On another terminal:

$ ~/epics/base/bin/linux-x86_64/pvput -V
pvput v6.0.1-SNAPSHOT

$ ~/epics/base/bin/linux-x86_64/pvput -p ca TEST Zero 2> /dev/null
Old : TEST                           Zero
fish: “~/epics/base/bin/linux-x86_64/p…” terminated by signal SIGSEGV (Address boundary error)

$ ~/epics/base/bin/linux-x86_64/pvput -p ca TEST 0 2> /dev/null
Old : TEST                           Zero
fish: “~/epics/base/bin/linux-x86_64/p…” terminated by signal SIGSEGV (Address boundary error)

$ gdb ~/epics/base/bin/linux-x86_64/pvput
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/martins/epics/base/bin/linux-x86_64/pvput...done.
(gdb) r -p ca TEST Zero
Starting program: /home/martins/epics/epics-base/bin/linux-x86_64/pvput -p ca TEST Zero
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff587e700 (LWP 25202)]
[Thread 0x7ffff587e700 (LWP 25202) exited]
[New Thread 0x7ffff483d700 (LWP 25203)]
[New Thread 0x7ffff7fca700 (LWP 25204)]
[New Thread 0x7ffff463c700 (LWP 25205)]
[New Thread 0x7ffff453b700 (LWP 25206)]
[New Thread 0x7ffff4137700 (LWP 25207)]
Old : TEST                           Zero

Thread 6 "CAC-TCP-recv" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff453b700 (LWP 25206)]
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007ffff7bb0fe2 in epics::pvData::detail::PVVectorStorage<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, epics::pvData::PVScalarArray>::reuse (this=0x65b840)
    at /home/martins/epics/epics-base/include/pv/pvData.h:1125
#2  epics::pvAccess::ca::ca_get_labels_handler (args=...) at ../caChannel.cpp:103
#3  0x00007ffff666ed0a in getCallback::completion (this=<optimized out>, guard=..., type=24, 
    count=1, pData=0x7fffe800a498) at ../getCallback.cpp:62
#4  0x00007ffff6668322 in netReadNotifyIO::completion (this=0x7fffe400c818, guard=..., 
    recycle=..., type=24, count=1, pData=0x7fffe800a498) at ../netReadNotifyIO.cpp:69
#5  0x00007ffff6647cb2 in cac::readNotifyRespAction (this=0x636850, iiu=..., hdr=..., 
    pMsgBdy=0x7fffe800a498) at ../cac.cpp:905
#6  0x00007ffff6664c0e in tcpiiu::processIncoming (this=0x7fffe80008c0, currentTime=..., mgr=...)
    at ../tcpiiu.cpp:1264
#7  0x00007ffff6666ed9 in tcpRecvThread::run (this=0x7fffe80009a8) at ../tcpiiu.cpp:519
#8  0x00007ffff7228ca9 in epicsThreadCallEntryPoint (pPvt=0x7fffe80009b0)
    at ../../src/osi/epicsThread.cpp:83
#9  0x00007ffff722f65c in start_routine (arg=0x7fffe8009ea0)
    at ../../src/osi/os/posix/osdThread.c:403
#10 0x00007ffff640a6ba in start_thread (arg=0x7ffff453b700) at pthread_create.c:333
#11 0x00007ffff699241d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) up 2
#2  epics::pvAccess::ca::ca_get_labels_handler (args=...) at ../caChannel.cpp:103
103             PVStringArray::svector labels(labelsArray->reuse());
(gdb) p labelsArray
$1 = (epics::pvData::PVStringArray *) 0x65b840
(gdb) p *labelsArray
$2 = {<epics::pvData::detail::PVVectorStorage<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, epics::pvData::PVScalarArray>> = {<epics::pvData::PVScalarArray> = {<epics::pvData::PVArray> = {<epics::pvData::PVField> = {<epics::pvData::Serializable> = <invalid address>, <std::tr1::enable_shared_from_this<epics::pvData::PVField>> = {
            _M_weak_this = std::tr1::weak_ptr (count 1, weak 1) 0x7ffff6c4fb78 <main_arena+88>}, 
          static num_instances = 14, fieldName = "choices", parent = 0x65b770, 
          field = std::tr1::shared_ptr (count 7, weak 1) 0x62f9a0, fieldOffset = 3, 
          nextFieldOffset = 4, immutable = false, 
          postHandler = std::tr1::shared_ptr (empty) 0x0}, <epics::pvData::SerializableArray> = {<epics::pvData::Serializable> = {_vptr.Serializable = 0x65b420}, <No data fields>}, 
        capacityMutable = true}, <No data fields>}, <No data fields>}, 
  static typeCode = epics::pvData::pvString, 
  value = {<epics::pvData::detail::shared_vector_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const>> = {m_sdata = std::tr1::shared_ptr (empty) 0x0, 
      m_offset = 0, m_count = 0, m_total = 0}, <No data fields>}}
(gdb) p labelsArray->value
$3 = {<epics::pvData::detail::shared_vector_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const>> = {m_sdata = std::tr1::shared_ptr (empty) 0x0, m_offset = 0, 
    m_count = 0, m_total = 0}, <No data fields>}
(gdb) p labelsArray->value.m_sdata
$4 = std::tr1::shared_ptr (empty) 0x0
(gdb)
mdavidsaver commented 6 years ago

Maybe it is related to #93 ?

This works for me with PVA (you can run 'softIocPVA' to test), so I think this is a caProvider bug. @mrkraimer is this a case which fixed by #96.

As a general request. When reporting a crash, if at all possible please try reproduce with a debug build (eg. 'linux-x86_64-debug') as this will yield better information.

mdavidsaver commented 6 years ago

valgrind reports this as a use after free

Thread 6 CAC-TCP-recv:
Invalid read of size 8
   at 0x4E801C7: epics::pvData::detail::PVVectorStorage<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, epics::pvData::PVScalarArray>::reuse() (pvData.h:1127)
   by 0x4E680DE: epics::pvAccess::ca::ca_get_labels_handler(event_handler_args) (caChannel.cpp:103)
   by 0x599D47C: getCallback::completion(epicsGuard<epicsMutex>&, unsigned int, unsigned long, void const*) (getCallback.cpp:62)
   by 0x59942DE: netReadNotifyIO::completion(epicsGuard<epicsMutex>&, cacRecycle&, unsigned int, unsigned long, void const*) (netReadNotifyIO.cpp:69)
   by 0x597468C: cac::readNotifyRespAction(callbackManager&, tcpiiu&, epicsTime const&, caHdrLargeArray const&, void*) (cac.cpp:905)                                                                                                               
   by 0x59756CE: cac::executeResponse(callbackManager&, tcpiiu&, epicsTime const&, caHdrLargeArray&, char*) (cac.cpp:1216)                                                                                                                         
   by 0x599018B: tcpiiu::processIncoming(epicsTime const&, callbackManager&) (tcpiiu.cpp:1264)                       
   by 0x598DAB2: tcpRecvThread::run() (tcpiiu.cpp:519)
   by 0x5BF72C2: epicsThreadCallEntryPoint (epicsThread.cpp:95)
   by 0x5BFF639: start_routine (osdThread.c:411)
   by 0x6A79493: start_thread (pthread_create.c:333)
   by 0x67BBACE: clone (clone.S:97)
 Address 0x765c710 is 0 bytes inside a block of size 176 free'd
   at 0x4C2D2DB: operator delete(void*) (vg_replace_malloc.c:576)
   by 0x5689F28: epics::pvData::PVValueArray<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::~PVValueArray() (pvData.h:1162)
   by 0x568B8C5: std::_Sp_counted_ptr<epics::pvData::PVValueArray<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:372)
   by 0x1256BD: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:150)
   by 0x123C94: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:662)
   by 0x121D5D: std::__shared_ptr<epics::pvData::PVField, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:928)
   by 0x121D79: std::shared_ptr<epics::pvData::PVField>::~shared_ptr() (shared_ptr.h:93)
   by 0x12943D: void std::_Destroy<std::shared_ptr<epics::pvData::PVField> >(std::shared_ptr<epics::pvData::PVField>*) (stl_construct.h:93)
   by 0x128862: void std::_Destroy_aux<false>::__destroy<std::shared_ptr<epics::pvData::PVField>*>(std::shared_ptr<epics::pvData::PVField>*, std::shared_ptr<epics::pvData::PVField>*) (stl_construct.h:103)
   by 0x12736E: void std::_Destroy<std::shared_ptr<epics::pvData::PVField>*>(std::shared_ptr<epics::pvData::PVField>*, std::shared_ptr<epics::pvData::PVField>*) (stl_construct.h:126)
   by 0x126033: void std::_Destroy<std::shared_ptr<epics::pvData::PVField>*, std::shared_ptr<epics::pvData::PVField> >(std::shared_ptr<epics::pvData::PVField>*, std::shared_ptr<epics::pvData::PVField>*, std::allocator<std::shared_ptr<epics::pvData::PVField> >&) (stl_construct.h:151)
   by 0x566E066: std::vector<std::shared_ptr<epics::pvData::PVField>, std::allocator<std::shared_ptr<epics::pvData::PVField> > >::~vector() (stl_vector.h:426)
 Block was alloc'd at
   at 0x4C2C21F: operator new(unsigned long) (vg_replace_malloc.c:334)
   by 0x567C122: epics::pvData::PVDataCreate::createPVScalarArray(std::shared_ptr<epics::pvData::ScalarArray const> const&) (PVDataCreateFactory.cpp:515)
   by 0x567B0EC: epics::pvData::PVDataCreate::createPVField(std::shared_ptr<epics::pvData::Field const> const&) (PVDataCreateFactory.cpp:368)
   by 0x566B860: epics::pvData::PVStructure::PVStructure(std::shared_ptr<epics::pvData::Structure const> const&) (PVStructure.cpp:38)
   by 0x567C4D6: epics::pvData::PVDataCreate::createPVStructure(std::shared_ptr<epics::pvData::Structure const> const&) (PVDataCreateFactory.cpp:546)
   by 0x567B158: epics::pvData::PVDataCreate::createPVField(std::shared_ptr<epics::pvData::Field const> const&) (PVDataCreateFactory.cpp:372)
   by 0x566B860: epics::pvData::PVStructure::PVStructure(std::shared_ptr<epics::pvData::Structure const> const&) (PVStructure.cpp:38)
   by 0x567C4D6: epics::pvData::PVDataCreate::createPVStructure(std::shared_ptr<epics::pvData::Structure const> const&) (PVDataCreateFactory.cpp:546)
   by 0x4E6897A: epics::pvAccess::ca::createPVStructure(std::shared_ptr<epics::pvAccess::ca::CAChannel> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<epics::pvData::PVStructure>) (caChannel.cpp:173)
   by 0x4E68D43: epics::pvAccess::ca::createPVStructure(std::shared_ptr<epics::pvAccess::ca::CAChannel> const&, long, std::shared_ptr<epics::pvData::PVStructure>) (caChannel.cpp:233)
   by 0x4E6F33B: epics::pvAccess::ca::CAChannelPut::activate() (caChannel.cpp:1311)
   by 0x4E6B6E7: epics::pvAccess::ca::CAChannel::createChannelPut(std::shared_ptr<epics::pvAccess::ChannelPutRequester> const&, std::shared_ptr<epics::pvData::PVStructure> const&) (caChannel.cpp:615)
mrkraimer commented 6 years ago

Why is this assigned to me? What is in the master branch is the ca provider code before I make major changes to the code that maps DBD data to pvData. But this code was never merged (pull request #96). Since I was not allowed to merge these changes I should not have this assigned to me.

brunoseivam commented 6 years ago

@mdavidsaver I can confirm that it works when using softIocPVA and pvput without -p ca. I tested with the debug build and gdb but it seems to yield the same information as before. It seems that there is a race condition because sometimes the put operation goes through without any errors.

brunoseivam commented 6 years ago

I poked around a bit more. Some context first: I discovered this issue while using ClientChannel::PutCallback for doing a put.

So I was allocating a PutCallback on the stack and got the same SEGFAULT as with pvput. Now, if I allocate a PutCallback on the heap instead (and leak it, just to test), the SEGFAULT never happens. So could it be that PutCallback is indicating that it is done too early?

Here's how I was using it originally, for context:

class PutCallback : public ClientChannel::PutCallback
{
    PVStructure::const_shared_pointer const & value;
    BitSet const & tosend;
    epicsEvent done;
    Operation operation;

public:
    PutCallback(pvac::ClientChannel& channel,
                PVStructure::const_shared_pointer const & request,
                PVStructure::const_shared_pointer const & value,
                BitSet const & bitSet) :
         value(value), tosend(bitSet),
         operation(channel.put(this, request)) // put() starts here
    {}

    virtual void putBuild(const StructureConstPtr &build,
            ClientChannel::PutCallback::Args& args) OVERRIDE FINAL
    {
        args.root = value;
        args.tosend = tosend;
    }

    virtual void putDone(const pvac::PutEvent &evt) OVERRIDE FINAL
    {
        switch(evt.event) {
        case pvac::PutEvent::Fail:
            fprintf(stderr, "PutCallback Error: %s\n", evt.message.c_str());
            break;
        case pvac::PutEvent::Cancel:
            fprintf(stderr, "PutCallback Canceled\n");
            break;
        case pvac::PutEvent::Success:
            break;
        }

        done.signal();
    }

    void wait(double timeout = 0.0)
    {
        if (timeout)
            done.wait(timeout);
        else
            done.wait();
    }
};

And the call site:

void Subscription::put(PVStructurePtr const & value, BitSet const & bitSet)
{
    if (mConnected) {
        // leaks, doesn't SEGFAULT
        PutCallback *pc = new PutCallback(mChannel, mRequest, value, bitSet);
        // doesn't leak, SEGFAULTs
        //PutCallback pc(mChannel, mRequest, value, bitSet);
        pc->wait();
    }
}
mdavidsaver commented 6 years ago

@brunoseivam The existing code is known to have a number of problems. Can you test Marty's #96 ? This is a large change, so imo. there isn't much point to spending time with the existing code.

brunoseivam commented 6 years ago

I'll do that tomorrow and report back

brunoseivam commented 6 years ago

Tests with @mrkraimer's fork.

Merge

$ pwd
/home/martins/epics/epics-base/modules/pvAccess
$ git branch
* master
$ git checkout -b test-mrkraimer
$ git remote add mrkraimer https://github.com/mrkraimer/pvAccessCPP
$ git pull mrkraimer master
From https://github.com/mrkraimer/pvAccessCPP
 * branch            master     -> FETCH_HEAD
 * [new branch]      master     -> mrkraimer/master
Removing src/ca/pv/caStatus.h
Removing src/ca/caStatus.cpp
Merge made by the 'recursive' strategy.
 Makefile                      |    5 +
 caProvider.md                 |  636 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 src/ca/Makefile               |    7 +-
 src/ca/caChannel.cpp          | 1620 +++++++++++++++++++++++++++++++------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 src/ca/caChannel.h            |  181 ++++++-----------------
 src/ca/caProvider.cpp         |   69 +++++----
 src/ca/caProviderPvt.h        |   17 ++-
 src/ca/caStatus.cpp           |   69 ---------
 src/ca/dbdToPv.cpp            |  800 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 src/ca/dbdToPv.h              |  136 +++++++++++++++++
 src/ca/monitorEventThread.cpp |  113 ++++++++++++++
 src/ca/monitorEventThread.h   |   71 +++++++++
 src/ca/pv/caProvider.h        |   24 ++-
 src/ca/pv/caStatus.h          |   29 ----
 testCa/Makefile               |   47 ++++++
 testCa/pvCaAllTests.c         |   18 +++
 testCa/qemuRunTest.sh         |    1 +
 testCa/rtemsTestHarness.c     |   12 ++
 testCa/testCaProvider.cpp     |  802 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 testCa/testCaProvider.db      |  278 ++++++++++++++++++++++++++++++++++
 20 files changed, 3282 insertions(+), 1653 deletions(-)
 create mode 100644 caProvider.md
 delete mode 100644 src/ca/caStatus.cpp
 create mode 100644 src/ca/dbdToPv.cpp
 create mode 100644 src/ca/dbdToPv.h
 create mode 100644 src/ca/monitorEventThread.cpp
 create mode 100644 src/ca/monitorEventThread.h
 delete mode 100644 src/ca/pv/caStatus.h
 create mode 100644 testCa/Makefile
 create mode 100644 testCa/pvCaAllTests.c
 create mode 100755 testCa/qemuRunTest.sh
 create mode 100644 testCa/rtemsTestHarness.c
 create mode 100644 testCa/testCaProvider.cpp
 create mode 100644 testCa/testCaProvider.db
$ cd ../../
$ make distclean all

Test

$ ~/epics/base/bin/linux-x86_64-debug/pvput -p ca TEST 0
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "TEST", Connecting to: 35.9.126.36:5064, Ignored: 192.168.100.1:5064"
    Source File: ../cac.cpp line 1320
    Current Time: Wed Jul 11 2018 10:23:35.641952040
..................................................................
Old : TEST                           One
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "TEST", Connecting to: 35.9.126.36:5064, Ignored: 10.42.0.1:5064"
    Source File: ../cac.cpp line 1320
    Current Time: Wed Jul 11 2018 10:23:35.642419448
..................................................................
New : TEST                           Zero
CAChannelProvider::~CAChannelProvider() returning
MonitorEventThread::~MonitorEventThread()
$ ~/epics/base/bin/linux-x86_64-debug/pvput -p ca TEST 1
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "TEST", Connecting to: 35.9.126.36:5064, Ignored: 192.168.100.1:5064"
    Source File: ../cac.cpp line 1320
    Current Time: Wed Jul 11 2018 10:23:38.714493647
..................................................................
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "TEST", Connecting to: 35.9.126.36:5064, Ignored: 10.42.0.1:5064"
    Source File: ../cac.cpp line 1320
    Current Time: Wed Jul 11 2018 10:23:38.714932520
..................................................................
Old : TEST                           Zero
New : TEST                           One
CAChannelProvider::~CAChannelProvider() returning
MonitorEventThread::~MonitorEventThread()
$ ~/epics/base/bin/linux-x86_64-debug/pvput -p ca TEST 0
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "TEST", Connecting to: 35.9.126.36:5064, Ignored: 192.168.100.1:5064"
    Source File: ../cac.cpp line 1320
    Current Time: Wed Jul 11 2018 10:23:41.683213214
..................................................................
Old : TEST                           One
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "TEST", Connecting to: 35.9.126.36:5064, Ignored: 10.42.0.1:5064"
    Source File: ../cac.cpp line 1320
    Current Time: Wed Jul 11 2018 10:23:41.684708730
..................................................................
*** Error in `/home/martins/epics/base/bin/linux-x86_64-debug/pvput': free(): corrupted unsorted chunks: 0x0000000001c998f0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f3a489237e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f3a4892c37a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f3a4893053c]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZNKSt3tr111_Sp_deleterIN5epics8pvAccess2ca7DbdToPvEEclEPS4_+0x2a)[0x7f3a49f5418c]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZNSt3tr121_Sp_counted_base_implIPN5epics8pvAccess2ca7DbdToPvENS_11_Sp_deleterIS4_EELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0x27)[0x7f3a49f54123]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_ZNSt3tr116_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x42)[0x4274fc]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_ZNSt3tr114__shared_countILN9__gnu_cxx12_Lock_policyE2EED1Ev+0x27)[0x42560d]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZNSt3tr112__shared_ptrIN5epics8pvAccess2ca7DbdToPvELN9__gnu_cxx12_Lock_policyE2EED1Ev+0x1c)[0x7f3a49f30dee]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZNSt3tr110shared_ptrIN5epics8pvAccess2ca7DbdToPvEED2Ev+0x18)[0x7f3a49f30e0a]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZN5epics8pvAccess2ca12CAChannelPutD1Ev+0x7b)[0x7f3a49f2d38f]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZN5epics8pvAccess2ca12CAChannelPutD0Ev+0x18)[0x7f3a49f2d524]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZNKSt3tr111_Sp_deleterIN5epics8pvAccess2ca12CAChannelPutEEclEPS4_+0x2e)[0x7f3a49f40004]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZNSt3tr121_Sp_counted_base_implIPN5epics8pvAccess2ca12CAChannelPutENS_11_Sp_deleterIS4_EELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0x27)[0x7f3a49f3fd37]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_ZNSt3tr116_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x42)[0x4274fc]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_ZNSt3tr114__shared_countILN9__gnu_cxx12_Lock_policyE2EED1Ev+0x27)[0x42560d]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZNSt3tr112__shared_ptrIN5epics8pvAccess10ChannelPutELN9__gnu_cxx12_Lock_policyE2EED2Ev+0x1c)[0x7f3a49f30c60]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZNSt3tr110shared_ptrIN5epics8pvAccess10ChannelPutEED1Ev+0x18)[0x7f3a49f30c7c]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(+0x219542)[0x7f3a49b19542]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(+0x219670)[0x7f3a49b19670]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(+0x21bdec)[0x7f3a49b1bdec]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(+0x21bd83)[0x7f3a49b1bd83]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_ZNSt3tr116_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x42)[0x4274fc]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_ZNSt3tr114__shared_countILN9__gnu_cxx12_Lock_policyE2EED1Ev+0x27)[0x42560d]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(+0x21998c)[0x7f3a49b1998c]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(+0x2199a8)[0x7f3a49b199a8]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(+0x21bac2)[0x7f3a49b1bac2]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(+0x21bd21)[0x7f3a49b1bd21]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_ZNSt3tr116_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x42)[0x4274fc]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_ZNSt3tr114__shared_countILN9__gnu_cxx12_Lock_policyE2EED1Ev+0x27)[0x42560d]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(_ZNSt3tr112__shared_ptrIN4pvac9Operation4ImplELN9__gnu_cxx12_Lock_policyE2EED1Ev+0x1c)[0x7f3a49b0f878]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(_ZNSt3tr110shared_ptrIN4pvac9Operation4ImplEED1Ev+0x18)[0x7f3a49b0f894]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(_ZN4pvac9OperationD1Ev+0x18)[0x7f3a49b0d9a2]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(_ZN4pvac13ClientChannel3getEdNSt3tr110shared_ptrIKN5epics6pvData11PVStructureEEE+0xc4)[0x7f3a49b15278]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(main+0xfca)[0x422bb7]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f3a488cc830]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_start+0x29)[0x41d479]
======= Memory map: ========
00400000-00436000 r-xp 00000000 08:05 12584255                           /home/martins/epics/epics-base/bin/linux-x86_64-debug/pvput
00635000-00636000 r--p 00035000 08:05 12584255                           /home/martins/epics/epics-base/bin/linux-x86_64-debug/pvput
00636000-00637000 rw-p 00036000 08:05 12584255                           /home/martins/epics/epics-base/bin/linux-x86_64-debug/pvput
01c58000-01cb6000 rw-p 00000000 00:00 0                                  [heap]
7f3a28000000-7f3a28021000 rw-p 00000000 00:00 0 
7f3a28021000-7f3a2c000000 ---p 00000000 00:00 0 
7f3a30000000-7f3a30021000 rw-p 00000000 00:00 0 
7f3a30021000-7f3a34000000 ---p 00000000 00:00 0 
7f3a34000000-7f3a34021000 rw-p 00000000 00:00 0 
7f3a34021000-7f3a38000000 ---p 00000000 00:00 0 
7f3a38000000-7f3a38028000 rw-p 00000000 00:00 0 
7f3a38028000-7f3a3c000000 ---p 00000000 00:00 0 
7f3a3c000000-7f3a3c021000 rw-p 00000000 00:00 0 
7f3a3c021000-7f3a40000000 ---p 00000000 00:00 0 
7f3a40000000-7f3a40021000 rw-p 00000000 00:00 0 
7f3a40021000-7f3a44000000 ---p 00000000 00:00 0 
7f3a45b0f000-7f3a45b10000 ---p 00000000 00:00 0 
7f3a45b10000-7f3a45b90000 rw-p 00000000 00:00 0 
7f3a45b90000-7f3a45ba7000 r-xp 00000000 08:05 11540472                   /lib/x86_64-linux-gnu/libresolv-2.23.so
7f3a45ba7000-7f3a45da7000 ---p 00017000 08:05 11540472                   /lib/x86_64-linux-gnu/libresolv-2.23.so
7f3a45da7000-7f3a45da8000 r--p 00017000 08:05 11540472                   /lib/x86_64-linux-gnu/libresolv-2.23.so
7f3a45da8000-7f3a45da9000 rw-p 00018000 08:05 11540472                   /lib/x86_64-linux-gnu/libresolv-2.23.so
7f3a45da9000-7f3a45dab000 rw-p 00000000 00:00 0 
7f3a45dab000-7f3a45db0000 r-xp 00000000 08:05 11540478                   /lib/x86_64-linux-gnu/libnss_dns-2.23.so
7f3a45db0000-7f3a45fb0000 ---p 00005000 08:05 11540478                   /lib/x86_64-linux-gnu/libnss_dns-2.23.so
7f3a45fb0000-7f3a45fb1000 r--p 00005000 08:05 11540478                   /lib/x86_64-linux-gnu/libnss_dns-2.23.so
7f3a45fb1000-7f3a45fb2000 rw-p 00006000 08:05 11540478                   /lib/x86_64-linux-gnu/libnss_dns-2.23.so
7f3a45fb2000-7f3a45fb4000 r-xp 00000000 08:05 11538999                   /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
7f3a45fb4000-7f3a461b3000 ---p 00002000 08:05 11538999                   /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
7f3a461b3000-7f3a461b4000 r--p 00001000 08:05 11538999                   /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
7f3a461b4000-7f3a461b5000 rw-p 00002000 08:05 11538999                   /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
7f3a461b5000-7f3a46236000 rw-p 00000000 00:00 0 
7f3a46236000-7f3a46237000 ---p 00000000 00:00 0 
7f3a46237000-7f3a46337000 rw-p 00000000 00:00 0 
7f3a46337000-7f3a46338000 ---p 00000000 00:00 0 
7f3a46338000-7f3a46538000 rw-p 00000000 00:00 0 
7f3a46538000-7f3a46539000 ---p 00000000 00:00 0 
7f3a46539000-7f3a46639000 rw-p 00000000 00:00 0 
7f3a46639000-7f3a4663a000 ---p 00000000 00:00 0 
7f3a4663a000-7f3a4683a000 rw-p 00000000 00:00 0 
7f3a4683a000-7f3a46845000 r-xp 00000000 08:05 11540481                   /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f3a46845000-7f3a46a44000 ---p 0000b000 08:05 11540481                   /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f3a46a44000-7f3a46a45000 r--p 0000a000 08:05 11540481                   /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f3a46a45000-7f3a46a46000 rw-p 0000b000 08:05 11540481                   /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f3a46a46000-7f3a46a4c000 rw-p 00000000 00:00 0 
7f3a46a4c000-7f3a46a57000 r-xp 00000000 08:05 11540485                   /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f3a46a57000-7f3a46c56000 ---p 0000b000 08:05 11540485                   /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f3a46c56000-7f3a46c57000 r--p 0000a000 08:05 11540485                   /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f3a46c57000-7f3a46c58000 rw-p 0000b000 08:05 11540485                   /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f3a46c58000-7f3a46c6e000 r-xp 00000000 08:05 11540465                   /lib/x86_64-linux-gnu/libnsl-2.23.so
7f3a46c6e000-7f3a46e6d000 ---p 00016000 08:05 11540465                   /lib/x86_64-linux-gnu/libnsl-2.23.so
7f3a46e6d000-7f3a46e6e000 r--p 00015000 08:05 11540465                   /lib/x86_64-linux-gnu/libnsl-2.23.so
7f3a46e6e000-7f3a46e6f000 rw-p 00016000 08:05 11540465                   /lib/x86_64-linux-gnu/libnsl-2.23.so
7f3a46e6f000-7f3a46e71000 rw-p 00000000 00:00 0 
7f3a46e71000-7f3a46e79000 r-xp 00000000 08:05 11540476                   /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f3a46e79000-7f3a47078000 ---p 00008000 08:05 11540476                   /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f3a47078000-7f3a47079000 r--p 00007000 08:05 11540476                   /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f3a47079000-7f3a4707a000 rw-p 00008000 08:05 11540476                   /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f3a4707a000-7f3a4707b000 ---p 00000000 00:00 0 
7f3a4707b000-7f3a4787b000 rw-p 00000000 00:00 0 
7f3a4787b000-7f3a478a0000 r-xp 00000000 08:05 11539066                   /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f3a478a0000-7f3a47a9f000 ---p 00025000 08:05 11539066                   /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f3a47a9f000-7f3a47aa3000 r--p 00024000 08:05 11539066                   /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f3a47aa3000-7f3a47aa4000 rw-p 00028000 08:05 11539066                   /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f3a47aa4000-7f3a47bac000 r-xp 00000000 08:05 11534513                   /lib/x86_64-linux-gnu/libm-2.23.so
7f3a47bac000-7f3a47dab000 ---p 00108000 08:05 11534513                   /lib/x86_64-linux-gnu/libm-2.23.so
7f3a47dab000-7f3a47dac000 r--p 00107000 08:05 11534513                   /lib/x86_64-linux-gnu/libm-2.23.so
7f3a47dac000-7f3a47dad000 rw-p 00108000 08:05 11534513                   /lib/x86_64-linux-gnu/libm-2.23.so
7f3a47dad000-7f3a47db0000 r-xp 00000000 08:05 11540470                   /lib/x86_64-linux-gnu/libdl-2.23.so
7f3a47db0000-7f3a47faf000 ---p 00003000 08:05 11540470                   /lib/x86_64-linux-gnu/libdl-2.23.so
7f3a47faf000-7f3a47fb0000 r--p 00002000 08:05 11540470                   /lib/x86_64-linux-gnu/libdl-2.23.so
7f3a47fb0000-7f3a47fb1000 rw-p 00003000 08:05 11540470                   /lib/x86_64-linux-gnu/libdl-2.23.so
7f3a47fb1000-7f3a47fb8000 r-xp 00000000 08:05 11540486                   /lib/x86_64-linux-gnu/librt-2.23.so
7f3a47fb8000-7f3a481b7000 ---p 00007000 08:05 11540486                   /lib/x86_64-linux-gnu/librt-2.23.so
7f3a481b7000-7f3a481b8000 r--p 00006000 08:05 11540486                   /lib/x86_64-linux-gnu/librt-2.23.so
7f3a481b8000-7f3a481b9000 rw-p 00007000 08:05 11540486                   /lib/x86_64-linux-gnu/librt-2.23.so
7f3a481b9000-7f3a481f6000 r-xp 00000000 08:05 11539045                   /lib/x86_64-linux-gnu/libreadline.so.6.3
7f3a481f6000-7f3a483f6000 ---p 0003d000 08:05 11539045                   /lib/x86_64-linux-gnu/libreadline.so.6.3
7f3a483f6000-7f3a483f8000 r--p 0003d000 08:05 11539045                   /lib/x86_64-linux-gnu/libreadline.so.6.3
7f3a483f8000-7f3a483fe000 rw-p 0003f000 08:05 11539045                   /lib/x86_64-linux-gnu/libreadline.so.6.3
7f3a483fe000-7f3a483ff000 rw-p 00000000 00:00 0 
7f3a483ff000-7f3a48417000 r-xp 00000000 08:05 11540467                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7f3a48417000-7f3a48616000 ---p 00018000 08:05 11540467                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7f3a48616000-7f3a48617000 r--p 00017000 08:05 11540467                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7f3a48617000-7f3a48618000 rw-p 00018000 08:05 11540467                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7f3a48618000-7f3a4861c000 rw-p 00000000 00:00 0 
7f3a4861c000-7f3a486a6000 r-xp 00000000 08:05 12583712                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libca.so.4.13.1
7f3a486a6000-7f3a488a6000 ---p 0008a000 08:05 12583712                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libca.so.4.13.1
7f3a488a6000-7f3a488a9000 r--p 0008a000 08:05 12583712                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libca.so.4.13.1
7f3a488a9000-7f3a488ac000 rw-p 0008d000 08:05 12583712                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libca.so.4.13.1
7f3a488ac000-7f3a48a6c000 r-xp 00000000 08:05 11540468                   /lib/x86_64-linux-gnu/libc-2.23.so
7f3a48a6c000-7f3a48c6c000 ---p 001c0000 08:05 11540468                   /lib/x86_64-linux-gnu/libc-2.23.so
7f3a48c6c000-7f3a48c70000 r--p 001c0000 08:05 11540468                   /lib/x86_64-linux-gnu/libc-2.23.so
7f3a48c70000-7f3a48c72000 rw-p 001c4000 08:05 11540468                   /lib/x86_64-linux-gnu/libc-2.23.so
7f3a48c72000-7f3a48c76000 rw-p 00000000 00:00 0 
7f3a48c76000-7f3a48c8c000 r-xp 00000000 08:05 11538932                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7f3a48c8c000-7f3a48e8b000 ---p 00016000 08:05 11538932                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7f3a48e8b000-7f3a48e8c000 rw-p 00015000 08:05 11538932                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7f3a48e8c000-7f3a48ffe000 r-xp 00000000 08:05 16777365                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f3a48ffe000-7f3a491fe000 ---p 00172000 08:05 16777365                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f3a491fe000-7f3a49208000 r--p 00172000 08:05 16777365                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f3a49208000-7f3a4920a000 rw-p 0017c000 08:05 16777365                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f3a4920a000-7f3a4920e000 rw-p 00000000 00:00 0 
7f3a4920e000-7f3a49290000 r-xp 00000000 08:05 12583672                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libCom.so.3.17.0
7f3a49290000-7f3a4948f000 ---p 00082000 08:05 12583672                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libCom.so.3.17.0
7f3a4948f000-7f3a49492000 r--p 00081000 08:05 12583672                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libCom.so.3.17.0
7f3a49492000-7f3a49495000 rw-p 00084000 08:05 12583672                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libCom.so.3.17.0
7f3a49495000-7f3a49498000 rw-p 00000000 00:00 0 
7f3a49498000-7f3a496ea000 r-xp 00000000 08:05 12584180                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvData.so.7.0.1
7f3a496ea000-7f3a498ea000 ---p 00252000 08:05 12584180                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvData.so.7.0.1
7f3a498ea000-7f3a498f6000 r--p 00252000 08:05 12584180                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvData.so.7.0.1
7f3a498f6000-7f3a49900000 rw-p 0025e000 08:05 12584180                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvData.so.7.0.1
7f3a49900000-7f3a49c7d000 r-xp 00000000 08:05 12584223                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1
7f3a49c7d000-7f3a49e7c000 ---p 0037d000 08:05 12584223                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1
7f3a49e7c000-7f3a49e8b000 r--p 0037c000 08:05 12584223                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1
7f3a49e8b000-7f3a49e9a000 rw-p 0038b000 08:05 12584223                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1
7f3a49e9a000-7f3a49e9c000 rw-p 00000000 00:00 0 
7f3a49e9c000-7f3a49f72000 r-xp 00000000 08:05 12584232                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1
7f3a49f72000-7f3a4a172000 ---p 000d6000 08:05 12584232                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1
7f3a4a172000-7f3a4a174000 r--p 000d6000 08:05 12584232                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1
7f3a4a174000-7f3a4a179000 rw-p 000d8000 08:05 12584232                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1
7f3a4a179000-7f3a4a19f000 r-xp 00000000 08:05 11540466                   /lib/x86_64-linux-gnu/ld-2.23.so
7f3a4a1cb000-7f3a4a1f0000 rw-p 00000000 00:00 0 
7f3a4a1f0000-7f3a4a1f1000 ---p 00000000 00:00 0 
7f3a4a1f1000-7f3a4a2f1000 rw-p 00000000 00:00 0 
7f3a4a2f1000-7f3a4a2f2000 ---p 00000000 00:00 0 
7f3a4a2f2000-7f3a4a37a000 rw-p 00000000 00:00 0 
7f3a4a39b000-7f3a4a39e000 rw-p 00000000 00:00 0 
7f3a4a39e000-7f3a4a39f000 r--p 00025000 08:05 11540466                   /lib/x86_64-linux-gnu/ld-2.23.so
7f3a4a39f000-7f3a4a3a0000 rw-p 00026000 08:05 11540466                   /lib/x86_64-linux-gnu/ld-2.23.so
7f3a4a3a0000-7f3a4a3a1000 rw-p 00000000 00:00 0 
7ffe72ea0000-7ffe72ec1000 rw-p 00000000 00:00 0                          [stack]
7ffe72f62000-7ffe72f65000 r--p 00000000 00:00 0                          [vvar]
7ffe72f65000-7ffe72f67000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
New : fish: “~/epics/base/bin/linux-x86_64-d…” terminated by signal SIGABRT (Abort)
$ ~/epics/base/bin/linux-x86_64-debug/pvput -p ca TEST Zero
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "TEST", Connecting to: 35.9.126.36:5064, Ignored: 192.168.100.1:5064"
    Source File: ../cac.cpp line 1320
    Current Time: Wed Jul 11 2018 10:23:44.411922797
..................................................................
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "TEST", Connecting to: 35.9.126.36:5064, Ignored: 10.42.0.1:5064"
    Source File: ../cac.cpp line 1320
    Current Time: Wed Jul 11 2018 10:23:44.412638592
Old : TEST                           Zero
..................................................................
*** Error in `/home/martins/epics/base/bin/linux-x86_64-debug/pvput': free(): corrupted unsorted chunks: 0x00000000016628f0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f03005707e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f030057937a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f030057d53c]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZNKSt3tr111_Sp_deleterIN5epics8pvAccess2ca7DbdToPvEEclEPS4_+0x2a)[0x7f0301ba118c]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZNSt3tr121_Sp_counted_base_implIPN5epics8pvAccess2ca7DbdToPvENS_11_Sp_deleterIS4_EELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0x27)[0x7f0301ba1123]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_ZNSt3tr116_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x42)[0x4274fc]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_ZNSt3tr114__shared_countILN9__gnu_cxx12_Lock_policyE2EED1Ev+0x27)[0x42560d]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZNSt3tr112__shared_ptrIN5epics8pvAccess2ca7DbdToPvELN9__gnu_cxx12_Lock_policyE2EED1Ev+0x1c)[0x7f0301b7ddee]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZNSt3tr110shared_ptrIN5epics8pvAccess2ca7DbdToPvEED2Ev+0x18)[0x7f0301b7de0a]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZN5epics8pvAccess2ca12CAChannelPutD1Ev+0x7b)[0x7f0301b7a38f]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZN5epics8pvAccess2ca12CAChannelPutD0Ev+0x18)[0x7f0301b7a524]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZNKSt3tr111_Sp_deleterIN5epics8pvAccess2ca12CAChannelPutEEclEPS4_+0x2e)[0x7f0301b8d004]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZNSt3tr121_Sp_counted_base_implIPN5epics8pvAccess2ca12CAChannelPutENS_11_Sp_deleterIS4_EELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0x27)[0x7f0301b8cd37]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_ZNSt3tr116_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x42)[0x4274fc]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_ZNSt3tr114__shared_countILN9__gnu_cxx12_Lock_policyE2EED1Ev+0x27)[0x42560d]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZNSt3tr112__shared_ptrIN5epics8pvAccess10ChannelPutELN9__gnu_cxx12_Lock_policyE2EED2Ev+0x1c)[0x7f0301b7dc60]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1(_ZNSt3tr110shared_ptrIN5epics8pvAccess10ChannelPutEED1Ev+0x18)[0x7f0301b7dc7c]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(+0x219542)[0x7f0301766542]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(+0x219670)[0x7f0301766670]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(+0x21bdec)[0x7f0301768dec]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(+0x21bd83)[0x7f0301768d83]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_ZNSt3tr116_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x42)[0x4274fc]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_ZNSt3tr114__shared_countILN9__gnu_cxx12_Lock_policyE2EED1Ev+0x27)[0x42560d]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(+0x21998c)[0x7f030176698c]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(+0x2199a8)[0x7f03017669a8]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(+0x21bac2)[0x7f0301768ac2]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(+0x21bd21)[0x7f0301768d21]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_ZNSt3tr116_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x42)[0x4274fc]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_ZNSt3tr114__shared_countILN9__gnu_cxx12_Lock_policyE2EED1Ev+0x27)[0x42560d]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(_ZNSt3tr112__shared_ptrIN4pvac9Operation4ImplELN9__gnu_cxx12_Lock_policyE2EED1Ev+0x1c)[0x7f030175c878]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(_ZNSt3tr110shared_ptrIN4pvac9Operation4ImplEED1Ev+0x18)[0x7f030175c894]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(_ZN4pvac9OperationD1Ev+0x18)[0x7f030175a9a2]
/home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1(_ZN4pvac13ClientChannel3getEdNSt3tr110shared_ptrIKN5epics6pvData11PVStructureEEE+0xc4)[0x7f0301762278]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(main+0xfca)[0x422bb7]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f0300519830]
/home/martins/epics/base/bin/linux-x86_64-debug/pvput(_start+0x29)[0x41d479]
======= Memory map: ========
00400000-00436000 r-xp 00000000 08:05 12584255                           /home/martins/epics/epics-base/bin/linux-x86_64-debug/pvput
00635000-00636000 r--p 00035000 08:05 12584255                           /home/martins/epics/epics-base/bin/linux-x86_64-debug/pvput
00636000-00637000 rw-p 00036000 08:05 12584255                           /home/martins/epics/epics-base/bin/linux-x86_64-debug/pvput
01621000-0167f000 rw-p 00000000 00:00 0                                  [heap]
7f02e0000000-7f02e0021000 rw-p 00000000 00:00 0 
7f02e0021000-7f02e4000000 ---p 00000000 00:00 0 
7f02e8000000-7f02e8021000 rw-p 00000000 00:00 0 
7f02e8021000-7f02ec000000 ---p 00000000 00:00 0 
7f02ec000000-7f02ec021000 rw-p 00000000 00:00 0 
7f02ec021000-7f02f0000000 ---p 00000000 00:00 0 
7f02f0000000-7f02f0028000 rw-p 00000000 00:00 0 
7f02f0028000-7f02f4000000 ---p 00000000 00:00 0 
7f02f4000000-7f02f4021000 rw-p 00000000 00:00 0 
7f02f4021000-7f02f8000000 ---p 00000000 00:00 0 
7f02f8000000-7f02f8021000 rw-p 00000000 00:00 0 
7f02f8021000-7f02fc000000 ---p 00000000 00:00 0 
7f02fd75c000-7f02fd75d000 ---p 00000000 00:00 0 
7f02fd75d000-7f02fd7dd000 rw-p 00000000 00:00 0 
7f02fd7dd000-7f02fd7f4000 r-xp 00000000 08:05 11540472                   /lib/x86_64-linux-gnu/libresolv-2.23.so
7f02fd7f4000-7f02fd9f4000 ---p 00017000 08:05 11540472                   /lib/x86_64-linux-gnu/libresolv-2.23.so
7f02fd9f4000-7f02fd9f5000 r--p 00017000 08:05 11540472                   /lib/x86_64-linux-gnu/libresolv-2.23.so
7f02fd9f5000-7f02fd9f6000 rw-p 00018000 08:05 11540472                   /lib/x86_64-linux-gnu/libresolv-2.23.so
7f02fd9f6000-7f02fd9f8000 rw-p 00000000 00:00 0 
7f02fd9f8000-7f02fd9fd000 r-xp 00000000 08:05 11540478                   /lib/x86_64-linux-gnu/libnss_dns-2.23.so
7f02fd9fd000-7f02fdbfd000 ---p 00005000 08:05 11540478                   /lib/x86_64-linux-gnu/libnss_dns-2.23.so
7f02fdbfd000-7f02fdbfe000 r--p 00005000 08:05 11540478                   /lib/x86_64-linux-gnu/libnss_dns-2.23.so
7f02fdbfe000-7f02fdbff000 rw-p 00006000 08:05 11540478                   /lib/x86_64-linux-gnu/libnss_dns-2.23.so
7f02fdbff000-7f02fdc01000 r-xp 00000000 08:05 11538999                   /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
7f02fdc01000-7f02fde00000 ---p 00002000 08:05 11538999                   /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
7f02fde00000-7f02fde01000 r--p 00001000 08:05 11538999                   /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
7f02fde01000-7f02fde02000 rw-p 00002000 08:05 11538999                   /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
7f02fde02000-7f02fde83000 rw-p 00000000 00:00 0 
7f02fde83000-7f02fde84000 ---p 00000000 00:00 0 
7f02fde84000-7f02fdf84000 rw-p 00000000 00:00 0 
7f02fdf84000-7f02fdf85000 ---p 00000000 00:00 0 
7f02fdf85000-7f02fe185000 rw-p 00000000 00:00 0 
7f02fe185000-7f02fe186000 ---p 00000000 00:00 0 
7f02fe186000-7f02fe286000 rw-p 00000000 00:00 0 
7f02fe286000-7f02fe287000 ---p 00000000 00:00 0 
7f02fe287000-7f02fe487000 rw-p 00000000 00:00 0 
7f02fe487000-7f02fe492000 r-xp 00000000 08:05 11540481                   /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f02fe492000-7f02fe691000 ---p 0000b000 08:05 11540481                   /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f02fe691000-7f02fe692000 r--p 0000a000 08:05 11540481                   /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f02fe692000-7f02fe693000 rw-p 0000b000 08:05 11540481                   /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f02fe693000-7f02fe699000 rw-p 00000000 00:00 0 
7f02fe699000-7f02fe6a4000 r-xp 00000000 08:05 11540485                   /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f02fe6a4000-7f02fe8a3000 ---p 0000b000 08:05 11540485                   /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f02fe8a3000-7f02fe8a4000 r--p 0000a000 08:05 11540485                   /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f02fe8a4000-7f02fe8a5000 rw-p 0000b000 08:05 11540485                   /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f02fe8a5000-7f02fe8bb000 r-xp 00000000 08:05 11540465                   /lib/x86_64-linux-gnu/libnsl-2.23.so
7f02fe8bb000-7f02feaba000 ---p 00016000 08:05 11540465                   /lib/x86_64-linux-gnu/libnsl-2.23.so
7f02feaba000-7f02feabb000 r--p 00015000 08:05 11540465                   /lib/x86_64-linux-gnu/libnsl-2.23.so
7f02feabb000-7f02feabc000 rw-p 00016000 08:05 11540465                   /lib/x86_64-linux-gnu/libnsl-2.23.so
7f02feabc000-7f02feabe000 rw-p 00000000 00:00 0 
7f02feabe000-7f02feac6000 r-xp 00000000 08:05 11540476                   /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f02feac6000-7f02fecc5000 ---p 00008000 08:05 11540476                   /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f02fecc5000-7f02fecc6000 r--p 00007000 08:05 11540476                   /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f02fecc6000-7f02fecc7000 rw-p 00008000 08:05 11540476                   /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f02fecc7000-7f02fecc8000 ---p 00000000 00:00 0 
7f02fecc8000-7f02ff4c8000 rw-p 00000000 00:00 0 
7f02ff4c8000-7f02ff4ed000 r-xp 00000000 08:05 11539066                   /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f02ff4ed000-7f02ff6ec000 ---p 00025000 08:05 11539066                   /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f02ff6ec000-7f02ff6f0000 r--p 00024000 08:05 11539066                   /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f02ff6f0000-7f02ff6f1000 rw-p 00028000 08:05 11539066                   /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f02ff6f1000-7f02ff7f9000 r-xp 00000000 08:05 11534513                   /lib/x86_64-linux-gnu/libm-2.23.so
7f02ff7f9000-7f02ff9f8000 ---p 00108000 08:05 11534513                   /lib/x86_64-linux-gnu/libm-2.23.so
7f02ff9f8000-7f02ff9f9000 r--p 00107000 08:05 11534513                   /lib/x86_64-linux-gnu/libm-2.23.so
7f02ff9f9000-7f02ff9fa000 rw-p 00108000 08:05 11534513                   /lib/x86_64-linux-gnu/libm-2.23.so
7f02ff9fa000-7f02ff9fd000 r-xp 00000000 08:05 11540470                   /lib/x86_64-linux-gnu/libdl-2.23.so
7f02ff9fd000-7f02ffbfc000 ---p 00003000 08:05 11540470                   /lib/x86_64-linux-gnu/libdl-2.23.so
7f02ffbfc000-7f02ffbfd000 r--p 00002000 08:05 11540470                   /lib/x86_64-linux-gnu/libdl-2.23.so
7f02ffbfd000-7f02ffbfe000 rw-p 00003000 08:05 11540470                   /lib/x86_64-linux-gnu/libdl-2.23.so
7f02ffbfe000-7f02ffc05000 r-xp 00000000 08:05 11540486                   /lib/x86_64-linux-gnu/librt-2.23.so
7f02ffc05000-7f02ffe04000 ---p 00007000 08:05 11540486                   /lib/x86_64-linux-gnu/librt-2.23.so
7f02ffe04000-7f02ffe05000 r--p 00006000 08:05 11540486                   /lib/x86_64-linux-gnu/librt-2.23.so
7f02ffe05000-7f02ffe06000 rw-p 00007000 08:05 11540486                   /lib/x86_64-linux-gnu/librt-2.23.so
7f02ffe06000-7f02ffe43000 r-xp 00000000 08:05 11539045                   /lib/x86_64-linux-gnu/libreadline.so.6.3
7f02ffe43000-7f0300043000 ---p 0003d000 08:05 11539045                   /lib/x86_64-linux-gnu/libreadline.so.6.3
7f0300043000-7f0300045000 r--p 0003d000 08:05 11539045                   /lib/x86_64-linux-gnu/libreadline.so.6.3
7f0300045000-7f030004b000 rw-p 0003f000 08:05 11539045                   /lib/x86_64-linux-gnu/libreadline.so.6.3
7f030004b000-7f030004c000 rw-p 00000000 00:00 0 
7f030004c000-7f0300064000 r-xp 00000000 08:05 11540467                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7f0300064000-7f0300263000 ---p 00018000 08:05 11540467                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7f0300263000-7f0300264000 r--p 00017000 08:05 11540467                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7f0300264000-7f0300265000 rw-p 00018000 08:05 11540467                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7f0300265000-7f0300269000 rw-p 00000000 00:00 0 
7f0300269000-7f03002f3000 r-xp 00000000 08:05 12583712                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libca.so.4.13.1
7f03002f3000-7f03004f3000 ---p 0008a000 08:05 12583712                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libca.so.4.13.1
7f03004f3000-7f03004f6000 r--p 0008a000 08:05 12583712                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libca.so.4.13.1
7f03004f6000-7f03004f9000 rw-p 0008d000 08:05 12583712                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libca.so.4.13.1
7f03004f9000-7f03006b9000 r-xp 00000000 08:05 11540468                   /lib/x86_64-linux-gnu/libc-2.23.so
7f03006b9000-7f03008b9000 ---p 001c0000 08:05 11540468                   /lib/x86_64-linux-gnu/libc-2.23.so
7f03008b9000-7f03008bd000 r--p 001c0000 08:05 11540468                   /lib/x86_64-linux-gnu/libc-2.23.so
7f03008bd000-7f03008bf000 rw-p 001c4000 08:05 11540468                   /lib/x86_64-linux-gnu/libc-2.23.so
7f03008bf000-7f03008c3000 rw-p 00000000 00:00 0 
7f03008c3000-7f03008d9000 r-xp 00000000 08:05 11538932                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7f03008d9000-7f0300ad8000 ---p 00016000 08:05 11538932                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0300ad8000-7f0300ad9000 rw-p 00015000 08:05 11538932                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0300ad9000-7f0300c4b000 r-xp 00000000 08:05 16777365                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f0300c4b000-7f0300e4b000 ---p 00172000 08:05 16777365                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f0300e4b000-7f0300e55000 r--p 00172000 08:05 16777365                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f0300e55000-7f0300e57000 rw-p 0017c000 08:05 16777365                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f0300e57000-7f0300e5b000 rw-p 00000000 00:00 0 
7f0300e5b000-7f0300edd000 r-xp 00000000 08:05 12583672                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libCom.so.3.17.0
7f0300edd000-7f03010dc000 ---p 00082000 08:05 12583672                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libCom.so.3.17.0
7f03010dc000-7f03010df000 r--p 00081000 08:05 12583672                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libCom.so.3.17.0
7f03010df000-7f03010e2000 rw-p 00084000 08:05 12583672                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libCom.so.3.17.0
7f03010e2000-7f03010e5000 rw-p 00000000 00:00 0 
7f03010e5000-7f0301337000 r-xp 00000000 08:05 12584180                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvData.so.7.0.1
7f0301337000-7f0301537000 ---p 00252000 08:05 12584180                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvData.so.7.0.1
7f0301537000-7f0301543000 r--p 00252000 08:05 12584180                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvData.so.7.0.1
7f0301543000-7f030154d000 rw-p 0025e000 08:05 12584180                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvData.so.7.0.1
7f030154d000-7f03018ca000 r-xp 00000000 08:05 12584223                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1
7f03018ca000-7f0301ac9000 ---p 0037d000 08:05 12584223                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1
7f0301ac9000-7f0301ad8000 r--p 0037c000 08:05 12584223                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1
7f0301ad8000-7f0301ae7000 rw-p 0038b000 08:05 12584223                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccess.so.6.0.1
7f0301ae7000-7f0301ae9000 rw-p 00000000 00:00 0 
7f0301ae9000-7f0301bbf000 r-xp 00000000 08:05 12584232                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1
7f0301bbf000-7f0301dbf000 ---p 000d6000 08:05 12584232                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1
7f0301dbf000-7f0301dc1000 r--p 000d6000 08:05 12584232                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1
7f0301dc1000-7f0301dc6000 rw-p 000d8000 08:05 12584232                   /home/martins/epics/epics-base/lib/linux-x86_64-debug/libpvAccessCA.so.6.0.1
7f0301dc6000-7f0301dec000 r-xp 00000000 08:05 11540466                   /lib/x86_64-linux-gnu/ld-2.23.so
7f0301e18000-7f0301e3d000 rw-p 00000000 00:00 0 
7f0301e3d000-7f0301e3e000 ---p 00000000 00:00 0 
7f0301e3e000-7f0301f3e000 rw-p 00000000 00:00 0 
7f0301f3e000-7f0301f3f000 ---p 00000000 00:00 0 
7f0301f3f000-7f0301fc7000 rw-p 00000000 00:00 0 
7f0301fe8000-7f0301feb000 rw-p 00000000 00:00 0 
7f0301feb000-7f0301fec000 r--p 00025000 08:05 11540466                   /lib/x86_64-linux-gnu/ld-2.23.so
7f0301fec000-7f0301fed000 rw-p 00026000 08:05 11540466                   /lib/x86_64-linux-gnu/ld-2.23.so
7f0301fed000-7f0301fee000 rw-p 00000000 00:00 0 
7fff8efe5000-7fff8f006000 rw-p 00000000 00:00 0                          [stack]
7fff8f0ef000-7fff8f0f2000 r--p 00000000 00:00 0                          [vvar]
7fff8f0f2000-7fff8f0f4000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
New : fish: “~/epics/base/bin/linux-x86_64-d…” terminated by signal SIGABRT (Abort)
$ ~/epics/base/bin/linux-x86_64-debug/pvput -p ca TEST One
Old : TEST                           Zero
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "TEST", Connecting to: 35.9.126.36:5064, Ignored: 192.168.100.1:5064"
    Source File: ../cac.cpp line 1320
    Current Time: Wed Jul 11 2018 10:23:47.249385794
..................................................................
New : TEST                           One
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "TEST", Connecting to: 35.9.126.36:5064, Ignored: 10.42.0.1:5064"
    Source File: ../cac.cpp line 1320
    Current Time: Wed Jul 11 2018 10:23:47.249845990
..................................................................
CAChannelProvider::~CAChannelProvider() returning
MonitorEventThread::~MonitorEventThread()
$

Maybe automatic merging didn't work very well?

Using @mrkraimer's fork as is

$ pwd
/home/martins/epics/epics-base/modules
$ mv pvAccess pvAccess-orig
$ git clone https://github.com/mrkraimer/pvAccessCPP pvAccess
Cloning into 'pvAccess'...
remote: Counting objects: 12312, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 12312 (delta 1), reused 5 (delta 1), pack-reused 12303
Receiving objects: 100% (12312/12312), 13.42 MiB | 8.68 MiB/s, done.
Resolving deltas: 100% (9082/9082), done.
Checking connectivity... done.
$ cd ..
$ make

Testing

I am not going to paste the results here because they are really similar to the previous ones. Sometimes pvput works, sometimes it segfaults, sometimes it gets a SIGABRT and shows a stack trace.

I'd be happy to run any more tests that you guys would want me to.

mrkraimer commented 6 years ago

I suspect that the problem is related to put callbacks being called from a ca thread.

In https://github.com/mrkraimer/pva2ca I added code so that all put and get client callbacks are called from a non ca thread. I ran extensive tests without any failures.

But not so easy for bruno to test because he is basing his code on pvAccessCPP/src/client/pva/client.h Since this is in pvAccessCPP I do not see how it can use the code from pva2ca.

If, however, Michael will let me move the code from mrkraimer/pva2ca to mrkraimer/pvAccessCPP/src/ca then bruno could test.

I do not want to spend time doing this unless I will also be allowed to merge this code into epics-base/pvAccessCPP.

mrkraimer commented 6 years ago

Bruno, Can you put your test case in github?

I can then clone it and see if I can reproduce what You see.

brunoseivam commented 6 years ago

Hi Marty,

I'll try to do that later on today, but in the mean time the issue can be reproduced with pvput

From Marty I did try and can not get it to fail. I also tried my test connecting to multiple bo and mbbo records and then issues monitors as well as puts and gets as fast as possible. I tried it both with pvAccessCa and with my pva2ca code. I could not get it to fail.

I am using the latest epics-base. I am also doing everything on my laptop.

what version of base are you using? What is your test setup?

brunoseivam commented 6 years ago

Hi Marty,

Sorry for the late reply.

My system:

$ lsb_release -a
No LSB modules are available.
Distributor ID: LinuxMint
Description:    Linux Mint 18.2 Sonya
Release:    18.2
Codename:   sonya
$ uname -a
Linux martins-win7 4.13.0-38-generic #43~16.04.1-Ubuntu SMP Wed Mar 14 17:48:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

I just freshly redownloaded and recompiled everything like so:

git clone --recursive https://github.com/epics-base/epics-base
cd epics-base/modules
mv pvAccess pvAccess-orig
git clone https://github.com/mrkraimer/pvAccessCPP pvAccess
cd ..
# added "CROSS_COMPILER_TARGET_ARCHS=linux-x86_64-debug" to configure/CONFIG_SITE
make -j4

On a terminal:

$ cat test.db 
record(bo, "TEST") {
    field(ZNAM, "Zero")
    field(ONAM, "One")
    field(VAL,  "0")
    field(PINI, "YES")
}
$  ~/test-pvput/epics-base/bin/linux-x86_64-debug/softIoc -d test.db 
Starting iocInit
############################################################################
## EPICS R7.0.1.2-DEV
## EPICS Base built Jul 13 2018
############################################################################
cas warning: Configured TCP port was unavailable.
cas warning: Using dynamically assigned TCP port 37711,
cas warning: but now two or more servers share the same UDP port.
cas warning: Depending on your IP kernel this server may not be
cas warning: reachable with UDP unicast (a host's IP in EPICS_CA_ADDR_LIST)
iocRun: All initialization complete
epics> dbl
TEST
epics>

On another terminal:

$ ~/test-pvput/epics-base/bin/linux-x86_64-debug/pvput -p ca TEST 1
Old : TEST                           Zero
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "TEST", Connecting to: 35.9.126.36:37711, Ignored: 192.168.100.1:37711"
    Source File: ../cac.cpp line 1320
    Current Time: Fri Jul 13 2018 09:57:02.554502615
*** Error in `..................................................................
New : fish: “~/test-pvput/epics-base/bin/lin…” terminated by signal SIGABRT (Abort)

Then I updated all submodules inside modules to master with:

$ git checkout master

Recompiled epics-base from scratch and tested again. Also crashes:

$ ~/test-pvput/epics-base/bin/linux-x86_64-debug/pvput -p ca TEST 0
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "TEST", Connecting to: 35.9.126.36:5064, Ignored: 192.168.100.1:5064"
    Source File: ../cac.cpp line 1320
    Current Time: Fri Jul 13 2018 10:04:57.475097905
..................................................................
Old : TEST                           One
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "TEST", Connecting to: 35.9.126.36:5064, Ignored: 10.42.0.1:5064"
    Source File: ../cac.cpp line 1320
    Current Time: Fri Jul 13 2018 10:04:57.475642607
..................................................................
fish: “~/test-pvput/epics-base/bin/lin…” terminated by signal SIGSEGV (Address boundary error)

Let me know if you want me to test anything else.

anjohnson commented 6 years ago

Replicated, with back-trace:

tux% cat > bo.db
record(bo, "anj:bo1") {
    field(ZNAM, "Zero")
    field(ONAM, "One")
    field(VAL,  "0")
    field(PINI, "YES")
}
tux% softIocPVA -d bo.db 
Starting iocInit
############################################################################
## EPICS R7.0.1.2-DEV
## EPICS Base built Jun 28 2018
############################################################################
iocRun: All initialization complete
epics> dbl
anj:bo1
epics>

Then elsewhere:

tux% caget anj:bo1
anj:bo1                        Zero
tux% pvget anj:bo1
anj:bo1                        Zero
tux% pvget -p ca anj:bo1
anj:bo1                        Zero
tux% pvput anj:bo1 1
Old : anj:bo1                        Zero
New : anj:bo1                        One
tux% pvput -p ca anj:bo1 0
Old : anj:bo1                        One
Segmentation fault
tux% gdb pvput
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/phoebus3/ANJ/epics/base/7.0/bin/linux-x86_64/pvput...done.
(gdb) run -p ca anj:bo1 0
Starting program: /home/phoebus/ANJ/Linux64/bin/pvput -p ca anj:bo1 0
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff57a2700 (LWP 19036)]
[Thread 0x7ffff57a2700 (LWP 19036) exited]
[New Thread 0x7ffff57a2700 (LWP 19038)]
[New Thread 0x7ffff7fa4700 (LWP 19039)]
[New Thread 0x7ffff4fa1700 (LWP 19040)]
[New Thread 0x7ffff4ea0700 (LWP 19042)]
[New Thread 0x7ffff4c9f700 (LWP 19043)]
Old : anj:bo1                        Zero

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff4ea0700 (LWP 19042)]
0x0000000000641c00 in ?? ()
Missing separate debuginfos, use: debuginfo-install glibc-2.17-222.el7.x86_64 libgcc-4.8.5-28.el7.x86_64 libstdc++-4.8.5-28.el7.x86_64 ncurses-libs-5.9-14.20130511.el7_4.x86_64 readline-6.2-10.el7.x86_64
(gdb) thread apply all bt

Thread 7 (Thread 0x7ffff4c9f700 (LWP 19043)):
#0  0x00007ffff6251945 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff7088a6b in epicsEventWait (pevent=0x7fffe800e710)
    at ../../src/osi/os/posix/osdEvent.c:103
#2  0x00007ffff708148c in epicsEvent::wait (this=<optimized out>)
    at ../../src/osi/epicsEvent.cpp:66
#3  0x00007ffff72e7126 in tcpSendThread::run (this=0x7fffe8000a00)
    at ../tcpiiu.cpp:87
#4  0x00007ffff707ff39 in epicsThreadCallEntryPoint (pPvt=0x7fffe8000a08)
    at ../../src/osi/epicsThread.cpp:83
#5  0x00007ffff708632c in start_routine (arg=0x7fffe800a360)
    at ../../src/osi/os/posix/osdThread.c:403
#6  0x00007ffff624ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007ffff6560b3d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7ffff4ea0700 (LWP 19042)):
#0  0x0000000000641c00 in ?? ()
#1  0x00007ffff7bc796f in epics::pvData::detail::PVVectorStorage<std::string, epics::pvData::PVScalarArray>::reuse (this=this@entry=0x641f80)
    at /home/phoebus3/ANJ/epics/base/7.0/include/pv/pvData.h:1125
#2  0x00007ffff7bb7525 in epics::pvAccess::ca::ca_get_labels_handler (args=...)
    at ../caChannel.cpp:103
#3  0x00007ffff72f0ad1 in getCallback::completion (this=<optimized out>, 
    guard=..., type=24, count=1, pData=0x7fffe800a6d8) at ../getCallback.cpp:62
#4  0x00007ffff72ea552 in netReadNotifyIO::completion (this=0x7fffec00d508, 
    guard=..., recycle=..., type=24, count=1, pData=0x7fffe800a6d8)
    at ../netReadNotifyIO.cpp:69
#5  0x00007ffff72cb32e in cac::readNotifyRespAction (this=0x61d3c0, iiu=..., 
    hdr=..., pMsgBdy=0x7fffe800a6d8) at ../cac.cpp:905
#6  0x00007ffff72e820d in tcpiiu::processIncoming (this=0x7fffe80008c0, 
    currentTime=..., mgr=...) at ../tcpiiu.cpp:1264
#7  0x00007ffff72e87d2 in tcpRecvThread::run (this=0x7fffe80009a8)
    at ../tcpiiu.cpp:519
#8  0x00007ffff707ff39 in epicsThreadCallEntryPoint (pPvt=0x7fffe80009b0)
    at ../../src/osi/epicsThread.cpp:83
#9  0x00007ffff708632c in start_routine (arg=0x7fffe8009ea0)
    at ../../src/osi/os/posix/osdThread.c:403
#10 0x00007ffff624ddd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ffff6560b3d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7ffff4fa1700 (LWP 19040)):
#0  0x00007ffff6561a53 in recvfrom () from /lib64/libc.so.6
#1  0x00007ffff72e135d in udpRecvThread::run (this=0x63dfe8)
    at ../udpiiu.cpp:411
#2  0x00007ffff707ff39 in epicsThreadCallEntryPoint (pPvt=0x63e008)
    at ../../src/osi/epicsThread.cpp:83
#3  0x00007ffff708632c in start_routine (arg=0x61f160)
    at ../../src/osi/os/posix/osdThread.c:403
#4  0x00007ffff624ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff6560b3d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7ffff7fa4700 (LWP 19039)):
#0  0x00007ffff6251cf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff7088be9 in epicsEventWaitWithTimeout (pevent=0x61e5b0, 
    timeout=0.026874959) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff70814dc in epicsEvent::wait (this=this@entry=0x61e380, 
    timeOut=<optimized out>) at ../../src/osi/epicsEvent.cpp:75
#3  0x00007ffff7091d5f in timerQueueActive::run (this=0x61e2c0)
    at ../../src/timer/timerQueueActive.cpp:95
#4  0x00007ffff707ff39 in epicsThreadCallEntryPoint (pPvt=0x61e390)
    at ../../src/osi/epicsThread.cpp:83
#5  0x00007ffff708632c in start_routine (arg=0x61e7e0)
    at ../../src/osi/os/posix/osdThread.c:403
---Type <return> to continue, or q <return> to quit---
#6  0x00007ffff624ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007ffff6560b3d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7ffff57a2700 (LWP 19038)):
#0  0x00007ffff6251945 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff7088a6b in epicsEventWait (pevent=0x61deb0)
    at ../../src/osi/os/posix/osdEvent.c:103
#2  0x00007ffff708148c in epicsEvent::wait (this=this@entry=0x61dd60)
    at ../../src/osi/epicsEvent.cpp:66
#3  0x00007ffff707dfe2 in (anonymous namespace)::ipAddrToAsciiGlobal::run (
    this=0x61d920) at ../../src/misc/ipAddrToAsciiAsynchronous.cpp:304
#4  0x00007ffff707ff39 in epicsThreadCallEntryPoint (pPvt=0x61dd70)
    at ../../src/osi/epicsThread.cpp:83
#5  0x00007ffff708632c in start_routine (arg=0x61e0e0)
    at ../../src/osi/os/posix/osdThread.c:403
#6  0x00007ffff624ddd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007ffff6560b3d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7ffff7fa6740 (LWP 19012)):
#0  0x00007ffff6251cf2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007ffff7088be9 in epicsEventWaitWithTimeout (pevent=0x6416a0, 
    timeout=3) at ../../src/osi/os/posix/osdEvent.c:130
#2  0x00007ffff70814dc in epicsEvent::wait (this=this@entry=0x7fffffffd760, 
    timeOut=timeOut@entry=3) at ../../src/osi/epicsEvent.cpp:75
#3  0x00007ffff78c1ea6 in wait (timeout=3, this=0x7fffffffd758)
    at ../../src/client/clientSync.cpp:38
#4  pvac::ClientChannel::get (this=this@entry=0x7fffffffd900, timeout=3, 
    pvRequest=std::tr1::shared_ptr (count 3, weak 1) 0x61bd00)
    at ../../src/client/clientSync.cpp:77
#5  0x0000000000405b8d in main (argc=<optimized out>, argv=<optimized out>)
    at ../pvput.cpp:644
(gdb)

That was with the pvAccess master branch version of pvput. Using the master branch from git@github.com:mrkraimer/pvAccessCPP.git though I get this:

tux% bin/linux-x86_64/pvput -p ca anj:bo1 1
Old : anj:bo1                        Zero
New : anj:bo1                        One
CAChannelProvider::~CAChannelProvider() returning
MonitorEventThread::~MonitorEventThread()
tux% bin/linux-x86_64/pvput -p ca anj:bo1 Zero
Old : anj:bo1                        One
New : anj:bo1                        Zero
CAChannelProvider::~CAChannelProvider() returning
MonitorEventThread::~MonitorEventThread()
tux% bin/linux-x86_64/pvput -p ca anj:bo1 One
Old : anj:bo1                        Zero
New : anj:bo1                        One
CAChannelProvider::~CAChannelProvider() returning
MonitorEventThread::~MonitorEventThread()
tux%

Thus I believe this problem should be fixed by Marty's work in #96 although the above shows that there are still some debugging messages that need removing. @mdavidsaver agreed to the merge at a Hangout that we have minutes for, although it hasn't happened yet; @mrkraimer thought I was supposed to hit the Merge button, but I would rather the maintainer do that himself.

We're going to be talking about this at the Hangout on Tuesday whether we want to or not...

brunoseivam commented 6 years ago

I wonder why @mrkraimer's repo fails on my setup then. Here's with the backtrace:

gdb ~/test-pvput/epics-base/bin/linux-x86_64-debug/pvput
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/martins/test-pvput/epics-base/bin/linux-x86_64-debug/pvput...done.
(gdb) r -p ca TEST 0
Starting program: /home/martins/test-pvput/epics-base/bin/linux-x86_64-debug/pvput -p ca TEST 0
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff5541700 (LWP 26244)]
[Thread 0x7ffff5541700 (LWP 26244) exited]
[New Thread 0x7ffff7fc9700 (LWP 26245)]
[New Thread 0x7ffff4500700 (LWP 26246)]
[New Thread 0x7ffff7f48700 (LWP 26247)]
[New Thread 0x7ffff42ff700 (LWP 26248)]
[New Thread 0x7fffeffff700 (LWP 26249)]
[New Thread 0x7ffff41fe700 (LWP 26250)]
Old : TEST                           Zero

Thread 7 "CAC-TCP-recv" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffeffff700 (LWP 26249)]
0x00007ffff6c749b8 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0  0x00007ffff6c749b8 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x000000000042a285 in __gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::construct (this=0x6791e0, __p=0x20, __val="One")
    at /usr/include/c++/5/ext/new_allocator.h:130
#2  0x00000000004286ab in __gnu_cxx::__alloc_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::construct<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (__a=..., __p=0x20, __arg="One") at /usr/include/c++/5/ext/alloc_traits.h:189
#3  0x00000000004270da in std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::push_back (this=0x6791e0, __x="One") at /usr/include/c++/5/bits/stl_vector.h:917
#4  0x00007ffff7ba06b1 in epics::pvAccess::ca::DbdToPv::getChoicesDone (this=0x6790a0, args=...) at ../dbdToPv.cpp:294
#5  0x00007ffff7b9e249 in epics::pvAccess::ca::enumChoicesHandler (args=...) at ../dbdToPv.cpp:36
#6  0x00007ffff634e2a6 in getCallback::completion (this=0x7fffdc003f20, guard=..., type=24, count=1, pData=0x7fffe800a498) at ../getCallback.cpp:62
#7  0x00007ffff634414d in netReadNotifyIO::completion (this=0x7fffdc00bf48, guard=..., recycle=..., type=24, count=1, pData=0x7fffe800a498) at ../netReadNotifyIO.cpp:69
#8  0x00007ffff632152e in cac::readNotifyRespAction (this=0x654880, iiu=..., hdr=..., pMsgBdy=0x7fffe800a498) at ../cac.cpp:905
#9  0x00007ffff63226c4 in cac::executeResponse (this=0x654880, mgr=..., iiu=..., currentTime=..., hdr=..., pMshBody=0x7fffe800a498 "") at ../cac.cpp:1216
#10 0x00007ffff633fb67 in tcpiiu::processIncoming (this=0x7fffe80008c0, currentTime=..., mgr=...) at ../tcpiiu.cpp:1264
#11 0x00007ffff633d1a5 in tcpRecvThread::run (this=0x7fffe80009a8) at ../tcpiiu.cpp:519
#12 0x00007ffff6f1df5a in epicsThreadCallEntryPoint (pPvt=0x7fffe80009b0) at ../../src/osi/epicsThread.cpp:83
#13 0x00007ffff6f2749f in start_routine (arg=0x7fffe8009ea0) at ../../src/osi/os/posix/osdThread.c:403
#14 0x00007ffff60cd6ba in start_thread (arg=0x7fffeffff700) at pthread_create.c:333
#15 0x00007ffff667a41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb)
mrkraimer commented 6 years ago

NOTE I did the following with pvAccessCPP from both mrkraimer and from epics-base. BOTH worked for me.

I started:

mrk> cat test.db
record(bo, "TEST") {
    field(ZNAM, "Zero")
    field(ONAM, "One")
    field(VAL,  "0")
    field(PINI, "YES")
}
mrk> /home/epicsv4/masterCPP/epics-base/bin/linux-x86_64/softIoc -d test.db
Starting iocInit
############################################################################
## EPICS R3.16.1-DEV
## EPICS Base built Jul 13 2018
############################################################################
cas warning: Configured TCP port was unavailable.
cas warning: Using dynamically assigned TCP port 40097,
cas warning: but now two or more servers share the same UDP port.
cas warning: Depending on your IP kernel this server may not be
cas warning: reachable with UDP unicast (a host's IP in EPICS_CA_ADDR_LIST)
iocRun: All initialization complete
epics>

Note that 

mrk> pwd
/home/epicsv4/masterCPP/epics-base
mrk> cat .git/config
[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
[remote "origin"]
    url = https://github.com/epics-base/epics-base.git
    fetch = +refs/heads/*:refs/remotes/origin/*
[branch "3.16"]
    remote = origin
    merge = refs/heads/3.16

Then in another window:

mrk> /home/epicsv4/masterCPP/pvAccessCPP/bin/linux-x86_64/pvput -p ca TEST One
Old : TEST                           Zero
New : TEST                           One
CAChannelProvider::~CAChannelProvider() returning
MonitorEventThread::~MonitorEventThread()
mrk> /home/epicsv4/masterCPP/pvAccessCPP/bin/linux-x86_64/pvput -p ca TEST 0
Old : TEST                           One
New : TEST                           Zero
CAChannelProvider::~CAChannelProvider() returning
MonitorEventThread::~MonitorEventThread()

And 

mrk> pwd
/home/epicsv4/masterCPP/pvAccessCPP
mrk> cat .git/config
[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
[remote "origin"]
    url = https://github.com/mrkraimer/pvAccessCPP.git
    fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
    remote = origin
    merge = refs/heads/master
mrk>

I have no idea why it fails for Bruno and succeeds for me. I also have no idea why Andrew failed with pvAccessCPP from epics-base. It looks like we are using the same version of everything involved with running the test. The only diff I see is Bruno has linux-x86_64-debug and I have linux-x86_64.

Does anyone have any idea what is happening?

Bruno what do you get when you

cd epics-base
cat .git.config

Lets make sure we have the same version of base

brunoseivam commented 6 years ago 
$ cat .git/config 
[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
[remote "origin"]
    url = https://github.com/epics-base/epics-base
    fetch = +refs/heads/*:refs/remotes/origin/*
[branch "7.0"]
    remote = origin
    merge = refs/heads/7.0
[submodule "modules/normativeTypes"]
    url = https://github.com/epics-base/normativeTypesCPP
[submodule "modules/pvAccess"]
    url = https://github.com/epics-base/pvAccessCPP
[submodule "modules/pvData"]
    url = https://github.com/epics-base/pvDataCPP
[submodule "modules/pvDatabase"]
    url = https://github.com/epics-base/pvDatabaseCPP
[submodule "modules/pva2pva"]
    url = https://github.com/epics-base/pva2pva
[submodule "modules/pvaClient"]
    url = https://github.com/epics-base/pvaClientCPP
anjohnson commented 6 years ago

One difference on the server-side is that I was running softIocPVA from base-7.0 while @mrkraimer is running softIoc from base-3.16, although the CA server versions shouldn't matter. My base-7.0 version on pvput still crashes when I use the 3.16 softIoc which confirms that.

anjohnson commented 6 years ago

Here are the SHAs of the PVA submodules I was running, which are newer than the latest commit to 7.0:

tux% git submodule status
 80920ae0b0cf1a0e41bb77f70bc2aca4df4e1ac3 modules/normativeTypes (5.2.0-3-g80920ae)
+59a65ce317aee13c4c7b0883b436779e15de26be modules/pvAccess (6.0.0-153-g59a65ce)
+a1c0e432ee605c8abaa8f34185503c2886e12545 modules/pvData (7.0.0-54-ga1c0e43)
+e667e39573eabef2162917fd89875fe65962e490 modules/pvDatabase (4.3.0-10-ge667e39)
+a4f35714a0865ec93e16e8238d56f7e456cf7ddc modules/pva2pva (1.0.0-75-ga4f3571)
 4e4554af4e86fa6d7814392e6d95d901c676287b modules/pvaClient (4.3.0-9-g4e4554a)
tux%

For pvAccess that last commit was this:

tux% git log -1
commit 59a65ce317aee13c4c7b0883b436779e15de26be
Author: Michael Davidsaver <mdavidsaver@gmail.com>
Date:   Mon Jul 2 15:56:16 2018 -0700

    discoverInterfaces() fixup bcast address on win32

    Apparently SIO_GET_INTERFACE_LIST sometimes gives
    the global broadcast (255.255.255.255) for one of
    the interfaces.  In this case silently compute
    the iface bcast address.

To update all the submodules to their latest commits, run the .ci/checkout-modules.sh script in base-7.0. I just did that and got some more new commits from @mdavidsaver so now my submodule versions are

tux% git submodule status
 80920ae0b0cf1a0e41bb77f70bc2aca4df4e1ac3 modules/normativeTypes (5.2.0-3-g80920ae)
+e29cda2a7442d229530e1dbc31f2d0fb7e13c5e6 modules/pvAccess (6.0.0-164-ge29cda2)
+4ef7db20f8cf63a64120c58eab25462eae6a35dd modules/pvData (7.0.0-61-g4ef7db2)
+e667e39573eabef2162917fd89875fe65962e490 modules/pvDatabase (4.3.0-10-ge667e39)
 a4f35714a0865ec93e16e8238d56f7e456cf7ddc modules/pva2pva (1.0.0-75-ga4f3571)
 4e4554af4e86fa6d7814392e6d95d901c676287b modules/pvaClient (4.3.0-9-g4e4554a)
tux% cd modules/pvAccess/
tux% git log -1
commit e29cda2a7442d229530e1dbc31f2d0fb7e13c5e6
Author: Michael Davidsaver <mdavidsaver@gmail.com>
Date:   Wed Jul 11 18:14:54 2018 -0700

    sharedstate: missed a reftrack

This version of pvget still segfaults, Marty's does not:

tux% bin/linux-x86_64/pvput -p ca anj:bo1 One
Old : anj:bo1                        Zero
New : anj:bo1                        One
CAChannelProvider::~CAChannelProvider() returning
MonitorEventThread::~MonitorEventThread()
tux% cd modules/pvAccess/
tux% git log -1
commit f83c6c87d62551c9d4a6bf6fb4acd8180502dad0
Author: mrkraimer <mrkraimer@comcast.net>
Date:   Tue Jun 26 08:43:40 2018 -0400

    remove warnings andrew saw while building testCaProvider.cpp
brunoseivam commented 6 years ago 
$ git submodule status
 80920ae0b0cf1a0e41bb77f70bc2aca4df4e1ac3 modules/normativeTypes (5.2.0-3-g80920ae)
+f83c6c87d62551c9d4a6bf6fb4acd8180502dad0 modules/pvAccess (4.0.5-773-gf83c6c8)
+4ef7db20f8cf63a64120c58eab25462eae6a35dd modules/pvData (7.0.0-61-g4ef7db2)
+e667e39573eabef2162917fd89875fe65962e490 modules/pvDatabase (4.3.0-10-ge667e39)
 a4f35714a0865ec93e16e8238d56f7e456cf7ddc modules/pva2pva (1.0.0-75-ga4f3571)
 4e4554af4e86fa6d7814392e6d95d901c676287b modules/pvaClient (4.3.0-9-g4e4554a)

Note that pvAccess hash differs because it is Marty's repo, all the other ones match with Andrew's.

$ cd modules/pvAcccess
$ git log -1
commit f83c6c87d62551c9d4a6bf6fb4acd8180502dad0
Author: mrkraimer <mrkraimer@comcast.net>
Date:   Tue Jun 26 08:43:40 2018 -0400

    remove warnings andrew saw while building testCaProvider.cpp

I still see segfaults.

mrkraimer commented 6 years ago

I think I know what is causing the crash. The main problem is that we now have put without callback as the default. Can I ask Bruno and Andrew to retry pvput specifying block? For example:

pvput -p ca -r "record[block=true]field(value)" TEST One

This is NOT a fix but I want to see if I am on the correct path. What I think is happening is that for an enum ca provider makes a separate request to get the choices. If a put without block is issued then the response for the choices occurs AFTER the put completes. But pvput destroys the put object as soon as the put completes.

Marty

anjohnson commented 6 years ago

You seem to be on the right track there Marty:

tux% bin/linux-x86_64/pvput -p ca -r "record[block=true]field(value)" anj:bo1 Zero
Old : 
epics:nt/NTEnum:1.0 
    enum_t value
        int index 1
        string[] choices [Zero,One]

New : 
epics:nt/NTEnum:1.0 
    enum_t value
        int index 0
        string[] choices [Zero,One]

tux% bin/linux-x86_64/pvput -p ca -r "record[block=false]field(value)" anj:bo1 1 
Old : 
epics:nt/NTEnum:1.0 
    enum_t value
        int index 0
        string[] choices [Zero,One]

Segmentation fault

So block=true prevents the crash, block=false causes it. I tried it a few more times as well, the above is representative of my results.

brunoseivam commented 6 years ago

Far fewer crashes when specifying block=false. No crashes so far when block=true.

anjohnson commented 6 years ago

Marty, I just tried this on your branch; with block=false it doesn't crash, but I did notice that the list of choices changes between the old and new values:

tux% bin/linux-x86_64/pvput -p ca -r "record[block=false]field(value)" anj:bo1 1 
Old : 
epics:nt/NTEnum:1.0 
    enum_t value
        int index 1
        string[] choices [Zero,One]

New : 
epics:nt/NTEnum:1.0 
    enum_t value
        int index 1
        string[] choices [Zero,One,Zero,One]

CAChannelProvider::~CAChannelProvider() returning
MonitorEventThread::~MonitorEventThread()

Don't know if this has anything to do with the original crash, but it's obviously a problem that needs fixing.

BTW I talked to Michael, once you have fixed these smaller issues I will merge your changes.

mrkraimer commented 6 years ago

I also get the same 

mrk> pvput -p ca -r "record[block=false]field(value)" TEST 1
Old : 
epics:nt/NTEnum:1.0 
    enum_t value
        int index 1
        string[] choices [Zero,One]

New : 
epics:nt/NTEnum:1.0 
    enum_t value
        int index 1
        string[] choices [Zero,One,Zero,One]

I am now convinced that block=false is the problem. In dbdToPv.cpp the following appears in the code that is asking for the choice strings.

// NOTE: we do not wait here, since all subsequent requests (over TCP) is serialized
 // and will guarantee that enumChoicesHandler is called first

But if block=false then ca calls the client put callback as soon as the put request is issued.

I am going to fix this first in https://github.com/mrkraimer/pva2ca and then make the ca provider code in https://github.com/mrkraimer/pvAccessCPP

be the same.

mrkraimer commented 6 years ago

I have pushed changes to https://github.com/mrkraimer/pvAccessCPP

brunoseivam commented 6 years ago

Hi Marty,

Your changes seem to have worked here, thanks!

However, on another note, when I issue a PutCallback thebitset argument seems to be ignored somehow, so I get the Provided put value with wrong type message.

I will put together some code that triggers it. I can open a separate issue if that's more appropriate.

As Andrew suggest in the next message please open a separate issue. Also try it with both pva and ca providers.

anjohnson commented 6 years ago

Thanks Marty, this commit seems to have solved all the problems I was seeing.

@brunoseivam please report your other problem as a separate issue (was that specific to the CA provider? That error message appears in the generic code, so it isn't clear).

brunoseivam commented 6 years ago

I created a new issue. See #115.

mdavidsaver commented 5 years ago

Released with Base 7.0.2