search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.93k stars 1.06k forks source link

[DocDB][ASAN Unit Test Failure][2.8] TwoDCTestParams/TwoDCTest.ApplyOperationsRandomFailures/0 #14059

Open def- opened 2 years ago

def- commented 2 years ago

Jira Link: DB-3553

Description

On 2.8.10.0-b6 there is a heap-use-after-free in TwoDCTestParams/TwoDCTest.ApplyOperationsRandomFailures/0 test:

==21354==ERROR: AddressSanitizer: heap-use-after-free on address 0x61d00000bea8 at pc 0x7f0ccf52b7e8 bp 0x7f0c64af32b0 sp 0x7f0c64af32a8
READ of size 8 at 0x61d00000bea8 thread T134 (Master_reactorx)
    #0 0x7f0ccf52b7e7 in scoped_refptr<yb::Counter>::operator bool() const /nfusr/centos-gcp-cloud/jenkins-worker-s1xxwz/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-asan-130/yugabyte-db/build/asan-clang7-dynamic-ninja/../../src/yb/gutil/ref_counted.h:347:43
W0916 07:59:58.956744 22317 meta_cache.cc:861] 0x00006140014f0258 -> LookupByIdRpc(tablet: 3f947e72a9ea4bef99de20a265a2091f, num_attempts: 2): Encountered a error from the Master: Network error (yb/util/net/socket.cc:557): recvmsg error: Connection refused (system error 111), retrying...
    #1 0x7f0cc24ba817 in yb::rpc::InboundCall::Serialize(boost::container::small_vector_base<yb::RefCntBuffer, boost::container::new_allocator<yb::RefCntBuffer> >*) /nfusr/centos-gcp-cloud/jenkins-worker-s1xxwz/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-asan-130/yugabyte-db/build/asan-clang7-dynamic-ninja/../../src/yb/rpc/inbound_call.cc:242:30
    #2 0x7f0cc2624a59 in yb::rpc::TcpStreamSendingData::TcpStreamSendingData(std::__1::shared_ptr<yb::rpc::OutboundData>, std::__1::shared_ptr<yb::MemTracker> const&) /nfusr/centos-gcp-cloud/jenkins-worker-s1xxwz/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-asan-130/yugabyte-db/build/asan-clang7-dynamic-ninja/../../src/yb/rpc/tcp_stream.cc:526:9
    #3 0x7f0cc26264b6 in void std::__1::allocator<yb::rpc::TcpStreamSendingData>::construct<yb::rpc::TcpStreamSendingData, std::__1::shared_ptr<yb::rpc::OutboundData>, std::__1::shared_ptr<yb::MemTracker>&>(yb::rpc::TcpStreamSendingData*, std::__1::shared_ptr<yb::rpc::OutboundData>&&, std::__1::shared_ptr<yb::MemTracker>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210916160346-44ba371965-centos7-x86_64-clang7/installed/asan/libcxx/include/c++/v1/memory:1811:31
    #4 0x7f0cc26264b6 in void std::__1::allocator_traits<std::__1::allocator<yb::rpc::TcpStreamSendingData> >::__construct<yb::rpc::TcpStreamSendingData, std::__1::shared_ptr<yb::rpc::OutboundData>, std::__1::shared_ptr<yb::MemTracker>&>(std::__1::integral_constant<bool, true>, std::__1::allocator<yb::rpc::TcpStreamSendingData>&, yb::rpc::TcpStreamSendingData*, std::__1::shared_ptr<yb::rpc::OutboundData>&&, std::__1::shared_ptr<yb::MemTracker>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210916160346-44ba371965-centos7-x86_64-clang7/installed/asan/libcxx/include/c++/v1/memory:1716
    #5 0x7f0cc26264b6 in void std::__1::allocator_traits<std::__1::allocator<yb::rpc::TcpStreamSendingData> >::construct<yb::rpc::TcpStreamSendingData, std::__1::shared_ptr<yb::rpc::OutboundData>, std::__1::shared_ptr<yb::MemTracker>&>(std::__1::allocator<yb::rpc::TcpStreamSendingData>&, yb::rpc::TcpStreamSendingData*, std::__1::shared_ptr<yb::rpc::OutboundData>&&, std::__1::shared_ptr<yb::MemTracker>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210916160346-44ba371965-centos7-x86_64-clang7/installed/asan/libcxx/include/c++/v1/memory:1562
    #6 0x7f0cc26264b6 in void std::__1::deque<yb::rpc::TcpStreamSendingData, std::__1::allocator<yb::rpc::TcpStreamSendingData> >::emplace_back<std::__1::shared_ptr<yb::rpc::OutboundData>, std::__1::shared_ptr<yb::MemTracker>&>(std::__1::shared_ptr<yb::rpc::OutboundData>&&, std::__1::shared_ptr<yb::MemTracker>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20210916160346-44ba371965-centos7-x86_64-clang7/installed/asan/libcxx/include/c++/v1/deque:1867
    #7 0x7f0cc26232c0 in yb::rpc::TcpStream::Send(std::__1::shared_ptr<yb::rpc::OutboundData>) /nfusr/centos-gcp-cloud/jenkins-worker-s1xxwz/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-asan-130/yugabyte-db/build/asan-clang7-dynamic-ninja/../../src/yb/rpc/tcp_stream.cc:465:12
    #8 0x7f0cc2562273 in yb::rpc::RefinedStream::Send(std::__1::shared_ptr<yb::rpc::OutboundData>) /nfusr/centos-gcp-cloud/jenkins-worker-s1xxwz/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-asan-130/yugabyte-db/build/asan-clang7-dynamic-ninja/../../src/yb/rpc/refined_stream.cc:94:27
    #9 0x7f0cc248ce0d in yb::rpc::Connection::DoQueueOutboundData(std::__1::shared_ptr<yb::rpc::OutboundData>, bool) /nfusr/centos-gcp-cloud/jenkins-worker-s1xxwz/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-asan-130/yugabyte-db/build/asan-clang7-dynamic-ninja/../../src/yb/rpc/connection.cc:270:26
    #10 0x7f0cc2491e86 in yb::rpc::Connection::ProcessResponseQueue() /nfusr/centos-gcp-cloud/jenkins-worker-s1xxwz/jenkins/jenkins-github-yugabyte-db-centos-master-clang7-asan-130/yugabyte-db/build/asan-clang7-dynamic-ninja/../../src/yb/rpc/connection.cc:444:7

https://gist.github.com/def-/53603fda27673dd4e4aab0c5ef10207f This might just be a test issue since the free comes from yb::MiniCluster::Shutdown()

bmatican commented 2 years ago

Sandeep, passing to you to clarify it test-only or product, to determine if we want any backports.