search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
24.87k stars 4.78k forks source link

`DownstreamProtocolIntegrationTest.BasicMaxStreamTimeout/` test is triggering a `use-after-free` (in asan/tsan tests) #26080

Closed phlax closed 1 year ago

phlax commented 1 year ago

This happens fairly frequently in both the asan and tsan tests and i have reported it privately in various places

typical tsan error:

[ RUN      ] DownstreamProtocols/DownstreamProtocolIntegrationTest.BasicMaxStreamTimeout/IPv6_Http3Downstream_HttpUpstreamHttpParserNghttp2NoDeferredProcessing
==================

WARNING: ThreadSanitizer: heap-use-after-free (virtual call vs free) (pid=18)
  Read of size 8 at 0x7b2800002a88 by main thread:
    #0 Envoy::Http::StreamCallbackHelper::runResetCallbacks(Envoy::Http::StreamResetReason) /proc/self/cwd/./source/common/http/codec_helper.h:51:20 (quic_protocol_integration_test+0x62770a4)
    #1 Envoy::Quic::EnvoyQuicClientStream::ResetWithError(quic::QuicResetStreamError) /proc/self/cwd/source/common/quic/envoy_quic_client_stream.cc:339:3 (quic_protocol_integration_test+0x6eebfb3)
    #2 quic::QuicStream::Reset(quic::QuicRstStreamErrorCode) /proc/self/cwd/external/com_github_google_quiche/quiche/quic/core/quic_stream.cc:588:3 (quic_protocol_integration_test+0x7275c5a)
    #3 Envoy::Quic::EnvoyQuicClientStream::resetStream(Envoy::Http::StreamResetReason) /proc/self/cwd/source/common/quic/envoy_quic_client_stream.cc:139:3 (quic_protocol_integration_test+0x6ee5ca9)
    #4 non-virtual thunk to Envoy::Quic::EnvoyQuicClientStream::resetStream(Envoy::Http::StreamResetReason) /proc/self/cwd/source/common/quic/envoy_quic_client_stream.cc (quic_protocol_integration_test+0x6ee5d08)
    #5 Envoy::Http::CodecClient::onEvent(Envoy::Network::ConnectionEvent) /proc/self/cwd/source/common/http/codec_client.cc:120:45 (quic_protocol_integration_test+0x603cb34)
    #6 non-virtual thunk to Envoy::Http::CodecClient::onEvent(Envoy::Network::ConnectionEvent) /proc/self/cwd/source/common/http/codec_client.cc (quic_protocol_integration_test+0x603de25)
    #7 Envoy::Network::ConnectionImplBase::raiseConnectionEvent(Envoy::Network::ConnectionEvent) /proc/self/cwd/source/common/network/connection_impl_base.cc:62:17 (quic_protocol_integration_test+0x787164a)
    #8 Envoy::Quic::QuicFilterManagerConnectionImpl::onConnectionCloseEvent(quic::QuicConnectionCloseFrame const&, quic::ConnectionCloseSource, quic::ParsedQuicVersion const&) /proc/self/cwd/source/common/quic/quic_filter_manager_connection_impl.cc:179:5 (quic_protocol_integration_test+0x71644b1)
    #9 Envoy::Quic::EnvoyQuicClientSession::OnConnectionClosed(quic::QuicConnectionCloseFrame const&, quic::ConnectionCloseSource) /proc/self/cwd/source/common/quic/envoy_quic_client_session.cc:99:3 (quic_protocol_integration_test+0x6ecccde)
    #10 non-virtual thunk to Envoy::Quic::EnvoyQuicClientSession::OnConnectionClosed(quic::QuicConnectionCloseFrame const&, quic::ConnectionCloseSource) /proc/self/cwd/source/common/quic/envoy_quic_client_session.cc (quic_protocol_integration_test+0x6eccd88)
    #11 quic::QuicConnection::TearDownLocalConnectionState(quic::QuicConnectionCloseFrame const&, quic::ConnectionCloseSource) /proc/self/cwd/external/com_github_google_quiche/quiche/quic/core/quic_connection.cc:4587:13 (quic_protocol_integration_test+0x7344254)
    #12 quic::QuicConnection::TearDownLocalConnectionState(quic::QuicErrorCode, quic::QuicIetfTransportErrorCodes, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, quic::ConnectionCloseSource) /proc/self/cwd/external/com_github_google_quiche/quiche/quic/core/quic_connection.cc:4573:10 (quic_protocol_integration_test+0x733267f)
    #13 quic::QuicConnection::CloseConnection(quic::QuicErrorCode, quic::QuicIetfTransportErrorCodes, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, quic::ConnectionCloseBehavior) /proc/self/cwd/external/com_github_google_quiche/quiche/quic/core/quic_connection.cc:4478:3 (quic_protocol_integration_test+0x73630d3)
    #14 quic::QuicConnection::CloseConnection(quic::QuicErrorCode, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, quic::ConnectionCloseBehavior) /proc/self/cwd/external/com_github_google_quiche/quiche/quic/core/quic_connection.cc:4449:3 (quic_protocol_integration_test+0x7362918)
    #15 Envoy::Quic::QuicFilterManagerConnectionImpl::closeConnectionImmediately() /proc/self/cwd/source/common/quic/quic_filter_manager_connection_impl.cc:208:21 (quic_protocol_integration_test+0x7165440)
    #16 Envoy::Quic::QuicFilterManagerConnectionImpl::close(Envoy::Network::ConnectionCloseType) /proc/self/cwd/source/common/quic/quic_filter_manager_connection_impl.cc:115:7 (quic_protocol_integration_test+0x716257c)
    #17 virtual thunk to Envoy::Quic::QuicFilterManagerConnectionImpl::close(Envoy::Network::ConnectionCloseType) /proc/self/cwd/source/common/quic/quic_filter_manager_connection_impl.cc (quic_protocol_integration_test+0x716264d)
    #18 Envoy::Http::CodecClient::close(Envoy::Network::ConnectionCloseType) /proc/self/cwd/source/common/http/codec_client.cc:62:75 (quic_protocol_integration_test+0x603b24d)
    #19 Envoy::HttpIntegrationTest::cleanupUpstreamAndDownstream() /proc/self/cwd/test/integration/http_integration.cc:478:20 (quic_protocol_integration_test+0x31d58c9)
    #15 testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5280:10 (quic_protocol_integration_test+0x9b0850d)
    #16 RUN_ALL_TESTS() /proc/self/cwd/external/com_google_googletest/googletest/include/gtest/gtest.h:2485:46 (quic_protocol_integration_test+0x75c46d7)
    #17 Envoy::TestRunner::RunTests(int, char**) /proc/self/cwd/test/test_runner.cc:159:10 (quic_protocol_integration_test+0x75c3b97)
    #18 main /proc/self/cwd/test/main.cc:34:10 (quic_protocol_integration_test+0x75c02c5)

https://dev.azure.com/cncf/envoy/_build/results?buildId=130615&view=logs&j=d1f76054-8f79-554b-6f4a-11d6a63b8e00&t=266e17e3-d213-54b5-deef-0dcee01da137&l=313

typical asan error (involuntarily truncated):

[ RUN      ] DownstreamProtocols/DownstreamProtocolIntegrationTest.BasicMaxStreamTimeout/IPv4_Http3Downstream_HttpUpstreamHttpParserNghttp2NoDeferredProcessing
=================================================================
==18==ERROR: AddressSanitizer: heap-use-after-free on address 0x60e00000f8c8 at pc 0x000008cd9d40 bp 0x7ffcae49d980 sp 0x7ffcae49d978
READ of size 8 at 0x60e00000f8c8 thread T0
  0x0c1c7fff9f40: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa
  0x0c1c7fff9f50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c1c7fff9f60: fd fd fd fd fa fa fa fa fa fa fa fa fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==18==ABORTING
================================================================================

https://dev.azure.com/cncf/envoy/_build/results?buildId=130615&view=logs&j=d1f76054-8f79-554b-6f4a-11d6a63b8e00&t=266e17e3-d213-54b5-deef-0dcee01da137&l=313

i believe the issue be only test-related

phlax commented 1 year ago

cc @danzh2010