search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
24.25k stars 4.68k forks source link

quic integration test Asan violation #32340

Open kyessenov opened 4 months ago

kyessenov commented 4 months ago 
[ RUN      ] QuicHttpIntegrationTests/QuicHttpIntegrationTest.MultipleQuicConnectionsNoBPF/IPv4
source/common/http/http1/codec_impl.cc:1067:9: runtime error: member call on address 0x61c0001a2880 which does not point to an object of type 'Envoy::Http::Http1::ServerConnectionImpl'
0x61c0001a2880: note: object has invalid vptr
 b5 00 00 00  12 8a 0c 00 00 00 00 00  be be be be be be be be  b8 d8 44 0f 00 00 00 00  e8 7d 95 0f
              ^~~~~~~~~~~~~~~~~~~~~~~
              invalid vptr
    #0 0xa6843a6 in void std::__1::__invoke_void_return_wrapper<void, true>::__call<Envoy::Http::Http1::ServerConnectionImpl::ServerConnectionImpl(Envoy::Network::Connection&, Envoy::Http::Http1::CodecStats&, Envoy::Http::ServerConnectionCallbacks&, Envoy::Http::Http1Settings const&, unsigned int, unsigned int, envoy::config::core::v3::HttpProtocolOptions_HeadersWithUnderscoresAction, Envoy::Server::OverloadManager&)::$_5&, Envoy::Buffer::OwnedBufferFragmentImpl const*>(Envoy::Http::Http1::ServerConnectionImpl::ServerConnectionImpl(Envoy::Network::Connection&, Envoy::Http::Http1::CodecStats&, Envoy::Http::ServerConnectionCallbacks&, Envoy::Http::Http1Settings const&, unsigned int, unsigned int, envoy::config::core::v3::HttpProtocolOptions_HeadersWithUnderscoresAction, Envoy::Server::OverloadManager&)::$_5&, Envoy::Buffer::OwnedBufferFragmentImpl const*&&) /proc/self/cwd/source/common/http/http1/codec_impl.cc:1067:9
    #1 0xa68408a in std::__1::__function::__func<Envoy::Http::Http1::ServerConnectionImpl::ServerConnectionImpl(Envoy::Network::Connection&, Envoy::Http::Http1::CodecStats&, Envoy::Http::ServerConnectionCallbacks&, Envoy::Http::Http1Settings const&, unsigned int, unsigned int, envoy::config::core::v3::HttpProtocolOptions_HeadersWithUnderscoresAction, Envoy::Server::OverloadManager&)::$_5, std::__1::allocator<Envoy::Http::Http1::ServerConnectionImpl::ServerConnectionImpl(Envoy::Network::Connection&, Envoy::Http::Http1::CodecStats&, Envoy::Http::ServerConnectionCallbacks&, Envoy::Http::Http1Settings const&, unsigned int, unsigned int, envoy::config::core::v3::HttpProtocolOptions_HeadersWithUnderscoresAction, Envoy::Server::OverloadManager&)::$_5>, void (Envoy::Buffer::OwnedBufferFragmentImpl const*)>::operator()(Envoy::Buffer::OwnedBufferFragmentImpl const*&&) /opt/llvm/bin/../include/c++/v1/__functional/function.h:180:16
    #2 0xa691600 in std::__1::__function::__value_func<void (Envoy::Buffer::OwnedBufferFragmentImpl const*)>::operator()(Envoy::Buffer::OwnedBufferFragmentImpl const*&&) const /opt/llvm/bin/../include/c++/v1/__functional/function.h:507:16
    #3 0xa6911bd in Envoy::Buffer::OwnedBufferFragmentImpl::done() /opt/llvm/bin/../include/c++/v1/__functional/function.h:1184:12

https://dev.azure.com/cncf/envoy/_build/results?buildId=162155&view=logs&j=1439b9f7-a348-5b50-b5fe-ea612ea91241&t=1002ac43-da84-5fae-70b2-98833b702d09

kyessenov commented 4 months ago

CC @danzh2010

github-actions[bot] commented 3 months ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

danzh2010 commented 3 months ago

The ASAN stack doesn't contain QUIC code:

[ RUN      ] QuicHttpIntegrationTests/QuicHttpIntegrationTest.MultipleQuicConnectionsNoBPF/IPv4
source/common/http/http1/codec_impl.cc:1067:9: runtime error: member call on address 0x61c0001a2880 which does not point to an object of type 'Envoy::Http::Http1::ServerConnectionImpl'
0x61c0001a2880: note: object has invalid vptr
 b5 00 00 00  12 8a 0c 00 00 00 00 00  be be be be be be be be  b8 d8 44 0f 00 00 00 00  e8 7d 95 0f
              ^~~~~~~~~~~~~~~~~~~~~~~
              invalid vptr
    #0 0xa6843a6 in void std::__1::__invoke_void_return_wrapper<void, true>::__call<Envoy::Http::Http1::ServerConnectionImpl::ServerConnectionImpl(Envoy::Network::Connection&, Envoy::Http::Http1::CodecStats&, Envoy::Http::ServerConnectionCallbacks&, Envoy::Http::Http1Settings const&, unsigned int, unsigned int, envoy::config::core::v3::HttpProtocolOptions_HeadersWithUnderscoresAction, Envoy::Server::OverloadManager&)::$_5&, Envoy::Buffer::OwnedBufferFragmentImpl const*>(Envoy::Http::Http1::ServerConnectionImpl::ServerConnectionImpl(Envoy::Network::Connection&, Envoy::Http::Http1::CodecStats&, Envoy::Http::ServerConnectionCallbacks&, Envoy::Http::Http1Settings const&, unsigned int, unsigned int, envoy::config::core::v3::HttpProtocolOptions_HeadersWithUnderscoresAction, Envoy::Server::OverloadManager&)::$_5&, Envoy::Buffer::OwnedBufferFragmentImpl const*&&) /proc/self/cwd/source/common/http/http1/codec_impl.cc:1067:9
    #1 0xa68408a in std::__1::__function::__func<Envoy::Http::Http1::ServerConnectionImpl::ServerConnectionImpl(Envoy::Network::Connection&, Envoy::Http::Http1::CodecStats&, Envoy::Http::ServerConnectionCallbacks&, Envoy::Http::Http1Settings const&, unsigned int, unsigned int, envoy::config::core::v3::HttpProtocolOptions_HeadersWithUnderscoresAction, Envoy::Server::OverloadManager&)::$_5, std::__1::allocator<Envoy::Http::Http1::ServerConnectionImpl::ServerConnectionImpl(Envoy::Network::Connection&, Envoy::Http::Http1::CodecStats&, Envoy::Http::ServerConnectionCallbacks&, Envoy::Http::Http1Settings const&, unsigned int, unsigned int, envoy::config::core::v3::HttpProtocolOptions_HeadersWithUnderscoresAction, Envoy::Server::OverloadManager&)::$_5>, void (Envoy::Buffer::OwnedBufferFragmentImpl const*)>::operator()(Envoy::Buffer::OwnedBufferFragmentImpl const*&&) /opt/llvm/bin/../include/c++/v1/__functional/function.h:180:16
    #2 0xa691600 in std::__1::__function::__value_func<void (Envoy::Buffer::OwnedBufferFragmentImpl const*)>::operator()(Envoy::Buffer::OwnedBufferFragmentImpl const*&&) const /opt/llvm/bin/../include/c++/v1/__functional/function.h:507:16
    #3 0xa6911bd in Envoy::Buffer::OwnedBufferFragmentImpl::done() /opt/llvm/bin/../include/c++/v1/__functional/function.h:1184:12
    #4 0xe4cff88 in void std::__1::__invoke_void_return_wrapper<void, true>::__call<Envoy::Buffer::Slice::Slice(Envoy::Buffer::BufferFragment&)::'lambda'()&>(Envoy::Buffer::Slice::Slice(Envoy::Buffer::BufferFragment&)::'lambda'()&) /proc/self/cwd/./source/common/buffer/buffer_impl.h:95:42
    #5 0xe4cfed8 in std::__1::__function::__alloc_func<Envoy::Buffer::Slice::Slice(Envoy::Buffer::BufferFragment&)::'lambda'(), std::__1::allocator<Envoy::Buffer::Slice::Slice(Envoy::Buffer::BufferFragment&)::'lambda'()>, void ()>::operator()() /opt/llvm/bin/../include/c++/v1/__functional/function.h:180:16
    #6 0xe4cf653 in std::__1::__function::__func<Envoy::Buffer::Slice::Slice(Envoy::Buffer::BufferFragment&)::'lambda'(), std::__1::allocator<Envoy::Buffer::Slice::Slice(Envoy::Buffer::BufferFragment&)::'lambda'()>, void ()>::operator()() /opt/llvm/bin/../include/c++/v1/__functional/function.h:354:12
    #7 0x59d93a9 in std::__1::__function::__value_func<void ()>::operator()() const /opt/llvm/bin/../include/c++/v1/__functional/function.h:507:16
    #8 0xa7b3e14 in Envoy::Buffer::Slice::operator=(Envoy::Buffer::Slice&&) /opt/llvm/bin/../include/c++/v1/__functional/function.h:1184:12
    #9 0xe4c7850 in Envoy::Buffer::SliceDeque::pop_front() /proc/self/cwd/./source/common/buffer/buffer_impl.h:506:13
    #10 0xe4b5c33 in Envoy::Buffer::OwnedImpl::drainImpl(unsigned long) /proc/self/cwd/source/common/buffer/buffer_impl.cc:181:13
    #11 0xe4b5796 in Envoy::Buffer::OwnedImpl::drain(unsigned long) /proc/self/cwd/source/common/buffer/buffer_impl.cc:160:40
    #12 0xc31d37a in Envoy::Buffer::WatermarkBuffer::drain(unsigned long) /proc/self/cwd/source/common/buffer/watermark_buffer.cc:55:14
    #13 0xcb36028 in Envoy::Network::IoSocketHandleImpl::write(Envoy::Buffer::Instance&) /proc/self/cwd/source/common/network/io_socket_handle_impl.cc:140:12
    #14 0xc66d4bd in Envoy::Network::RawBufferSocket::doWrite(Envoy::Buffer::Instance&, bool) /proc/self/cwd/source/common/network/raw_buffer_socket.cc:67:61
    #15 0xc2d8a34 in Envoy::Network::ConnectionImpl::onWriteReady() /proc/self/cwd/source/common/network/connection_impl.cc:765:40
    #16 0xc2d4c2f in Envoy::Network::ConnectionImpl::onFileEvent(unsigned int) /proc/self/cwd/source/common/network/connection_impl.cc:644:5
    #17 0xc2f95c3 in void std::__1::__invoke_void_return_wrapper<void, true>::__call<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_6&, unsigned int>(Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_6&, unsigned int&&) /proc/self/cwd/source/common/network/connection_impl.cc:101:54
    #18 0xc2f92ba in std::__1::__function::__func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_6, std::__1::allocator<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_6>, void (unsigned int)>::operator()(unsigned int&&) /opt/llvm/bin/../include/c++/v1/__functional/function.h:180:16
    #19 0x6aa9340 in std::__1::__function::__value_func<void (unsigned int)>::operator()(unsigned int&&) const /opt/llvm/bin/../include/c++/v1/__functional/function.h:507:16
    #20 0xc237b4d in decltype(static_cast<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5&>(fp)(static_cast<unsigned int>(fp0))) std::__1::__invoke<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5&, unsigned int>(Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5&, unsigned int&&) /opt/llvm/bin/../include/c++/v1/__functional/function.h:1184:12
    #21 0xc2379d0 in void std::__1::__invoke_void_return_wrapper<void, true>::__call<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5&, unsigned int>(Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5&, unsigned int&&) /opt/llvm/bin/../include/c++/v1/__functional/invoke.h:61:9
    #22 0xc2376ba in std::__1::__function::__func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5, std::__1::allocator<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_5>, void (unsigned int)>::operator()(unsigned int&&) /opt/llvm/bin/../include/c++/v1/__functional/function.h:180:16
    #23 0x6aa9340 in std::__1::__function::__value_func<void (unsigned int)>::operator()(unsigned int&&) const /opt/llvm/bin/../include/c++/v1/__functional/function.h:507:16
    #24 0xc250dec in Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb(unsigned int) /opt/llvm/bin/../include/c++/v1/__functional/function.h:1184:12
    #25 0xc252a64 in decltype(static_cast<Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_0&>(fp)()) std::__1::__invoke<Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_0&>(Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_0&) /proc/self/cwd/source/common/event/file_event_impl.cc:18:9
    #26 0xc252613 in std::__1::__function::__func<Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_0, std::__1::allocator<Envoy::Event::FileEventImpl::FileEventImpl(Envoy::Event::DispatcherImpl&, int, std::__1::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_0>, void ()>::operator()() /opt/llvm/bin/../include/c++/v1/__functional/invoke.h:61:9
    #27 0x59d93a9 in std::__1::__function::__value_func<void ()>::operator()() const /opt/llvm/bin/../include/c++/v1/__functional/function.h:507:16
    #28 0xc238857 in decltype(static_cast<Envoy::Event::DispatcherImpl::createSchedulableCallback(std::__1::function<void ()>)::$_6&>(fp)()) std::__1::__invoke<Envoy::Event::DispatcherImpl::createSchedulableCallback(std::__1::function<void ()>)::$_6&>(Envoy::Event::DispatcherImpl::createSchedulableCallback(std::__1::function<void ()>)::$_6&) /opt/llvm/bin/../include/c++/v1/__functional/function.h:1184:12
    #29 0xc2384d3 in std::__1::__function::__func<Envoy::Event::DispatcherImpl::createSchedulableCallback(std::__1::function<void ()>)::$_6, std::__1::allocator<Envoy::Event::DispatcherImpl::createSchedulableCallback(std::__1::function<void ()>)::$_6>, void ()>::operator()() /opt/llvm/bin/../include/c++/v1/__functional/invoke.h:61:9
    #30 0x59d93a9 in std::__1::__function::__value_func<void ()>::operator()() const /opt/llvm/bin/../include/c++/v1/__functional/function.h:507:16
    #31 0xcb630c4 in Envoy::Event::SchedulableCallbackImpl::SchedulableCallbackImpl(Envoy::CSmartPtr<event_base, &(event_base_free)>&, std::__1::function<void ()>)::$_0::__invoke(int, short, void*) /opt/llvm/bin/../include/c++/v1/__functional/function.h:1184:12
    #32 0xe514eee in event_process_active_single_queue /b/f/w/external/com_github_libevent_libevent/event.c:1713:4
    #33 0xe4f64e2 in event_process_active /b/f/w/external/com_github_libevent_libevent/event.c
    #34 0xe4f64e2 in event_base_loop /b/f/w/external/com_github_libevent_libevent/event.c:2047:12
    #35 0xcb5e44c in Envoy::Event::LibeventScheduler::run(Envoy::Event::Dispatcher::RunType) /proc/self/cwd/source/common/event/libevent_scheduler.cc:60:3
    #36 0xc22bd01 in Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) /proc/self/cwd/source/common/event/dispatcher_impl.cc:277:19
    #37 0x5e3a0f8 in Envoy::FakeUpstream::threadRoutine() /proc/self/cwd/test/integration/fake_upstream.cc:748:16
    #38 0x5e5fcc7 in void std::__1::__invoke_void_return_wrapper<void, true>::__call<Envoy::FakeUpstream::initializeServer()::$_27&>(Envoy::FakeUpstream::initializeServer()::$_27&) /proc/self/cwd/test/integration/fake_upstream.cc:695:67
    #39 0x5e5f9e3 in std::__1::__function::__func<Envoy::FakeUpstream::initializeServer()::$_27, std::__1::allocator<Envoy::FakeUpstream::initializeServer()::$_27>, void ()>::operator()() /opt/llvm/bin/../include/c++/v1/__functional/function.h:180:16
    #40 0x59d93a9 in std::__1::__function::__value_func<void ()>::operator()() const /opt/llvm/bin/../include/c++/v1/__functional/function.h:507:16
    #41 0xe4ad89c in Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, std::__1::optional<Envoy::Thread::Options> const&)::'lambda'(void*)::operator()(void*) const /opt/llvm/bin/../include/c++/v1/__functional/function.h:1184:12
    #42 0xe4ad813 in Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, std::__1::optional<Envoy::Thread::Options> const&)::'lambda'(void*)::__invoke(void*) /proc/self/cwd/source/common/common/posix/thread_impl.cc:48:9
    #43 0x7fcc7ebb6608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x8608) (BuildId: 0c044ba611aeeeaebb8374e660061f341ebc0bac)
    #44 0x7fcc7eab4352 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f352) (BuildId: eebe5d5f4b608b8a53ec446b63981bba373ca0ca)

It looks like an ASAN issue in upstream HTTP/1.

danzh2010 commented 3 weeks ago

The ASAN call stack is the FakeUpstream with Envoy::Http::Http1::ServerConnectionImpl::ServerConnectionImpl. Not HTTP/3 related.

danzh2010 commented 2 weeks ago

This is a duplicate of #34353