search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
24.72k stars 4.75k forks source link

tcp_async_client_integration_test is flaky #35770

Open antoniovleonti opened 3 weeks ago

antoniovleonti commented 3 weeks ago

https://dev.azure.com/cncf/envoy/_build/results?buildId=178186&view=logs&j=76424b50-4542-51d2-ddfa-1b282263c744&t=ee414bb6-1c71-528f-760c-893708054276

antoniovleonti commented 3 weeks ago

This test was added recently in this PR: https://github.com/envoyproxy/envoy/pull/35410/files#diff-33b09843331506a563f8f47fdfed56cd03dc444164c4b5ef296f22b7556304faR166

@ohadvano PTAL

phlax commented 1 week ago

im seeing this on uncached runs in other repos

phlax commented 1 week ago

the error ...

Executing tests from //test/integration:tcp_async_client_integration_test
-----------------------------------------------------------------------------
[==========] Running 16 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 16 tests from IpVersions/TcpAsyncClientIntegrationTest
[ RUN      ] IpVersions/TcpAsyncClientIntegrationTest.SingleRequest/0
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_runtime_initialized to: false
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_quic_always_support_server_preferred_address to: true
[       OK ] IpVersions/TcpAsyncClientIntegrationTest.SingleRequest/0 (10039 ms)
[ RUN      ] IpVersions/TcpAsyncClientIntegrationTest.SingleRequest/1
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_runtime_initialized to: false
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_quic_always_support_server_preferred_address to: true
[       OK ] IpVersions/TcpAsyncClientIntegrationTest.SingleRequest/1 (10041 ms)
[ RUN      ] IpVersions/TcpAsyncClientIntegrationTest.MultipleRequestFrames/0
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_runtime_initialized to: false
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_quic_always_support_server_preferred_address to: true
[       OK ] IpVersions/TcpAsyncClientIntegrationTest.MultipleRequestFrames/0 (60 ms)
[ RUN      ] IpVersions/TcpAsyncClientIntegrationTest.MultipleRequestFrames/1
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_runtime_initialized to: false
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_quic_always_support_server_preferred_address to: true
[       OK ] IpVersions/TcpAsyncClientIntegrationTest.MultipleRequestFrames/1 (53 ms)
[ RUN      ] IpVersions/TcpAsyncClientIntegrationTest.MultipleResponseFrames/0
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_runtime_initialized to: false
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_quic_always_support_server_preferred_address to: true
[       OK ] IpVersions/TcpAsyncClientIntegrationTest.MultipleResponseFrames/0 (76 ms)
[ RUN      ] IpVersions/TcpAsyncClientIntegrationTest.MultipleResponseFrames/1
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_runtime_initialized to: false
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_quic_always_support_server_preferred_address to: true
[       OK ] IpVersions/TcpAsyncClientIntegrationTest.MultipleResponseFrames/1 (42 ms)
[ RUN      ] IpVersions/TcpAsyncClientIntegrationTest.Reconnect/0
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_runtime_initialized to: false
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_quic_always_support_server_preferred_address to: true
[       OK ] IpVersions/TcpAsyncClientIntegrationTest.Reconnect/0 (100 ms)
[ RUN      ] IpVersions/TcpAsyncClientIntegrationTest.Reconnect/1
[       OK ] IpVersions/TcpAsyncClientIntegrationTest.Reconnect/1 (22 ms)
[ RUN      ] IpVersions/TcpAsyncClientIntegrationTest.ClientTearDown/0
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_reloadable_features_runtime_initialized to: false
[external/com_google_absl/absl/flags/internal/flag.cc : 115] RAW: Restore saved value of envoy_quic_always_support_server_preferred_address to: true
[       OK ] IpVersions/TcpAsyncClientIntegrationTest.ClientTearDown/0 (60 ms)
[ RUN      ] IpVersions/TcpAsyncClientIntegrationTest.ClientTearDown/1
[2024-09-03 11:02:52.748][73768][critical][backtrace] [./source/server/backtrace.h:127] Caught Segmentation fault, suspect faulting address 0x0
[2024-09-03 11:02:52.748][73768][critical][backtrace] [./source/server/backtrace.h:111] Backtrace (use tools/stack_decode.py to get line numbers):
[2024-09-03 11:02:52.748][73768][critical][backtrace] [./source/server/backtrace.h:112] Envoy version: 0/1.32.0-dev/test/RELEASE/BoringSSL
[2024-09-03 11:02:52.748][73768][critical][backtrace] [./source/server/backtrace.h:119] #0: __kernel_rt_sigreturn [0xffbc6fb697e0]
[2024-09-03 11:02:52.761][73768][critical][backtrace] [./source/server/backtrace.h:119] #1: Envoy::Network::FilterManagerImpl::onContinueReading() [0x18f9f98]
[2024-09-03 11:02:52.770][73768][critical][backtrace] [./source/server/backtrace.h:119] #2: Envoy::Network::ConnectionImpl::onReadReady() [0x18f13fc]
[2024-09-03 11:02:52.779][73768][critical][backtrace] [./source/server/backtrace.h:119] #3: Envoy::Network::ConnectionImpl::onFileEvent() [0x18edd78]
[2024-09-03 11:02:52.788][73768][critical][backtrace] [./source/server/backtrace.h:119] #4: std::__1::__function::__func<>::operator()() [0x18f7194]
[2024-09-03 11:02:52.797][73768][critical][backtrace] [./source/server/backtrace.h:119] #5: std::__1::__function::__func<>::operator()() [0x18dfac8]
[2024-09-03 11:02:52.806][73768][critical][backtrace] [./source/server/backtrace.h:119] #6: Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb() [0x18e0d0c]
[2024-09-03 11:02:52.815][73768][critical][backtrace] [./source/server/backtrace.h:119] #7: event_process_active_single_queue [0x1ce2738]
[2024-09-03 11:02:52.824][73768][critical][backtrace] [./source/server/backtrace.h:119] #8: event_base_loop [0x1ce0fd8]
[2024-09-03 11:02:52.833][73768][critical][backtrace] [./source/server/backtrace.h:119] #9: Envoy::Server::WorkerImpl::threadRoutine() [0xfb99e4]
[2024-09-03 11:02:52.842][73768][critical][backtrace] [./source/server/backtrace.h:119] #10: Envoy::Thread::PosixThreadFactory::createPthread()::$_2::__invoke() [0x1cd9698]
[2024-09-03 11:02:52.842][73768][critical][backtrace] [./source/server/backtrace.h:119] #11: start_thread [0xffbc6fa38624]
ConnectionImpl 0x150b3f0bcc80, connecting_: 0, bind_error_: 0, state(): Open, read_buffer_limit_: 1048576
socket_: 
  ListenSocketImpl 0x150b3ed634d0, transport_protocol_: raw_buffer
  connection_info_provider_: 
    ConnectionInfoSetterImpl 0x150b3ef1cd18, remote_address_: [::1]:46026, direct_remote_address_: [::1]:46026, local_address_: [::1]:33381, server_name_: ]]>