search

solo-io / gloo

The Cloud-Native API Gateway and AI Gateway
https://docs.solo.io/
Apache License 2.0
4.1k stars 446 forks source link

Upstream SSLConfig SNI causes Envoy Crash (Segmentation Fault) #6956

Open pwoelfle opened 2 years ago

pwoelfle commented 2 years ago

Gloo Edge Version

1.12.5

Kubernetes Version

No response

Describe the bug

While updating from Gloo 1.7.x to 1.12.5, we experienced reproducible Gateway Proxy Envoy crashes when using Gloo's Upstream HttpProxyHostname in combination with UpstreamSslConfig's SNI. 

[2022-08-16 05:26:36.412][19][debug][connection] [external/envoy/source/common/network/connection_impl.h:89] [C2] current connecting state: false
[2022-08-16 05:26:36.412][19][debug][router] [external/envoy/source/common/router/router.cc:471] [C2][S17131771344863302925] cluster 'solo_io_generated_self_cluster_local-1_default' match for URL '/test'
[2022-08-16 05:26:36.413][19][critical][backtrace] [external/envoy/source/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x0
[2022-08-16 05:26:36.413][19][critical][backtrace] [external/envoy/source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2022-08-16 05:26:36.413][19][critical][backtrace] [external/envoy/source/server/backtrace.h:92] Envoy version: 258afb6da9c15dd943f202a4d3f4cb5d6e27dace/1.23.0-dev/Distribution/RELEASE/BoringSSL
[2022-08-16 05:26:36.415][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #0: [0x7f5a0e54f930]
[2022-08-16 05:26:36.415][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #1: [0x557c7ccb89cf]
[2022-08-16 05:26:36.415][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #2: [0x557c7ccb7a79]
[2022-08-16 05:26:36.416][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #3: [0x557c7cc9d922]
[2022-08-16 05:26:36.416][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #4: [0x557c7cc9d330]
[2022-08-16 05:26:36.416][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #5: [0x557c7c03552a]
[2022-08-16 05:26:36.417][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #6: [0x557c7d131783]
[2022-08-16 05:26:36.417][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #7: [0x557c7d1382cd]
[2022-08-16 05:26:36.417][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #8: [0x557c7d136590]
[2022-08-16 05:26:36.418][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #9: [0x557c7d05aec4]
[2022-08-16 05:26:36.418][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #10: [0x557c7d04c542]
[2022-08-16 05:26:36.418][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #11: [0x557c7d0872b7]
[2022-08-16 05:26:36.419][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #12: [0x557c7d087622]
[2022-08-16 05:26:36.419][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #13: [0x557c7d084a97]
[2022-08-16 05:26:36.419][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #14: [0x557c7d083fb8]
[2022-08-16 05:26:36.419][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #15: [0x557c7d444879]
[2022-08-16 05:26:36.419][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #16: [0x557c7d083504]
[2022-08-16 05:26:36.419][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #17: [0x557c7d082d5f]
[2022-08-16 05:26:36.419][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #18: [0x557c7d087c18]
[2022-08-16 05:26:36.419][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #19: [0x557c7d04825c]
[2022-08-16 05:26:36.419][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #20: [0x557c7d342565]
[2022-08-16 05:26:36.419][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #21: [0x557c7d33ab3e]
[2022-08-16 05:26:36.419][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #22: [0x557c7d338943]
[2022-08-16 05:26:36.419][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #23: [0x557c7d32b941]
[2022-08-16 05:26:36.419][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #24: [0x557c7d32cd0d]
[2022-08-16 05:26:36.419][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #25: [0x557c7d43fb60]
[2022-08-16 05:26:36.419][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #26: [0x557c7d43e4a1]
[2022-08-16 05:26:36.419][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #27: [0x557c7cc6a70f]
[2022-08-16 05:26:36.420][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #28: [0x557c7d643303]
[2022-08-16 05:26:36.420][19][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #29: [0x7f5a0e5452d6]
ActiveStream 0x458ebf3fa000, stream_id_: 17131771344863302925&filter_manager_: 
  FilterManager 0x458ebf3fa078, state_.has_1xx_headers_: 0
  filter_manager_callbacks_.requestHeaders(): 
    ':authority', 'localhost:11082'
    ':path', '/test'
    ':method', 'POST'
    ':scheme', 'http'
    'user-agent', 'Go-http-client/1.1'
    'content-length', '25'
    'accept-encoding', 'gzip'
    'x-forwarded-proto', 'http'
    'x-request-id', 'fde1c134-e6fb-466b-aff9-d595dabb2f64'
  filter_manager_callbacks_.requestTrailers():   null
  filter_manager_callbacks_.responseHeaders():   null
  filter_manager_callbacks_.responseTrailers():   null
  &stream_info_: 
    StreamInfoImpl 0x458ebf3fa190, protocol_: 1, response_code_: null, response_code_details_: null, attempt_count_: null, health_check_request_: 0, route_name_: gloo-system_vs-route-0-matcher-0    upstream_info_:     null
    OverridableRemoteConnectionInfoSetterStreamInfo 0x458ebf3fa190, remoteAddress(): 172.17.0.1:61874, directRemoteAddress(): 172.17.0.1:61874, localAddress(): 172.17.0.2:11082
Http1::ConnectionImpl 0x458ebf3fd708, dispatching_: 1, dispatching_slice_already_drained_: 0, reset_stream_called_: 0, handling_upgrade_: 0, deferred_end_stream_headers_: 0, processing_trailers_: 0, buffered_body_.length(): 0, header_parsing_state_: Field, current_header_field_: , current_header_value_: 
active_request_: 
, request_url_: null, response_encoder_.local_end_stream_: 0
absl::get<RequestHeaderMapPtr>(headers_or_trailers_): null
current_dispatching_buffer_ front_slice length: 146 contents: "POST /test HTTP/1.1\r\nHost: localhost:11082\r\nUser-Agent: Go-http-client/1.1\r\nContent-Length: 25\r\nAccept-Encoding: gzip\r\n\r\n{\"value\":\"Hello, world!\"}"
ConnectionImpl 0x458ebf387e30, connecting_: 0, bind_error_: 0, state(): Open, read_buffer_limit_: 1048576
socket_: 
  ListenSocketImpl 0x458ebf329100, transport_protocol_: raw_buffer
  connection_info_provider_: 
    ConnectionInfoSetterImpl 0x458ebf382018, remote_address_: 172.17.0.1:61874, direct_remote_address_: 172.17.0.1:61874, local_address_: 172.17.0.2:11082, server_name_:

The crash happens as soon as we send a request using cURL.

Steps to reproduce the bug

The bug is reproducible in Gloo's HTTP Tunneling E2E tests: test/e2e/http_tunneling_test.go

  1. Configure a SNI on the TLS Upstream: test/e2e/http_tunneling_test.go:207

                       sslCfg := &gloov1.UpstreamSslConfig{
            SslSecrets: &gloov1.UpstreamSslConfig_SecretRef{
                SecretRef: &core.ResourceRef{Name: "secret", Namespace: "default"},
            },
        }

        if tlsUpstream {
            up.SslConfig = sslCfg
            up.SslConfig.Sni = envoyInstance.LocalAddr() // <---- an arbitrary SNI
        }
  2. Run the tests in context with back TLS.
  3. The tests will fail and the segmentation fault error is presented in the logs.

Expected Behavior

The Gloo Gateway Proxy Envoy should not crash even if the SNI is not matching.

Additional Context

No response

chrisgaun commented 2 years ago

Thanks for filing. We will take a look and possibly reach out.

pwoelfle commented 2 years ago

The issue only occurs when using an address pipe. Everything works as expected if I change to a loopback socket address.

diff --git a/projects/gloo/pkg/plugins/tunneling/plugin.go b/projects/gloo/pkg/plugins/tunneling/plugin.go
index d5463ff9f..9ff1cb846 100644
--- a/projects/gloo/pkg/plugins/tunneling/plugin.go
+++ b/projects/gloo/pkg/plugins/tunneling/plugin.go
@@ -157,9 +157,11 @@ func generateSelfCluster(selfCluster, selfPipe string, originalTransportSocket *
                            HostIdentifier: &envoy_config_endpoint_v3.LbEndpoint_Endpoint{
                                Endpoint: &envoy_config_endpoint_v3.Endpoint{
                                    Address: &envoy_config_core_v3.Address{
-                                       Address: &envoy_config_core_v3.Address_Pipe{
-                                           Pipe: &envoy_config_core_v3.Pipe{
-                                               Path: selfPipe,
+                                       Address: &envoy_config_core_v3.Address_SocketAddress{
+                                           SocketAddress: &envoy_config_core_v3.SocketAddress{
+                                               Protocol:      envoy_config_core_v3.SocketAddress_TCP,
+                                               Address:       "127.0.0.1",
+                                               PortSpecifier: &envoy_config_core_v3.SocketAddress_PortValue{PortValue: 40000},
                                            },
                                        },
                                    },
@@ -184,9 +186,11 @@ func generateForwardingTcpListener(cluster, selfPipe, tunnelingHostname string,
    return &envoy_config_listener_v3.Listener{
        Name: "solo_io_generated_self_listener_" + cluster,
        Address: &envoy_config_core_v3.Address{
-           Address: &envoy_config_core_v3.Address_Pipe{
-               Pipe: &envoy_config_core_v3.Pipe{
-                   Path: selfPipe,
+           Address: &envoy_config_core_v3.Address_SocketAddress{
+               SocketAddress: &envoy_config_core_v3.SocketAddress{
+                   Protocol:      envoy_config_core_v3.SocketAddress_TCP,
+                   Address:       "127.0.0.1",
+                   PortSpecifier: &envoy_config_core_v3.SocketAddress_PortValue{PortValue: 40000},
                },
            },
        },
pwoelfle commented 2 years ago

I could decrypt the backtrace by building gloo-envoy on my own with debug information (I could not get the debug information of this release from your s3 storage). Unfortunately, the tools/stack_decode.py does not provide any line numbers.

[2022-08-24 05:24:38.213][13][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:304] [C18] new stream
[2022-08-24 05:24:38.213][13][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:908] [C18][S11568683038580642340] request headers complete (end_stream=true):
':authority', 'localhost:11082'
':path', '/test'
':method', 'POST'
'user-agent', 'curl/7.79.1'
'accept', '*/*'

[2022-08-24 05:24:38.214][13][debug][http] [external/envoy/source/common/http/filter_manager.cc:784] [C18][S11568683038580642340] request end stream
[2022-08-24 05:24:38.214][13][debug][connection] [external/envoy/source/common/network/connection_impl.h:88] [C18] current connecting state: false
[2022-08-24 05:24:38.214][13][debug][router] [external/envoy/source/common/router/router.cc:470] [C18][S11568683038580642340] cluster 'solo_io_generated_self_cluster_local-1_default' match for URL '/test'
[2022-08-24 05:24:38.214][13][critical][backtrace] [external/envoy/source/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x0
[2022-08-24 05:24:38.214][13][critical][backtrace] [external/envoy/source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2022-08-24 05:24:38.214][13][critical][backtrace] [external/envoy/source/server/backtrace.h:92] Envoy version: 258afb6da9c15dd943f202a4d3f4cb5d6e27dace/1.23.0-dev/Clean/DEBUG/BoringSSL
[2022-08-24 05:24:38.277][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x564570981beb]
[2022-08-24 05:24:38.277][13][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #1: [0x7fe2a3baa930]
[2022-08-24 05:24:38.332][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #2: Envoy::Upstream::ProdClusterManagerFactory::allocateConnPool() [0x56456f6cc3fd]
[2022-08-24 05:24:38.385][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #3: Envoy::Upstream::ClusterManagerImpl::ThreadLocalClusterManagerImpl::ClusterEntry::httpConnPoolImpl()::{lambda()#1}::operator()() [0x56456f6c9c6f]
[2022-08-24 05:24:38.438][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #4: std::_Function_handler<>::_M_invoke() [0x56456f6d1877]
[2022-08-24 05:24:38.491][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #5: std::function<>::operator()() [0x56456f6fcf11]
[2022-08-24 05:24:38.543][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #6: Envoy::Upstream::ConnPoolMap<>::getPool() [0x56456f6f3cd2]
[2022-08-24 05:24:38.596][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #7: Envoy::Upstream::PriorityConnPoolMap<>::getPool() [0x56456f6e630a]
[2022-08-24 05:24:38.649][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #8: Envoy::Upstream::ClusterManagerImpl::ThreadLocalClusterManagerImpl::ClusterEntry::httpConnPoolImpl() [0x56456f6ca6c7]
[2022-08-24 05:24:38.705][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #9: Envoy::Upstream::ClusterManagerImpl::ThreadLocalClusterManagerImpl::ClusterEntry::httpConnPool() [0x56456f6c08ac]
[2022-08-24 05:24:38.757][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #10: Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::HttpConnPool() [0x56456ccb5535]
[2022-08-24 05:24:38.810][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #11: std::make_unique<>() [0x56456ccb57e3]
[2022-08-24 05:24:38.863][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #12: Envoy::Extensions::Upstreams::Http::Generic::GenericGenericConnPoolFactory::createGenericConnPool() [0x5645701618de]
[2022-08-24 05:24:38.919][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #13: Envoy::Router::Filter::createConnPool() [0x56457016d608]
[2022-08-24 05:24:38.974][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #14: Envoy::Router::Filter::decodeHeaders() [0x56457016b4df]
[2022-08-24 05:24:39.028][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #15: Envoy::Http::ActiveStreamDecoderFilter::decodeHeaders() [0x56456ff66a9f]
[2022-08-24 05:24:39.082][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #16: Envoy::Http::FilterManager::decodeHeaders() [0x56456ff500f8]
[2022-08-24 05:24:39.136][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #17: Envoy::Http::FilterManager::decodeHeaders() [0x56456ff31c58]
[2022-08-24 05:24:39.189][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #18: Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders() [0x56456ff24bb4]
[2022-08-24 05:24:39.242][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #19: Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase() [0x56456ffa6058]
[2022-08-24 05:24:39.299][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #20: Envoy::Http::Http1::ConnectionImpl::onMessageCompleteImpl() [0x56456ffa0b97]
[2022-08-24 05:24:39.355][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #21: Envoy::Http::Http1::ConnectionImpl::onMessageComplete() [0x56456ff9ce74]
[2022-08-24 05:24:39.409][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #22: Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl()::{lambda()#8}::operator()() [0x56456ffb85ad]
[2022-08-24 05:24:39.463][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #23: Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl()::{lambda()#8}::_FUN() [0x56456ffb85cc]
[2022-08-24 05:24:39.517][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #24: http_parser_execute [0x564570960b6b]
[2022-08-24 05:24:39.570][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #25: Envoy::Http::Http1::LegacyHttpParserImpl::Impl::execute() [0x56456ffb8731]
[2022-08-24 05:24:39.623][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #26: Envoy::Http::Http1::LegacyHttpParserImpl::execute() [0x56456ffb802a]
[2022-08-24 05:24:39.676][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #27: Envoy::Http::Http1::ConnectionImpl::dispatchSlice() [0x56456ff9c496]
[2022-08-24 05:24:39.731][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #28: Envoy::Http::Http1::ConnectionImpl::dispatch() [0x56456ff9af87]
[2022-08-24 05:24:39.785][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #29: Envoy::Http::Http1::ServerConnectionImpl::dispatch() [0x56456ffa589f]
[2022-08-24 05:24:39.839][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #30: Envoy::Http::ConnectionManagerImpl::onData() [0x56456ff1aa28]
[2022-08-24 05:24:39.892][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #31: Envoy::Network::FilterManagerImpl::onContinueReading() [0x5645706e1e09]
[2022-08-24 05:24:39.946][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #32: Envoy::Network::FilterManagerImpl::onRead() [0x5645706e2186]
[2022-08-24 05:24:39.999][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #33: Envoy::Network::ConnectionImpl::onRead() [0x5645706cc5f6]
[2022-08-24 05:24:40.054][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #34: Envoy::Network::ConnectionImpl::onReadReady() [0x5645706d2f07]
[2022-08-24 05:24:40.108][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #35: Envoy::Network::ConnectionImpl::onFileEvent() [0x5645706d243c]
[2022-08-24 05:24:40.162][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #36: Envoy::Network::ConnectionImpl::ConnectionImpl()::{lambda()#7}::operator()() [0x5645706c8313]
[2022-08-24 05:24:40.215][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #37: std::_Function_handler<>::_M_invoke() [0x5645706daafb]
[2022-08-24 05:24:40.268][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #38: std::function<>::operator()() [0x56456f2e4ffe]
[2022-08-24 05:24:40.322][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #39: Envoy::Event::DispatcherImpl::createFileEvent()::{lambda()#1}::operator()() [0x56457069cc91]
[2022-08-24 05:24:40.376][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #40: std::_Function_handler<>::_M_invoke() [0x5645706a2f8d]
[2022-08-24 05:24:40.376][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #41: std::function<>::operator()() [0x56456f2e4ffe]
[2022-08-24 05:24:40.430][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #42: Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb() [0x5645706afab5]
[2022-08-24 05:24:40.483][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #43: Envoy::Event::FileEventImpl::assignEvents()::{lambda()#1}::operator()() [0x5645706ade13]
[2022-08-24 05:24:40.537][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #44: Envoy::Event::FileEventImpl::assignEvents()::{lambda()#1}::_FUN() [0x5645706adeb2]
[2022-08-24 05:24:40.590][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #45: event_persist_closure [0x5645709497e9]
[2022-08-24 05:24:40.645][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #46: event_process_active_single_queue [0x564570949ac9]
[2022-08-24 05:24:40.699][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #47: event_process_active [0x56457094a05a]
[2022-08-24 05:24:40.752][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #48: event_base_loop [0x56457094a915]
[2022-08-24 05:24:40.806][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #49: Envoy::Event::LibeventScheduler::run() [0x56457092266c]
[2022-08-24 05:24:40.859][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #50: Envoy::Event::DispatcherImpl::run() [0x56457069fa62]
[2022-08-24 05:24:40.912][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #51: Envoy::Server::WorkerImpl::threadRoutine() [0x56456f61f852]
[2022-08-24 05:24:40.965][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #52: Envoy::Server::WorkerImpl::start()::{lambda()#1}::operator()() [0x56456f61ecae]
[2022-08-24 05:24:41.017][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #53: std::_Function_handler<>::_M_invoke() [0x56456f6208ca]
[2022-08-24 05:24:41.052][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #54: std::function<>::operator()() [0x56456bb0275e]
[2022-08-24 05:24:41.106][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #55: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::operator()() [0x564570cb9534]
[2022-08-24 05:24:41.159][13][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #56: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::_FUN() [0x564570cb9558]
[2022-08-24 05:24:41.159][13][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #57: [0x7fe2a3ba02d6]
ActiveStream 0x64dff01a000, stream_id_: 11568683038580642340&filter_manager_: 
  FilterManager 0x64dff01a078, state_.has_1xx_headers_: 0
  filter_manager_callbacks_.requestHeaders(): 
    ':authority', 'localhost:11082'
    ':path', '/test'
    ':method', 'POST'
    ':scheme', 'http'
    'user-agent', 'curl/7.79.1'
    'accept', '*/*'
    'x-forwarded-proto', 'http'
    'x-request-id', 'c4d987aa-d4c0-47d0-b538-c4fdda82e11d'
  filter_manager_callbacks_.requestTrailers():   null
  filter_manager_callbacks_.responseHeaders():   null
  filter_manager_callbacks_.responseTrailers():   null
  &stream_info_: 
    StreamInfoImpl 0x64dff01a190, protocol_: 1, response_code_: null, response_code_details_: null, attempt_count_: null, health_check_request_: 0, route_name_: gloo-system_vs-route-0-matcher-0    upstream_info_:     null
    OverridableRemoteConnectionInfoSetterStreamInfo 0x64dff01a190, remoteAddress(): 172.17.0.1:60412, directRemoteAddress(): 172.17.0.1:60412, localAddress(): 172.17.0.2:11082
Http1::ConnectionImpl 0x64dff3e0008, dispatching_: 1, dispatching_slice_already_drained_: 0, reset_stream_called_: 0, handling_upgrade_: 0, deferred_end_stream_headers_: 1, processing_trailers_: 0, buffered_body_.length(): 0, header_parsing_state_: Done, current_header_field_: , current_header_value_: 
active_request_: 
, request_url_: null, response_encoder_.local_end_stream_: 0
absl::get<RequestHeaderMapPtr>(headers_or_trailers_): null
current_dispatching_buffer_ front_slice length: 84 contents: "POST /test HTTP/1.1\r\nHost: localhost:11082\r\nUser-Agent: curl/7.79.1\r\nAccept: */*\r\n\r\n"
ConnectionImpl 0x64dfeea7e30, connecting_: 0, bind_error_: 0, state(): Open, read_buffer_limit_: 1048576
socket_: 
  ListenSocketImpl 0x64dff36fa00, transport_protocol_: raw_buffer
  connection_info_provider_: 
    ConnectionInfoSetterImpl 0x64dff3fa630, remote_address_: 172.17.0.1:60412, direct_remote_address_: 172.17.0.1:60412, local_address_: 172.17.0.2:11082, server_name_:
pwoelfle commented 2 years ago

I tried to make a reproducer on the plain Envoy, here I couldn't reproduce the issue.

Reproducer Setup

Squid Proxy (file squid-conf/squid.conf)

http_port 8080
debug_options ALL,1 11,3 31,3 65,3 83,9

dns_v4_first on

# To disable caching
cache deny all
cache_mem 8 MB
cache_dir null /tmp

http_access allow all

Start Squid Proxy container: docker run --name=forward_proxy -v "$(pwd)/squid-conf:/etc/squid" --network bridge --publish 8080:8080 ubuntu/squid:4.13-21.04_edge

Envoy (file envoy-conf/envoy.yaml)

static_resources:
  listeners:
    - name: main_listener
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 8088
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: http
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: app
                      domains:
                        - "*"
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            host_rewrite_literal: webhook.site:443
                            cluster: "loopback_cluster"
                upgrade_configs:
                  - upgrade_type: "websocket"
                http_filters:
                  - name: envoy.filters.http.fault
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault
                  - name: envoy.filters.http.cors
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors
                  - name: envoy.filters.http.grpc_web
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb
                  - name: envoy.filters.http.csrf
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.csrf.v3.CsrfPolicy
                      filter_enabled:
                        default_value: {}
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
    - name: proxy_listener
      address:
        pipe:
          path: "@/cluster_0"
      filter_chains:
        - filters:
            - name: tcp
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
                stat_prefix: tcp_stats
                cluster: "proxy_cluster"
                tunneling_config:
                  hostname: webhook.site:443
  clusters:
    - name: proxy_cluster
      connect_timeout: 0.25s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: proxy_cluster
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: 192.168.50.113 # <------ change to local ip to access squid forward proxy
                      port_value: 8080
    - name: loopback_cluster
      connect_timeout: 5s
      type: STATIC
      load_assignment:
        cluster_name: loopback_cluster
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    pipe:
                      path: "@/cluster_0"
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
          sni: webhook.site

As mentioned in the config, the proxy_cluster cluster socket address must point to your local ip.

Start Envoy container: docker run --rm --name envoy -v $(pwd)/envoy-conf:/etc/envoy --publish 8088:8088 --network bridge envoyproxy/envoy-debug:v1.23.0 --log-level trace --config-path /etc/envoy/envoy.yaml

Reproducer Access https://webhook.site/ and copy path segement (e.g. /742dcb41-d646-4f0c-b1c6-db9b8aec0dc9). On this page you see what the actual server received.

$> curl -v http://localhost:8088/742dcb41-d646-4f0c-b1c6-db9b8aec0dc9

*   Trying 127.0.0.1:8088...
* Connected to localhost (127.0.0.1) port 8088 (#0)
> GET /742dcb41-d646-4f0c-b1c6-db9b8aec0dc9 HTTP/1.1
> Host: localhost:8088
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< server: envoy
< content-type: text/plain; charset=UTF-8
< vary: Accept-Encoding
< x-request-id: 464788f8-6f0b-4de1-a3b1-7b32fdbd61d2
< x-token-id: 742dcb41-d646-4f0c-b1c6-db9b8aec0dc9
< cache-control: no-cache, private
< date: Wed, 24 Aug 2022 09:38:34 GMT
< x-envoy-upstream-service-time: 441
< transfer-encoding: chunked
< 
* Connection #0 to host localhost left intact
pwoelfle commented 2 years ago

If I switch in the reproducer from the plain Envoy image envoyproxy/envoy-debug:v1.23.0 to Gloo's Envoy image quay.io/solo-io/envoy-gloo:1.23.0-patch3 it crashes with same above mentioned segmentation fault.

[2022-08-24 09:51:51.293][15][debug][router] [external/envoy/source/common/router/router.cc:470] [C0][S3319862045371312601] cluster 'loopback_cluster' match for URL '/742dcb41-d646-4f0c-b1c6-db9b8aec0dc9'
[2022-08-24 09:51:51.294][15][critical][backtrace] [external/envoy/source/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x0
[2022-08-24 09:51:51.294][15][critical][backtrace] [external/envoy/source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2022-08-24 09:51:51.294][15][critical][backtrace] [external/envoy/source/server/backtrace.h:92] Envoy version: 258afb6da9c15dd943f202a4d3f4cb5d6e27dace/1.23.0-dev/Clean/DEBUG/BoringSSL
[2022-08-24 09:51:51.365][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x561f557fdbeb]
[2022-08-24 09:51:51.365][15][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #1: [0x7fe8a0ff1930]
[2022-08-24 09:51:51.425][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #2: Envoy::Upstream::ProdClusterManagerFactory::allocateConnPool() [0x561f545483fd]
[2022-08-24 09:51:51.478][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #3: Envoy::Upstream::ClusterManagerImpl::ThreadLocalClusterManagerImpl::ClusterEntry::httpConnPoolImpl()::{lambda()#1}::operator()() [0x561f54545c6f]
[2022-08-24 09:51:51.531][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #4: std::_Function_handler<>::_M_invoke() [0x561f5454d877]
[2022-08-24 09:51:51.585][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #5: std::function<>::operator()() [0x561f54578f11]
[2022-08-24 09:51:51.639][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #6: Envoy::Upstream::ConnPoolMap<>::getPool() [0x561f5456fcd2]
[2022-08-24 09:51:51.692][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #7: Envoy::Upstream::PriorityConnPoolMap<>::getPool() [0x561f5456230a]
[2022-08-24 09:51:51.746][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #8: Envoy::Upstream::ClusterManagerImpl::ThreadLocalClusterManagerImpl::ClusterEntry::httpConnPoolImpl() [0x561f545466c7]
[2022-08-24 09:51:51.801][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #9: Envoy::Upstream::ClusterManagerImpl::ThreadLocalClusterManagerImpl::ClusterEntry::httpConnPool() [0x561f5453c8ac]
[2022-08-24 09:51:51.857][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #10: Envoy::Extensions::Upstreams::Http::Http::HttpConnPool::HttpConnPool() [0x561f51b31535]
[2022-08-24 09:51:51.918][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #11: std::make_unique<>() [0x561f51b317e3]
[2022-08-24 09:51:52.087][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #12: Envoy::Extensions::Upstreams::Http::Generic::GenericGenericConnPoolFactory::createGenericConnPool() [0x561f54fdd8de]
[2022-08-24 09:51:52.300][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #13: Envoy::Router::Filter::createConnPool() [0x561f54fe9608]
[2022-08-24 09:51:52.564][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #14: Envoy::Router::Filter::decodeHeaders() [0x561f54fe74df]
[2022-08-24 09:51:52.770][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #15: Envoy::Http::ActiveStreamDecoderFilter::decodeHeaders() [0x561f54de2a9f]
[2022-08-24 09:51:52.920][7][debug][main] [external/envoy/source/server/server.cc:251] flushing stats
[2022-08-24 09:51:52.920][7][trace][upstream] [external/envoy/source/common/upstream/strict_dns_cluster.cc:105] starting async DNS resolution for 192.168.50.113
[2022-08-24 09:51:52.920][7][debug][dns] [external/envoy/source/extensions/network/dns_resolver/cares/dns_impl.cc:341] dns resolution for 192.168.50.113 started
[2022-08-24 09:51:52.921][7][trace][dns] [external/envoy/source/extensions/network/dns_resolver/cares/dns_impl.cc:304] Setting DNS resolution timer for 5000 milliseconds
[2022-08-24 09:51:52.926][7][debug][dns] [external/envoy/source/extensions/network/dns_resolver/cares/dns_impl.cc:260] dns resolution for 192.168.50.113 completed with status 0
[2022-08-24 09:51:52.926][7][trace][upstream] [external/envoy/source/common/upstream/strict_dns_cluster.cc:113] async DNS resolution complete for 192.168.50.113
[2022-08-24 09:51:52.926][7][debug][upstream] [external/envoy/source/common/upstream/upstream_impl.cc:273] transport socket match, socket default selected for host with address 192.168.50.113:8080
[2022-08-24 09:51:52.926][7][debug][upstream] [external/envoy/source/common/upstream/strict_dns_cluster.cc:177] DNS refresh rate reset for 192.168.50.113, refresh rate 5000 ms
[2022-08-24 09:51:52.946][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #16: Envoy::Http::FilterManager::decodeHeaders() [0x561f54dcc0f8]
[2022-08-24 09:51:53.065][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #17: Envoy::Http::FilterManager::decodeHeaders() [0x561f54dadc58]
[2022-08-24 09:51:53.240][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #18: Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders() [0x561f54da0bb4]
[2022-08-24 09:51:53.365][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #19: Envoy::Http::Http1::ServerConnectionImpl::onMessageCompleteBase() [0x561f54e22058]
[2022-08-24 09:51:53.449][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #20: Envoy::Http::Http1::ConnectionImpl::onMessageCompleteImpl() [0x561f54e1cb97]
[2022-08-24 09:51:53.505][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #21: Envoy::Http::Http1::ConnectionImpl::onMessageComplete() [0x561f54e18e74]
[2022-08-24 09:51:53.560][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #22: Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl()::{lambda()#8}::operator()() [0x561f54e345ad]
[2022-08-24 09:51:53.614][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #23: Envoy::Http::Http1::LegacyHttpParserImpl::Impl::Impl()::{lambda()#8}::_FUN() [0x561f54e345cc]
[2022-08-24 09:51:53.668][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #24: http_parser_execute [0x561f557dcb6b]
[2022-08-24 09:51:53.721][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #25: Envoy::Http::Http1::LegacyHttpParserImpl::Impl::execute() [0x561f54e34731]
[2022-08-24 09:51:53.775][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #26: Envoy::Http::Http1::LegacyHttpParserImpl::execute() [0x561f54e3402a]
[2022-08-24 09:51:53.828][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #27: Envoy::Http::Http1::ConnectionImpl::dispatchSlice() [0x561f54e18496]
[2022-08-24 09:51:53.883][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #28: Envoy::Http::Http1::ConnectionImpl::dispatch() [0x561f54e16f87]
[2022-08-24 09:51:53.936][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #29: Envoy::Http::Http1::ServerConnectionImpl::dispatch() [0x561f54e2189f]
[2022-08-24 09:51:53.989][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #30: Envoy::Http::ConnectionManagerImpl::onData() [0x561f54d96a28]
[2022-08-24 09:51:54.045][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #31: Envoy::Network::FilterManagerImpl::onContinueReading() [0x561f5555de09]
[2022-08-24 09:51:54.098][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #32: Envoy::Network::FilterManagerImpl::onRead() [0x561f5555e186]
[2022-08-24 09:51:54.152][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #33: Envoy::Network::ConnectionImpl::onRead() [0x561f555485f6]
[2022-08-24 09:51:54.206][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #34: Envoy::Network::ConnectionImpl::onReadReady() [0x561f5554ef07]
[2022-08-24 09:51:54.259][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #35: Envoy::Network::ConnectionImpl::onFileEvent() [0x561f5554e43c]
[2022-08-24 09:51:54.313][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #36: Envoy::Network::ConnectionImpl::ConnectionImpl()::{lambda()#7}::operator()() [0x561f55544313]
[2022-08-24 09:51:54.368][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #37: std::_Function_handler<>::_M_invoke() [0x561f55556afb]
[2022-08-24 09:51:54.422][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #38: std::function<>::operator()() [0x561f54160ffe]
[2022-08-24 09:51:54.477][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #39: Envoy::Event::DispatcherImpl::createFileEvent()::{lambda()#1}::operator()() [0x561f55518c91]
[2022-08-24 09:51:54.531][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #40: std::_Function_handler<>::_M_invoke() [0x561f5551ef8d]
[2022-08-24 09:51:54.531][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #41: std::function<>::operator()() [0x561f54160ffe]
[2022-08-24 09:51:54.585][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #42: Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb() [0x561f5552bab5]
[2022-08-24 09:51:54.639][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #43: Envoy::Event::FileEventImpl::assignEvents()::{lambda()#1}::operator()() [0x561f55529e13]
[2022-08-24 09:51:54.693][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #44: Envoy::Event::FileEventImpl::assignEvents()::{lambda()#1}::_FUN() [0x561f55529eb2]
[2022-08-24 09:51:54.747][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #45: event_persist_closure [0x561f557c57e9]
[2022-08-24 09:51:54.801][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #46: event_process_active_single_queue [0x561f557c5ac9]
[2022-08-24 09:51:54.856][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #47: event_process_active [0x561f557c605a]
[2022-08-24 09:51:54.910][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #48: event_base_loop [0x561f557c6915]
[2022-08-24 09:51:54.964][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #49: Envoy::Event::LibeventScheduler::run() [0x561f5579e66c]
[2022-08-24 09:51:55.018][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #50: Envoy::Event::DispatcherImpl::run() [0x561f5551ba62]
[2022-08-24 09:51:55.075][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #51: Envoy::Server::WorkerImpl::threadRoutine() [0x561f5449b852]
[2022-08-24 09:51:55.131][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #52: Envoy::Server::WorkerImpl::start()::{lambda()#1}::operator()() [0x561f5449acae]
[2022-08-24 09:51:55.185][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #53: std::_Function_handler<>::_M_invoke() [0x561f5449c8ca]
[2022-08-24 09:51:55.238][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #54: std::function<>::operator()() [0x561f5097e75e]
[2022-08-24 09:51:55.294][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #55: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::operator()() [0x561f55b35534]
[2022-08-24 09:51:55.350][15][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #56: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::_FUN() [0x561f55b35558]
[2022-08-24 09:51:55.350][15][critical][backtrace] [external/envoy/source/server/backtrace.h:98] #57: [0x7fe8a0fe72d6]
ActiveStream 0x169d7fc62600, stream_id_: 3319862045371312601&filter_manager_: 
  FilterManager 0x169d7fc62678, state_.has_1xx_headers_: 0
  filter_manager_callbacks_.requestHeaders(): 
    ':authority', 'localhost:8088'
    ':path', '/742dcb41-d646-4f0c-b1c6-db9b8aec0dc9'
    ':method', 'GET'
    ':scheme', 'http'
    'user-agent', 'curl/7.79.1'
    'accept', '*/*'
    'x-forwarded-proto', 'http'
    'x-request-id', '211440da-7ccc-4d07-96b4-70f70fe8268e'
  filter_manager_callbacks_.requestTrailers():   null
  filter_manager_callbacks_.responseHeaders():   null
  filter_manager_callbacks_.responseTrailers():   null
  &stream_info_: 
    StreamInfoImpl 0x169d7fc62790, protocol_: 1, response_code_: null, response_code_details_: null, attempt_count_: null, health_check_request_: 0, route_name_:     upstream_info_:     null
    OverridableRemoteConnectionInfoSetterStreamInfo 0x169d7fc62790, remoteAddress(): 172.17.0.1:59630, directRemoteAddress(): 172.17.0.1:59630, localAddress(): 172.17.0.3:8088
Http1::ConnectionImpl 0x169d7fcba688, dispatching_: 1, dispatching_slice_already_drained_: 0, reset_stream_called_: 0, handling_upgrade_: 0, deferred_end_stream_headers_: 1, processing_trailers_: 0, buffered_body_.length(): 0, header_parsing_state_: Done, current_header_field_: , current_header_value_: 
active_request_: 
, request_url_: null, response_encoder_.local_end_stream_: 0
absl::get<RequestHeaderMapPtr>(headers_or_trailers_): null
current_dispatching_buffer_ front_slice length: 114 contents: "GET /742dcb41-d646-4f0c-b1c6-db9b8aec0dc9 HTTP/1.1\r\nHost: localhost:8088\r\nUser-Agent: curl/7.79.1\r\nAccept: */*\r\n\r\n"
ConnectionImpl 0x169d7f299e30, connecting_: 0, bind_error_: 0, state(): Open, read_buffer_limit_: 1048576
socket_: 
  ListenSocketImpl 0x169d7f382500, transport_protocol_: raw_buffer
  connection_info_provider_: 
    ConnectionInfoSetterImpl 0x169d7f3a1430, remote_address_: 172.17.0.1:59630, direct_remote_address_: 172.17.0.1:59630, local_address_: 172.17.0.3:8088, server_name_:
pwoelfle commented 2 years ago

I've tested all older releases of quay.io/solo-io/envoy-gloo and figured out that the segmentation fault was introduced in version 1.23.0-patch1. With the previous version 1.22.0-patch1 everything is working as expected.

In 1.23.0-patch1, Support API Gateway Transformer in AWS Lambda filter (#167) was introduced as the only feature (see commit 258afb6da9).

pwoelfle commented 2 years ago

I've just locally built the enovy-gloo on commit e4f7873b301c and there the segmentation fault occurs in the reproducer. So, the Support API Gateway Transformer in AWS Lambda filter is not the reason for the segmentation fault.

pwoelfle commented 2 years ago

If I run the reproducer against the envoy 1.23 pre-ga version which you use (see https://github.com/solo-io/envoy-gloo/blob/v1.23.0-patch3/bazel/repository_locations.bzl#L4), then I can reproduce the segmentation fault.

Just use the Envoy image envoyproxy/envoy-dev:44634cc608f11bed6852c67f40902cea0d5c9554 in my reproducer above and you get the segmentation fault.

pwoelfle commented 2 years ago

Problem has been fixed in Gloo's Envoy with release v1.23.0-patch4.

@chrisgaun Is it possible to have this patch in Gloo v1.12.x? We really need this fix.

github-actions[bot] commented 5 months ago

This issue has been marked as stale because of no activity in the last 180 days. It will be closed in the next 180 days unless it is tagged "no stalebot" or other activity occurs.