search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
24.85k stars 4.78k forks source link

Envoy `Lua` filter causes a crash when code in either `envoy_on_request` or `envoy_on_response ` calls `requestInfo():dynamicMetadata()` #4305

Closed yskopets closed 6 years ago

yskopets commented 6 years ago

Issue

Envoy Lua filter causes a crash when code inside function envoy_on_response(response_handle) calls response_handle:requestInfo():dynamicMetadata()

Description:

I'm trying to use Lua filter to implement redirect to a Login Page whenever HTTP Status 401 happens.

Here is my Envoy config (relevant part):

  http_filters:

  #
  # first, I want to save ':authority' in dynamic metadata for later use
  #
  - name: envoy.filters.http.header_to_metadata
    config:
      request_rules:
        - header: ':authority'
          on_header_present:
            metadata_namespace: my.gateway
            key: ':authority'
            type: STRING
          on_header_missing:
            metadata_namespace: my.gateway
            key: ':authority'
            value: 'unknown'
            type: STRING
          remove: false

  #
  # then, I want to replace HTTP Status 401 with HTTP Status 301
  #
  - name: envoy.lua
    config:
      inline_code: |
        function envoy_on_response(response_handle)
          if response_handle:headers():get(":status") == "401" then
            response_handle:headers():replace(":status", "301")
            response_handle:headers():replace("location", "https://login.example.org/login?redirect=https://" .. response_handle:requestInfo():dynamicMetadata():get("my.gateway")[":authority"])
          end
        end

and here are Envoy logs:

[2018-08-30 13:29:45.066][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Segmentation fault, suspect faulting address 0x0
[2018-08-30 13:29:45.067][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<36> obj</usr/local/bin/envoy> (If unsymbolized, use tools/stack_decode.py):
[2018-08-30 13:29:45.069][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #0 0x60d8af lua_rawgeti
[2018-08-30 13:29:45.072][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #1 0x61b9f2 luaL_unref
[2018-08-30 13:29:45.074][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #2 0x5ffadf Envoy::Extensions::HttpFilters::Lua::RequestInfoWrapper::~RequestInfoWrapper()
[2018-08-30 13:29:45.076][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #3 0x5ffdb6 Envoy::Extensions::Filters::Common::Lua::BaseLuaObject<>::registerType()::{lambda()#1}::_FUN()
[2018-08-30 13:29:45.079][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #4 0x61f058 lj_BC_FUNCC
[2018-08-30 13:29:45.081][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #5 0x62134b gc_call_finalizer
[2018-08-30 13:29:45.083][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #6 0x621427 gc_finalize
[2018-08-30 13:29:45.086][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #7 0x622146 gc_onestep
[2018-08-30 13:29:45.102][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #8 0x62264b lj_gc_step
[2018-08-30 13:29:45.104][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #9 0x60d0c4 lua_pushstring
[2018-08-30 13:29:45.106][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #10 0x603d58 Envoy::Extensions::HttpFilters::Lua::HeaderMapWrapper::luaGet()
[2018-08-30 13:29:45.106][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #11 0x61f058 lj_BC_FUNCC
[2018-08-30 13:29:45.108][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #12 0x606fa7 Envoy::Extensions::Filters::Common::Lua::Coroutine::resume()
[2018-08-30 13:29:45.110][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #13 0x5fca10 Envoy::Extensions::HttpFilters::Lua::StreamHandleWrapper::start()
[2018-08-30 13:29:45.112][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #14 0x6015c7 Envoy::Extensions::HttpFilters::Lua::Filter::doHeaders()
[2018-08-30 13:29:45.114][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #15 0x6016bd Envoy::Extensions::HttpFilters::Lua::Filter::encodeHeaders()
[2018-08-30 13:29:45.116][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #16 0x93d8c9 Envoy::Http::ConnectionManagerImpl::ActiveStream::encodeHeaders()
[2018-08-30 13:29:45.119][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #17 0x9b3c84 Envoy::Http::Utility::sendLocalReply()
[2018-08-30 13:29:45.121][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #18 0x9369c0 Envoy::Http::ConnectionManagerImpl::ActiveStream::sendLocalReply()
[2018-08-30 13:29:45.123][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #19 0x936c27 Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter::sendLocalReply()
[2018-08-30 13:29:45.125][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #20 0x47a4cf Envoy::Http::Istio::AuthN::AuthenticationFilter::rejectRequest()
[2018-08-30 13:29:45.127][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #21 0x48259e Envoy::Http::Istio::AuthN::AuthenticationFilter::decodeHeaders()
[2018-08-30 13:29:45.129][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #22 0x93c7c4 Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders()
[2018-08-30 13:29:45.131][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #23 0x93cf6d Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders()
[2018-08-30 13:29:45.133][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #24 0x97b649 Envoy::Http::Http1::ServerConnectionImpl::onMessageComplete()
[2018-08-30 13:29:45.135][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #25 0x97ef78 Envoy::Http::Http1::ConnectionImpl::onMessageCompleteBase()
[2018-08-30 13:29:45.137][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #26 0x97efec Envoy::Http::Http1::ConnectionImpl::{lambda()#7}::_FUN()
[2018-08-30 13:29:45.140][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #27 0x98207e http_parser_execute
[2018-08-30 13:29:45.142][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #28 0x97c5b0 Envoy::Http::Http1::ConnectionImpl::dispatchSlice()
[2018-08-30 13:29:45.144][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #29 0x97ea8a Envoy::Http::Http1::ConnectionImpl::dispatch()
[2018-08-30 13:29:45.146][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #30 0x93a59f Envoy::Http::ConnectionManagerImpl::onData()
[2018-08-30 13:29:45.148][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #31 0x7c1d36 Envoy::Network::FilterManagerImpl::onContinueReading()
[2018-08-30 13:29:45.150][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #32 0x7c03be Envoy::Network::ConnectionImpl::onReadReady()
[2018-08-30 13:29:45.152][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #33 0x7c0bad Envoy::Network::ConnectionImpl::onFileEvent()
[2018-08-30 13:29:45.154][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #34 0x7ba007 Envoy::Event::FileEventImpl::assignEvents()::{lambda()#1}::_FUN()
[2018-08-30 13:29:45.156][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #35 0xabd184 event_process_active_single_queue.isra.29
[2018-08-30 13:29:45.158][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #36 0xabd8ce event_process_active
[2018-08-30 13:29:45.160][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #37 0xac0657 event_base_loop
[2018-08-30 13:29:45.163][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #38 0x7b5775 Envoy::Server::WorkerImpl::threadRoutine()
[2018-08-30 13:29:45.165][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #39 0xb0de0d Envoy::Thread::Thread::Thread()::{lambda()#1}::_FUN()
[2018-08-30 13:29:45.165][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<36> obj</lib/x86_64-linux-gnu/libpthread.so.0>
[2018-08-30 13:29:45.165][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<36> #40 0x7fd108dca6b9 start_thread
[2018-08-30 13:29:45.165][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<36> obj</lib/x86_64-linux-gnu/libc.so.6>
[2018-08-30 13:29:45.165][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<36> #41 0x7fd1087f741c (unknown)
[2018-08-30 13:29:45.165][36][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:121] end backtrace thread 36
2018-08-30T13:29:45.194548Z warn    Epoch 0 terminated with an error: signal: segmentation fault (core dumped)
2018-08-30T13:29:45.194630Z warn    Aborted all epochs
2018-08-30T13:29:45.194721Z info    Epoch 0: set retry delay to 200ms, budget to 9
2018-08-30T13:29:45.395197Z info    Reconciling configuration (budget 9)
2018-08-30T13:29:45.395313Z info    Epoch 0 starting
2018-08-30T13:29:45.396136Z info    Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~10.34.4.18~istio-ingressgateway-67989599df-8tqm8.istio-ingress-internal~istio-ingress-internal.svc.cluster.local --max-obj-name-len 189 -l warn --v2-config-only]
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:183] initializing epoch 0 (hot restart version=10.200.16384.256.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=4882536)
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:185] statically linked extensions:
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:187]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:190]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash,istio_authn,jwt-auth,mixer
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:193]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:196]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy,mixer
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:198]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:200]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.zipkin
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:203]   transport_sockets.downstream: alts,envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-30 13:29:45.413][43][info][main] external/envoy/source/server/server.cc:206]   transport_sockets.upstream: alts,envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-30 13:29:45.416][43][info][config] external/envoy/source/server/configuration_impl.cc:50] loading 0 static secret(s)
[2018-08-30 13:29:45.418][43][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:240] gRPC config stream closed: 14, no healthy upstream
[2018-08-30 13:29:45.418][43][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:41] Unable to establish new stream
[2018-08-30 13:29:45.418][43][info][config] external/envoy/source/server/configuration_impl.cc:60] loading 0 listener(s)
[2018-08-30 13:29:45.418][43][info][config] external/envoy/source/server/configuration_impl.cc:94] loading tracing configuration
[2018-08-30 13:29:45.418][43][info][config] external/envoy/source/server/configuration_impl.cc:103]   loading tracing driver: envoy.zipkin
[2018-08-30 13:29:45.418][43][info][config] external/envoy/source/server/configuration_impl.cc:116] loading stats sink configuration
[2018-08-30 13:29:45.419][43][info][main] external/envoy/source/server/server.cc:410] starting main dispatch loop
[2018-08-30 13:29:45.454][43][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:129] cm init: initializing cds

Repro steps:

  1. Use example configuration that comes with Envoy (google proxy) and add the following Lua code:
          http_filters:
          - name: envoy.lua
            config:
              inline_code: |
                function envoy_on_request(request_handle)
                  request_handle:headers():add("foo", "bar")
                end
                function envoy_on_response(response_handle)
                  response_handle:requestInfo():dynamicMetadata()
                end
  1. Make a few requests to cause a crash (in my case, Envoy crashes consistently after 5-10 requests)

Config:

Here is the minimal Envoy configuration to reproduce the issue:

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address:
      protocol: TCP
      address: 127.0.0.1
      port_value: 9901
static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  host_rewrite: www.google.com
                  cluster: service_google
          http_filters:
          - name: envoy.lua
            config:
              inline_code: |
                function envoy_on_request(request_handle)
                  request_handle:headers():add("foo", "bar")
                end
                function envoy_on_response(response_handle)
                  response_handle:requestInfo():dynamicMetadata()
                end
          - name: envoy.router
  clusters:
  - name: service_google
    connect_timeout: 0.25s
    type: LOGICAL_DNS
    # Comment out the following line to test on v6 networks
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    hosts:
      - socket_address:
          address: google.com
          port_value: 443
    tls_context: { sni: www.google.com }

Call Stack:

[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:191] initializing epoch 0 (hot restart version=10.200.16384.127.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=2654312)
[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:193] statically linked extensions:
[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:195]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:198]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:201]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:204]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.rbac,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:206]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:208]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.zipkin
[2018-08-30 14:45:22.338][8][info][main] source/server/server.cc:211]   transport_sockets.downstream: envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-30 14:45:22.339][8][info][main] source/server/server.cc:214]   transport_sockets.upstream: envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-30 14:45:22.345][8][info][config] source/server/configuration_impl.cc:50] loading 0 static secret(s)
[2018-08-30 14:45:22.345][8][info][config] source/server/configuration_impl.cc:56] loading 1 cluster(s)
[2018-08-30 14:45:22.346][8][info][config] source/server/configuration_impl.cc:61] loading 1 listener(s)
[2018-08-30 14:45:22.348][8][info][config] source/server/configuration_impl.cc:95] loading tracing configuration
[2018-08-30 14:45:22.348][8][info][config] source/server/configuration_impl.cc:117] loading stats sink configuration
[2018-08-30 14:45:22.349][8][info][main] source/server/server.cc:435] starting main dispatch loop
[2018-08-30 14:45:22.356][8][info][upstream] source/common/upstream/cluster_manager_impl.cc:134] cm init: all clusters initialized
[2018-08-30 14:45:22.356][8][info][main] source/server/server.cc:404] all clusters initialized. initializing init manager
[2018-08-30 14:45:22.356][8][info][config] source/server/listener_manager_impl.cc:907] all dependencies initialized. starting workers
[2018-08-30 14:45:36.211][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Segmentation fault, suspect faulting address 0x18
[2018-08-30 14:45:36.212][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<16> obj</usr/local/bin/envoy> (If unsymbolized, use tools/stack_decode.py):
[2018-08-30 14:45:36.215][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #0 0x51ea66 lua_rawgeti
[2018-08-30 14:45:36.218][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #1 0x52cbc2 luaL_unref
[2018-08-30 14:45:36.221][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #2 0x510c4f Envoy::Extensions::HttpFilters::Lua::RequestInfoWrapper::~RequestInfoWrapper()
[2018-08-30 14:45:36.223][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #3 0x510ce6 Envoy::Extensions::Filters::Common::Lua::BaseLuaObject<>::registerType()::{lambda()#1}::_FUN()
[2018-08-30 14:45:36.225][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #4 0x530228 lj_BC_FUNCC
[2018-08-30 14:45:36.228][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #5 0x53251b gc_call_finalizer
[2018-08-30 14:45:36.231][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #6 0x5325f7 gc_finalize
[2018-08-30 14:45:36.234][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #7 0x533316 gc_onestep
[2018-08-30 14:45:36.237][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #8 0x53381b lj_gc_step
[2018-08-30 14:45:36.240][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #9 0x51e7f8 lua_newuserdata
[2018-08-30 14:45:36.242][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #10 0x511d23 Envoy::Extensions::HttpFilters::Lua::StreamHandleWrapper::luaHeaders()
[2018-08-30 14:45:36.242][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #11 0x530228 lj_BC_FUNCC
[2018-08-30 14:45:36.244][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #12 0x518177 Envoy::Extensions::Filters::Common::Lua::Coroutine::resume()
[2018-08-30 14:45:36.246][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #13 0x50db90 Envoy::Extensions::HttpFilters::Lua::StreamHandleWrapper::start()
[2018-08-30 14:45:36.248][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #14 0x512737 Envoy::Extensions::HttpFilters::Lua::Filter::doHeaders()
[2018-08-30 14:45:36.250][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #15 0x51289d Envoy::Extensions::HttpFilters::Lua::Filter::decodeHeaders()
[2018-08-30 14:45:36.251][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #16 0x8764a4 Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders()
[2018-08-30 14:45:36.253][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #17 0x876c65 Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders()
[2018-08-30 14:45:36.256][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #18 0x8b6b49 Envoy::Http::Http1::ServerConnectionImpl::onMessageComplete()
[2018-08-30 14:45:36.258][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #19 0x8ba53b Envoy::Http::Http1::ConnectionImpl::onMessageCompleteBase()
[2018-08-30 14:45:36.260][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #20 0x8ba5ac Envoy::Http::Http1::ConnectionImpl::{lambda()#7}::_FUN()
[2018-08-30 14:45:36.262][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #21 0x8bd67e http_parser_execute
[2018-08-30 14:45:36.265][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #22 0x8b7ab0 Envoy::Http::Http1::ConnectionImpl::dispatchSlice()
[2018-08-30 14:45:36.268][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #23 0x8ba04a Envoy::Http::Http1::ConnectionImpl::dispatch()
[2018-08-30 14:45:36.270][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #24 0x8749ff Envoy::Http::ConnectionManagerImpl::onData()
[2018-08-30 14:45:36.272][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #25 0x6f50a6 Envoy::Network::FilterManagerImpl::onContinueReading()
[2018-08-30 14:45:36.275][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #26 0x6f36be Envoy::Network::ConnectionImpl::onReadReady()
[2018-08-30 14:45:36.276][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #27 0x6f3ead Envoy::Network::ConnectionImpl::onFileEvent()
[2018-08-30 14:45:36.278][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #28 0x6ed057 Envoy::Event::FileEventImpl::assignEvents()::{lambda()#1}::_FUN()
[2018-08-30 14:45:36.280][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #29 0xa02604 event_process_active_single_queue.isra.29
[2018-08-30 14:45:36.282][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #30 0xa02d4e event_process_active
[2018-08-30 14:45:36.285][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #31 0xa05ad7 event_base_loop
[2018-08-30 14:45:36.287][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #32 0x6e8785 Envoy::Server::WorkerImpl::threadRoutine()
[2018-08-30 14:45:36.291][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<16> #33 0xa53ead Envoy::Thread::Thread::Thread()::{lambda()#1}::_FUN()
[2018-08-30 14:45:36.291][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<16> obj</usr/glibc-compat/lib/libpthread.so.0>
[2018-08-30 14:45:36.291][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<16> #34 0x7fd0d9737d2b (unknown)
[2018-08-30 14:45:36.291][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<16> obj</usr/glibc-compat/lib/libc.so.6>
[2018-08-30 14:45:36.291][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<16> #35 0x7fd0d90e7e3e (unknown)
[2018-08-30 14:45:36.291][16][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:121] end backtrace thread 16
dio commented 6 years ago

@yskopets thanks for this.

This happens when the ~RequestInfoWrapper is called (when the gc kicks in). Need to check the sequence.

dio commented 6 years ago

It's missing resetting dynamic_metadata_wrapper_ when RequestInfoWrapper receives onMarkDead. Thinking about tests now.

yskopets commented 6 years ago

@dio Thanks a lot for the quick feedback!

I have another question about correctness of my approach.

As you can see, I'm trying to change response entity inside envoy_on_response callback.

Since respond() function is not available for use inside envoy_on_response, I have to fallback to response_handle:headers():replace(":status", "301"), which seems very brittle:

So, I'm wondering if I'm using Lua filter the right way.

Do you have any plans to add support for respond() inside envoy_on_response ? Or is it problematic due to the reasons I'm not seeing ?

yskopets commented 6 years ago

I've notice that the issue is also reproducible when function envoy_on_request(request_handle) calls request_handle:requestInfo():dynamicMetadata()

Repro steps:

  1. Use example configuration that comes with Envoy (google proxy) and add the following Lua code:
          http_filters:
          - name: envoy.lua
            config:
              inline_code: |
                function envoy_on_request(request_handle)
                  request_handle:requestInfo():dynamicMetadata()
                end
          - name: envoy.router
  1. Make a few requests to cause a crash (in my case, Envoy crashes consistently after 5-10 requests)

Config:

Here is the minimal Envoy configuration to reproduce the issue:

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address:
      protocol: TCP
      address: 127.0.0.1
      port_value: 9901
static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  host_rewrite: www.google.com
                  cluster: service_google
          http_filters:
          - name: envoy.lua
            config:
              inline_code: |
                function envoy_on_request(request_handle)
                  request_handle:requestInfo():dynamicMetadata()
                end
          - name: envoy.router
  clusters:
  - name: service_google
    connect_timeout: 0.25s
    type: LOGICAL_DNS
    # Comment out the following line to test on v6 networks
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    hosts:
      - socket_address:
          address: google.com
          port_value: 443
    tls_context: { sni: www.google.com }

Call Stack:

[2018-08-31 15:53:30.021][8][info][main] source/server/server.cc:202] initializing epoch 0 (hot restart version=10.200.16384.127.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=2654312)
[2018-08-31 15:53:30.021][8][info][main] source/server/server.cc:204] statically linked extensions:
[2018-08-31 15:53:30.021][8][info][main] source/server/server.cc:206]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2018-08-31 15:53:30.021][8][info][main] source/server/server.cc:209]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2018-08-31 15:53:30.022][8][info][main] source/server/server.cc:212]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2018-08-31 15:53:30.022][8][info][main] source/server/server.cc:215]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.rbac,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2018-08-31 15:53:30.022][8][info][main] source/server/server.cc:217]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2018-08-31 15:53:30.022][8][info][main] source/server/server.cc:219]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.zipkin
[2018-08-31 15:53:30.022][8][info][main] source/server/server.cc:222]   transport_sockets.downstream: envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-31 15:53:30.022][8][info][main] source/server/server.cc:225]   transport_sockets.upstream: envoy.transport_sockets.capture,raw_buffer,tls
[2018-08-31 15:53:30.032][8][info][config] source/server/configuration_impl.cc:50] loading 0 static secret(s)
[2018-08-31 15:53:30.032][8][info][config] source/server/configuration_impl.cc:56] loading 1 cluster(s)
[2018-08-31 15:53:30.034][8][info][config] source/server/configuration_impl.cc:61] loading 1 listener(s)
[2018-08-31 15:53:30.035][8][info][lua] source/extensions/filters/http/lua/lua_filter.cc:449] envoy_on_response() function not found. Lua filter will not hook responses.
[2018-08-31 15:53:30.035][8][info][config] source/server/configuration_impl.cc:95] loading tracing configuration
[2018-08-31 15:53:30.035][8][info][config] source/server/configuration_impl.cc:117] loading stats sink configuration
[2018-08-31 15:53:30.036][8][info][main] source/server/server.cc:446] starting main dispatch loop
[2018-08-31 15:53:30.043][8][info][upstream] source/common/upstream/cluster_manager_impl.cc:134] cm init: all clusters initialized
[2018-08-31 15:53:30.043][8][info][main] source/server/server.cc:415] all clusters initialized. initializing init manager
[2018-08-31 15:53:30.043][8][info][config] source/server/listener_manager_impl.cc:907] all dependencies initialized. starting workers
[2018-08-31 15:53:36.920][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Segmentation fault, suspect faulting address 0x98
[2018-08-31 15:53:36.920][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<17> obj</usr/local/bin/envoy> (If unsymbolized, use tools/stack_decode.py):
[2018-08-31 15:53:36.923][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #0 0x51eaa4 lua_rawgeti
[2018-08-31 15:53:36.925][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #1 0x52cc02 luaL_unref
[2018-08-31 15:53:36.928][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #2 0x510c8f Envoy::Extensions::HttpFilters::Lua::RequestInfoWrapper::~RequestInfoWrapper()
[2018-08-31 15:53:36.930][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #3 0x510d26 Envoy::Extensions::Filters::Common::Lua::BaseLuaObject<>::registerType()::{lambda()#1}::_FUN()
[2018-08-31 15:53:36.932][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #4 0x530268 lj_BC_FUNCC
[2018-08-31 15:53:36.934][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #5 0x53255b gc_call_finalizer
[2018-08-31 15:53:36.936][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #6 0x532637 gc_finalize
[2018-08-31 15:53:36.938][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #7 0x533356 gc_onestep
[2018-08-31 15:53:36.942][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #8 0x53385b lj_gc_step
[2018-08-31 15:53:36.944][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #9 0x51e838 lua_newuserdata
[2018-08-31 15:53:36.948][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #10 0x5125b6 Envoy::Extensions::HttpFilters::Lua::Filter::doHeaders()
[2018-08-31 15:53:36.950][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #11 0x5128dd Envoy::Extensions::HttpFilters::Lua::Filter::decodeHeaders()
[2018-08-31 15:53:36.952][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #12 0x879de4 Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders()
[2018-08-31 15:53:36.954][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #13 0x87a5a5 Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders()
[2018-08-31 15:53:36.956][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #14 0x8ba489 Envoy::Http::Http1::ServerConnectionImpl::onMessageComplete()
[2018-08-31 15:53:36.959][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #15 0x8bde7b Envoy::Http::Http1::ConnectionImpl::onMessageCompleteBase()
[2018-08-31 15:53:36.962][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #16 0x8bdeec Envoy::Http::Http1::ConnectionImpl::{lambda()#7}::_FUN()
[2018-08-31 15:53:36.965][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #17 0x8c03ee http_parser_execute
[2018-08-31 15:53:36.967][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #18 0x8bb3f0 Envoy::Http::Http1::ConnectionImpl::dispatchSlice()
[2018-08-31 15:53:36.970][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #19 0x8bd98a Envoy::Http::Http1::ConnectionImpl::dispatch()
[2018-08-31 15:53:36.972][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #20 0x87833f Envoy::Http::ConnectionManagerImpl::onData()
[2018-08-31 15:53:36.975][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #21 0x6f89e6 Envoy::Network::FilterManagerImpl::onContinueReading()
[2018-08-31 15:53:36.978][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #22 0x6f6ffe Envoy::Network::ConnectionImpl::onReadReady()
[2018-08-31 15:53:36.982][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #23 0x6f77ed Envoy::Network::ConnectionImpl::onFileEvent()
[2018-08-31 15:53:36.984][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #24 0x6f0997 Envoy::Event::FileEventImpl::assignEvents()::{lambda()#1}::_FUN()
[2018-08-31 15:53:36.987][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #25 0xa05f44 event_process_active_single_queue.isra.29
[2018-08-31 15:53:36.990][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #26 0xa0668e event_process_active
[2018-08-31 15:53:36.993][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #27 0xa09417 event_base_loop
[2018-08-31 15:53:36.996][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #28 0x6ec0c5 Envoy::Server::WorkerImpl::threadRoutine()
[2018-08-31 15:53:36.998][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<17> #29 0xa577ed Envoy::Thread::Thread::Thread()::{lambda()#1}::_FUN()
[2018-08-31 15:53:36.998][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<17> obj</usr/glibc-compat/lib/libpthread.so.0>
[2018-08-31 15:53:36.998][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<17> #30 0x7f9304102d2b (unknown)
[2018-08-31 15:53:36.998][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<17> obj</usr/glibc-compat/lib/libc.so.6>
[2018-08-31 15:53:36.998][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<17> #31 0x7f9303ab2e3e (unknown)
[2018-08-31 15:53:36.998][17][critical][backtrace] bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:121] end backtrace thread 17
dio commented 6 years ago

@yskopets yes, thanks to you that I can reproduce the issue well. Would you mind building envoy with this patch: https://github.com/envoyproxy/envoy/pull/4312 and see if that fixes yours? (If you have a problem on building envoy from the source you can ping me via slack (@dio at https://envoyslack.cncf.io/) or email). Thanks!

yskopets commented 6 years ago

@dio Thanks a lot!

I confirm that your changes fixed my problem.

ceastman-ibm commented 5 years ago

@yskopets after this patch was your code for redirecting on status code 401 working?