search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
24.75k stars 4.75k forks source link

Empty reply when rate limit grpc service initial_metadata uses certain command operators #19325

Closed petedmarsh closed 2 years ago

petedmarsh commented 2 years ago

If you are reporting any crash or any potential security issue, do not open an issue in this repo. Please report the issue via emailing envoy-security@googlegroups.com where the issue will be triaged appropriately.

Title: Empty reply when rate limit grpc service initial_metadata uses certain command operators

Description:

If you attempt to use certain command operators (e.g. %ROUTE_NAME%) as the value of a header in rate_limit_service.grpc_service.initial_metadata requests will fail and receive and empty body response while a stacktrace will appear in Envoy logs.

e.g.

    rate_limit_service:
      grpc_service:
        envoy_grpc:
          cluster_name: ratelimit
        initial_metadata:
          - key: my-lovely-key
            value: "%ROUTE_NAME%" # using %ROUTE_NAME% here causes an issue, others like %REQ(:PATH)% work fine

To my mind these operators should work in this context, i.e. in the example %ROUTE_NAME% should resolve to ghi, but I can see how that may not be possible, they should resolve to "-" if it's not possible to make them work at all.

Repro steps:

Working example (steps in commit message):

https://github.com/envoyproxy/ratelimit/compare/main...petedmarsh:rate_limit_service-route_name-empty-reply-example?expand=1

Note: The Envoy_collect tool gathers a tarball with debug logs, config and the following admin endpoints: /stats, /clusters and /server_info. Please note if there are privacy concerns, sanitize the data prior to sharing the tarball/pasting.

Admin and Stats Output:

Include the admin output for the following endpoints: /stats, /clusters, /routes, /server_info. For more information, refer to the admin endpoint documentation.

Note: If there are privacy concerns, sanitize the data prior to sharing.

Config:

See working example

Logs:

envoy-proxy_1  | [2021-12-20 22:12:19.369][14][critical][main] [source/exe/terminate_handler.cc:12] std::terminate called! (possible uncaught exception, see trace)
envoy-proxy_1  | [2021-12-20 22:12:19.370][14][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
envoy-proxy_1  | [2021-12-20 22:12:19.370][14][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 2fe62422af30091ededf216a64db158ed073f3f3/1.21.0-dev/Clean/RELEASE/BoringSSL
envoy-proxy_1  | [2021-12-20 22:12:19.370][14][critical][backtrace] [./source/server/backtrace.h:98] #0: [0x55be98d31dce]
envoy-proxy_1  | [2021-12-20 22:12:19.370][14][critical][backtrace] [./source/server/backtrace.h:98] #1: [0x55be98d31c99]
envoy-proxy_1  | [2021-12-20 22:12:19.371][14][critical][backtrace] [./source/server/backtrace.h:98] #2: [0x55be991de1f3]
envoy-proxy_1  | [2021-12-20 22:12:19.371][14][critical][backtrace] [./source/server/backtrace.h:98] #3: [0x55be9895099d]
envoy-proxy_1  | [2021-12-20 22:12:19.372][14][critical][backtrace] [./source/server/backtrace.h:98] #4: [0x55be98951a17]
envoy-proxy_1  | [2021-12-20 22:12:19.372][14][critical][backtrace] [./source/server/backtrace.h:98] #5: [0x55be988dcb5a]
envoy-proxy_1  | [2021-12-20 22:12:19.372][14][critical][backtrace] [./source/server/backtrace.h:98] #6: [0x55be984f203b]
envoy-proxy_1  | [2021-12-20 22:12:19.372][14][critical][backtrace] [./source/server/backtrace.h:98] #7: [0x55be984f2998]
envoy-proxy_1  | [2021-12-20 22:12:19.373][14][critical][backtrace] [./source/server/backtrace.h:98] #8: [0x55be9763eb73]
envoy-proxy_1  | [2021-12-20 22:12:19.373][14][critical][backtrace] [./source/server/backtrace.h:98] #9: [0x55be97416be8]
envoy-proxy_1  | [2021-12-20 22:12:19.373][14][critical][backtrace] [./source/server/backtrace.h:98] #10: [0x55be987f9dad]
envoy-proxy_1  | [2021-12-20 22:12:19.373][14][critical][backtrace] [./source/server/backtrace.h:98] #11: [0x55be9882bdde]
envoy-proxy_1  | [2021-12-20 22:12:19.373][14][critical][backtrace] [./source/server/backtrace.h:98] #12: [0x55be98815ac4]
envoy-proxy_1  | [2021-12-20 22:12:19.374][14][critical][backtrace] [./source/server/backtrace.h:98] #13: [0x55be98859ef3]
envoy-proxy_1  | [2021-12-20 22:12:19.374][14][critical][backtrace] [./source/server/backtrace.h:98] #14: [0x55be98857587]
envoy-proxy_1  | [2021-12-20 22:12:19.374][14][critical][backtrace] [./source/server/backtrace.h:98] #15: [0x55be9886053f]
envoy-proxy_1  | [2021-12-20 22:12:19.374][14][critical][backtrace] [./source/server/backtrace.h:98] #16: [0x55be98be3e73]
envoy-proxy_1  | [2021-12-20 22:12:19.375][14][critical][backtrace] [./source/server/backtrace.h:98] #17: [0x55be9885feef]
envoy-proxy_1  | [2021-12-20 22:12:19.375][14][critical][backtrace] [./source/server/backtrace.h:98] #18: [0x55be988559b4]
envoy-proxy_1  | [2021-12-20 22:12:19.375][14][critical][backtrace] [./source/server/backtrace.h:98] #19: [0x55be9885515c]
envoy-proxy_1  | [2021-12-20 22:12:19.375][14][critical][backtrace] [./source/server/backtrace.h:98] #20: [0x55be98855bf5]
envoy-proxy_1  | [2021-12-20 22:12:19.376][14][critical][backtrace] [./source/server/backtrace.h:98] #21: [0x55be9881193c]
envoy-proxy_1  | [2021-12-20 22:12:19.376][14][critical][backtrace] [./source/server/backtrace.h:98] #22: [0x55be98af2cef]
envoy-proxy_1  | [2021-12-20 22:12:19.376][14][critical][backtrace] [./source/server/backtrace.h:98] #23: [0x55be98aebbea]
envoy-proxy_1  | [2021-12-20 22:12:19.376][14][critical][backtrace] [./source/server/backtrace.h:98] #24: [0x55be98ae9719]
envoy-proxy_1  | [2021-12-20 22:12:19.376][14][critical][backtrace] [./source/server/backtrace.h:98] #25: [0x55be98adf2b1]
envoy-proxy_1  | [2021-12-20 22:12:19.376][14][critical][backtrace] [./source/server/backtrace.h:98] #26: [0x55be98ae07bc]
envoy-proxy_1  | [2021-12-20 22:12:19.376][14][critical][backtrace] [./source/server/backtrace.h:98] #27: [0x55be98befa10]
envoy-proxy_1  | [2021-12-20 22:12:19.377][14][critical][backtrace] [./source/server/backtrace.h:98] #28: [0x55be98bee421]
envoy-proxy_1  | [2021-12-20 22:12:19.377][14][critical][backtrace] [./source/server/backtrace.h:98] #29: [0x55be98480b02]
envoy-proxy_1  | [2021-12-20 22:12:19.377][14][critical][backtrace] [./source/server/backtrace.h:98] #30: [0x55be98dca6f3]
envoy-proxy_1  | [2021-12-20 22:12:19.377][14][critical][backtrace] [./source/server/backtrace.h:96] #31: start_thread [0x7f042a79f6db]
envoy-proxy_1  | [2021-12-20 22:12:19.377][14][critical][backtrace] [./source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x1
envoy-proxy_1  | [2021-12-20 22:12:19.377][14][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
envoy-proxy_1  | [2021-12-20 22:12:19.377][14][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 2fe62422af30091ededf216a64db158ed073f3f3/1.21.0-dev/Clean/RELEASE/BoringSSL
envoy-proxy_1  | [2021-12-20 22:12:19.377][14][critical][backtrace] [./source/server/backtrace.h:96] #0: __restore_rt [0x7f042a7aa980]
envoy-proxy_1  | [2021-12-20 22:12:19.377][14][critical][backtrace] [./source/server/backtrace.h:98] #1: [0x55be98d31c99]
envoy-proxy_1  | [2021-12-20 22:12:19.377][14][critical][backtrace] [./source/server/backtrace.h:98] #2: [0x55be991de1f3]
envoy-proxy_1  | [2021-12-20 22:12:19.377][14][critical][backtrace] [./source/server/backtrace.h:98] #3: [0x55be9895099d]
envoy-proxy_1  | [2021-12-20 22:12:19.377][14][critical][backtrace] [./source/server/backtrace.h:98] #4: [0x55be98951a17]
envoy-proxy_1  | [2021-12-20 22:12:19.377][14][critical][backtrace] [./source/server/backtrace.h:98] #5: [0x55be988dcb5a]
envoy-proxy_1  | [2021-12-20 22:12:19.377][14][critical][backtrace] [./source/server/backtrace.h:98] #6: [0x55be984f203b]
envoy-proxy_1  | [2021-12-20 22:12:19.378][14][critical][backtrace] [./source/server/backtrace.h:98] #7: [0x55be984f2998]
envoy-proxy_1  | [2021-12-20 22:12:19.378][14][critical][backtrace] [./source/server/backtrace.h:98] #8: [0x55be9763eb73]
envoy-proxy_1  | [2021-12-20 22:12:19.378][14][critical][backtrace] [./source/server/backtrace.h:98] #9: [0x55be97416be8]
envoy-proxy_1  | [2021-12-20 22:12:19.378][14][critical][backtrace] [./source/server/backtrace.h:98] #10: [0x55be987f9dad]
envoy-proxy_1  | [2021-12-20 22:12:19.378][14][critical][backtrace] [./source/server/backtrace.h:98] #11: [0x55be9882bdde]
envoy-proxy_1  | [2021-12-20 22:12:19.378][14][critical][backtrace] [./source/server/backtrace.h:98] #12: [0x55be98815ac4]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #13: [0x55be98859ef3]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #14: [0x55be98857587]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #15: [0x55be9886053f]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #16: [0x55be98be3e73]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #17: [0x55be9885feef]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #18: [0x55be988559b4]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #19: [0x55be9885515c]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #20: [0x55be98855bf5]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #21: [0x55be9881193c]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #22: [0x55be98af2cef]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #23: [0x55be98aebbea]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #24: [0x55be98ae9719]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #25: [0x55be98adf2b1]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #26: [0x55be98ae07bc]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #27: [0x55be98befa10]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #28: [0x55be98bee421]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #29: [0x55be98480b02]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:98] #30: [0x55be98dca6f3]
envoy-proxy_1  | [2021-12-20 22:12:19.379][14][critical][backtrace] [./source/server/backtrace.h:96] #31: start_thread [0x7f042a79f6db]
envoy-proxy_1  | ActiveStream 0xeb13fa7c100, stream_id_: 2543176068307093099&filter_manager_:
envoy-proxy_1  |   FilterManager 0xeb13fa7c178, state_.has_1xx_headers_: 0
envoy-proxy_1  |   filter_manager_callbacks_.requestHeaders():
envoy-proxy_1  |     ':authority', '0.0.0.0:8888'
envoy-proxy_1  |     ':path', '/twoheader'
envoy-proxy_1  |     ':method', 'GET'
envoy-proxy_1  |     ':scheme', 'http'
envoy-proxy_1  |     'user-agent', 'curl/7.64.1'
envoy-proxy_1  |     'accept', '*/*'
envoy-proxy_1  |     'foo', 'a'
envoy-proxy_1  |     'bar', 'b'
envoy-proxy_1  |     'x-forwarded-proto', 'http'
envoy-proxy_1  |     'x-request-id', '79804002-7b1d-4bd0-9367-c007c772a111'
envoy-proxy_1  |   filter_manager_callbacks_.requestTrailers():   null
envoy-proxy_1  |   filter_manager_callbacks_.responseHeaders():   null
envoy-proxy_1  |   filter_manager_callbacks_.responseTrailers():   null
envoy-proxy_1  |   &stream_info_:
envoy-proxy_1  |     StreamInfoImpl 0xeb13fa7c278, protocol_: 1, response_code_: null, response_code_details_: null, attempt_count_: null, health_check_request_: 0, route_name_:     upstream_info_:     null
envoy-proxy_1  |     OverridableRemoteConnectionInfoSetterStreamInfo 0xeb13fa7c278, remoteAddress(): 172.19.0.1:60654, directRemoteAddress(): 172.19.0.1:60654, localAddress(): 172.19.0.5:8888
envoy-proxy_1  | Http1::ConnectionImpl 0xeb13fa7bb88, dispatching_: 1, dispatching_slice_already_drained_: 0, reset_stream_called_: 0, handling_upgrade_: 0, deferred_end_stream_headers_: 1, require_strict_1xx_and_204_headers_: 1, send_strict_1xx_and_204_headers_: 1, processing_trailers_: 0, no_chunked_encoding_header_for_304_: 1, buffered_body_.length(): 0, header_parsing_state_: Done, current_header_field_: , current_header_value_:
envoy-proxy_1  | active_request_:
envoy-proxy_1  | , request_url_: null, response_encoder_.local_end_stream_: 0
envoy-proxy_1  | absl::get<RequestHeaderMapPtr>(headers_or_trailers_): null
envoy-proxy_1  | current_dispatching_buffer_ front_slice length: 101 contents: "GET /twoheader HTTP/1.1\r\nHost: 0.0.0.0:8888\r\nUser-Agent: curl/7.64.1\r\nAccept: */*\r\nfoo: a\r\nbar: b\r\n\r\n"
envoy-proxy_1  | ConnectionImpl 0xeb13fc52fd0, connecting_: 0, bind_error_: 0, state(): Open, read_buffer_limit_: 1048576
envoy-proxy_1  | socket_:
envoy-proxy_1  |   ListenSocketImpl 0xeb13f63f100, transport_protocol_: raw_buffer
envoy-proxy_1  |   connection_info_provider_:
envoy-proxy_1  |     ConnectionInfoSetterImpl 0xeb13f65c908, remote_address_: 172.19.0.1:60654, direct_remote_address_: 172.19.0.1:60654, local_address_: 172.19.0.5:8888, server_name_:

Note: If there are privacy concerns, sanitize the data prior to sharing.

Call Stack:

If the Envoy binary is crashing, a call stack is required. Please refer to the Bazel Stack trace documentation.

petedmarsh commented 2 years ago

I believe the actual issue is that %ROUTE_NAME% was never supported and I mixed up access log command operators and custom header command operators.