search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
24.98k stars 4.81k forks source link

grpc-bridge should set TE header #4342

Open ivasonn opened 6 years ago

ivasonn commented 6 years ago

After replacing go service with python in grpc-bridge example. I'm getting follow in headers with empty response

{'content-length': '0', 'grpc-message': 'upstream connect error or disconnect/reset before headers', 'server': 'envoy', 'x-envoy-upstream-service-time': '1', 'grpc-status': '14', 'date': 'Mon, 03 Sep 2018 18:48:08 GMT', 'content-type': 'application/grpc'}

repo - https://github.com/ivasonn/grpc-bridge-python

Admin and Stats Output:

cluster.local_service_grpc.bind_errors: 0
cluster.local_service_grpc.http2.header_overflow: 0
cluster.local_service_grpc.http2.headers_cb_no_stream: 0
cluster.local_service_grpc.http2.rx_messaging_error: 0
cluster.local_service_grpc.http2.rx_reset: 1
cluster.local_service_grpc.http2.too_many_header_frames: 0
cluster.local_service_grpc.http2.trailers: 0
cluster.local_service_grpc.http2.tx_reset: 0
cluster.local_service_grpc.internal.upstream_rq_503: 1
cluster.local_service_grpc.internal.upstream_rq_5xx: 1
cluster.local_service_grpc.internal.upstream_rq_completed: 1
cluster.local_service_grpc.lb_healthy_panic: 0
cluster.local_service_grpc.lb_local_cluster_not_ok: 0
cluster.local_service_grpc.lb_recalculate_zone_structures: 0
cluster.local_service_grpc.lb_subsets_active: 0
cluster.local_service_grpc.lb_subsets_created: 0
cluster.local_service_grpc.lb_subsets_fallback: 0
cluster.local_service_grpc.lb_subsets_removed: 0
cluster.local_service_grpc.lb_subsets_selected: 0
cluster.local_service_grpc.lb_zone_cluster_too_small: 0
cluster.local_service_grpc.lb_zone_no_capacity_left: 0
cluster.local_service_grpc.lb_zone_number_differs: 0
cluster.local_service_grpc.lb_zone_routing_all_directly: 0
cluster.local_service_grpc.lb_zone_routing_cross_zone: 0
cluster.local_service_grpc.lb_zone_routing_sampled: 0
cluster.local_service_grpc.max_host_weight: 0
cluster.local_service_grpc.membership_change: 1
cluster.local_service_grpc.membership_healthy: 1
cluster.local_service_grpc.membership_total: 1
cluster.local_service_grpc.original_dst_host_invalid: 0
cluster.local_service_grpc.retry_or_shadow_abandoned: 0
cluster.local_service_grpc.update_attempt: 0
cluster.local_service_grpc.update_empty: 0
cluster.local_service_grpc.update_failure: 0
cluster.local_service_grpc.update_no_rebuild: 0
cluster.local_service_grpc.update_success: 0
cluster.local_service_grpc.upstream_cx_active: 1
cluster.local_service_grpc.upstream_cx_close_notify: 0
cluster.local_service_grpc.upstream_cx_connect_attempts_exceeded: 0
cluster.local_service_grpc.upstream_cx_connect_fail: 0
cluster.local_service_grpc.upstream_cx_connect_timeout: 0
cluster.local_service_grpc.upstream_cx_destroy: 0
cluster.local_service_grpc.upstream_cx_destroy_local: 0
cluster.local_service_grpc.upstream_cx_destroy_local_with_active_rq: 0
cluster.local_service_grpc.upstream_cx_destroy_remote: 0
cluster.local_service_grpc.upstream_cx_destroy_remote_with_active_rq: 0
cluster.local_service_grpc.upstream_cx_destroy_with_active_rq: 0
cluster.local_service_grpc.upstream_cx_http1_total: 0
cluster.local_service_grpc.upstream_cx_http2_total: 1
cluster.local_service_grpc.upstream_cx_idle_timeout: 0
cluster.local_service_grpc.upstream_cx_max_requests: 0
cluster.local_service_grpc.upstream_cx_none_healthy: 0
cluster.local_service_grpc.upstream_cx_overflow: 0
cluster.local_service_grpc.upstream_cx_protocol_error: 0
cluster.local_service_grpc.upstream_cx_rx_bytes_buffered: 140
cluster.local_service_grpc.upstream_cx_rx_bytes_total: 203
cluster.local_service_grpc.upstream_cx_total: 1
cluster.local_service_grpc.upstream_cx_tx_bytes_buffered: 0
cluster.local_service_grpc.upstream_cx_tx_bytes_total: 333
cluster.local_service_grpc.upstream_flow_control_backed_up_total: 0
cluster.local_service_grpc.upstream_flow_control_drained_total: 0
cluster.local_service_grpc.upstream_flow_control_paused_reading_total: 0
cluster.local_service_grpc.upstream_flow_control_resumed_reading_total: 0
cluster.local_service_grpc.upstream_rq_503: 1
cluster.local_service_grpc.upstream_rq_5xx: 1
cluster.local_service_grpc.upstream_rq_active: 0
cluster.local_service_grpc.upstream_rq_cancelled: 0
cluster.local_service_grpc.upstream_rq_completed: 1
cluster.local_service_grpc.upstream_rq_maintenance_mode: 0
cluster.local_service_grpc.upstream_rq_pending_active: 0
cluster.local_service_grpc.upstream_rq_pending_failure_eject: 0
cluster.local_service_grpc.upstream_rq_pending_overflow: 0
cluster.local_service_grpc.upstream_rq_pending_total: 0
cluster.local_service_grpc.upstream_rq_per_try_timeout: 0
cluster.local_service_grpc.upstream_rq_retry: 0
cluster.local_service_grpc.upstream_rq_retry_overflow: 0
cluster.local_service_grpc.upstream_rq_retry_success: 0
cluster.local_service_grpc.upstream_rq_rx_reset: 1
cluster.local_service_grpc.upstream_rq_timeout: 0
cluster.local_service_grpc.upstream_rq_total: 1
cluster.local_service_grpc.upstream_rq_tx_reset: 0
cluster.local_service_grpc.version: 0
cluster_manager.active_clusters: 1
cluster_manager.cluster_added: 1
cluster_manager.cluster_modified: 0
cluster_manager.cluster_removed: 0
cluster_manager.cluster_updated: 0
cluster_manager.cluster_updated_via_merge: 0
cluster_manager.update_merge_cancelled: 0
cluster_manager.update_out_of_merge_window: 0
cluster_manager.warming_clusters: 0
filesystem.flushed_by_timer: 0
filesystem.reopen_failed: 0
filesystem.write_buffered: 0
filesystem.write_completed: 0
filesystem.write_total_buffered: 0
http.admin.downstream_cx_active: 1
http.admin.downstream_cx_destroy: 0
http.admin.downstream_cx_destroy_active_rq: 0
http.admin.downstream_cx_destroy_local: 0
http.admin.downstream_cx_destroy_local_active_rq: 0
http.admin.downstream_cx_destroy_remote: 0
http.admin.downstream_cx_destroy_remote_active_rq: 0
http.admin.downstream_cx_drain_close: 0
http.admin.downstream_cx_http1_active: 1
http.admin.downstream_cx_http1_total: 1
http.admin.downstream_cx_http2_active: 0
http.admin.downstream_cx_http2_total: 0
http.admin.downstream_cx_idle_timeout: 0
http.admin.downstream_cx_protocol_error: 0
http.admin.downstream_cx_rx_bytes_buffered: 451
http.admin.downstream_cx_rx_bytes_total: 451
http.admin.downstream_cx_ssl_active: 0
http.admin.downstream_cx_ssl_total: 0
http.admin.downstream_cx_total: 1
http.admin.downstream_cx_tx_bytes_buffered: 0
http.admin.downstream_cx_tx_bytes_total: 0
http.admin.downstream_cx_websocket_active: 0
http.admin.downstream_cx_websocket_total: 0
http.admin.downstream_flow_control_paused_reading_total: 0
http.admin.downstream_flow_control_resumed_reading_total: 0
http.admin.downstream_rq_1xx: 0
http.admin.downstream_rq_2xx: 0
http.admin.downstream_rq_3xx: 0
http.admin.downstream_rq_4xx: 0
http.admin.downstream_rq_5xx: 0
http.admin.downstream_rq_active: 1
http.admin.downstream_rq_completed: 0
http.admin.downstream_rq_http1_total: 1
http.admin.downstream_rq_http2_total: 0
http.admin.downstream_rq_idle_timeout: 0
http.admin.downstream_rq_non_relative_path: 0
http.admin.downstream_rq_response_before_rq_complete: 0
http.admin.downstream_rq_rx_reset: 0
http.admin.downstream_rq_too_large: 0
http.admin.downstream_rq_total: 1
http.admin.downstream_rq_tx_reset: 0
http.admin.downstream_rq_ws_on_non_ws_route: 0
http.admin.rs_too_large: 0
http.async-client.no_cluster: 0
http.async-client.no_route: 0
http.async-client.rq_direct_response: 0
http.async-client.rq_redirect: 0
http.async-client.rq_total: 0
http.ingress_http.downstream_cx_active: 1
http.ingress_http.downstream_cx_destroy: 0
http.ingress_http.downstream_cx_destroy_active_rq: 0
http.ingress_http.downstream_cx_destroy_local: 0
http.ingress_http.downstream_cx_destroy_local_active_rq: 0
http.ingress_http.downstream_cx_destroy_remote: 0
http.ingress_http.downstream_cx_destroy_remote_active_rq: 0
http.ingress_http.downstream_cx_drain_close: 0
http.ingress_http.downstream_cx_http1_active: 1
http.ingress_http.downstream_cx_http1_total: 1
http.ingress_http.downstream_cx_http2_active: 0
http.ingress_http.downstream_cx_http2_total: 0
http.ingress_http.downstream_cx_idle_timeout: 0
http.ingress_http.downstream_cx_protocol_error: 0
http.ingress_http.downstream_cx_rx_bytes_buffered: 390
http.ingress_http.downstream_cx_rx_bytes_total: 390
http.ingress_http.downstream_cx_ssl_active: 0
http.ingress_http.downstream_cx_ssl_total: 0
http.ingress_http.downstream_cx_total: 1
http.ingress_http.downstream_cx_tx_bytes_buffered: 0
http.ingress_http.downstream_cx_tx_bytes_total: 212
http.ingress_http.downstream_cx_websocket_active: 0
http.ingress_http.downstream_cx_websocket_total: 0
http.ingress_http.downstream_flow_control_paused_reading_total: 0
http.ingress_http.downstream_flow_control_resumed_reading_total: 0
http.ingress_http.downstream_rq_1xx: 0
http.ingress_http.downstream_rq_2xx: 1
http.ingress_http.downstream_rq_3xx: 0
http.ingress_http.downstream_rq_4xx: 0
http.ingress_http.downstream_rq_5xx: 0
http.ingress_http.downstream_rq_active: 0
http.ingress_http.downstream_rq_completed: 1
http.ingress_http.downstream_rq_http1_total: 1
http.ingress_http.downstream_rq_http2_total: 0
http.ingress_http.downstream_rq_idle_timeout: 0
http.ingress_http.downstream_rq_non_relative_path: 0
http.ingress_http.downstream_rq_response_before_rq_complete: 0
http.ingress_http.downstream_rq_rx_reset: 0
http.ingress_http.downstream_rq_too_large: 0
http.ingress_http.downstream_rq_total: 1
http.ingress_http.downstream_rq_tx_reset: 0
http.ingress_http.downstream_rq_ws_on_non_ws_route: 0
http.ingress_http.no_cluster: 0
http.ingress_http.no_route: 0
http.ingress_http.rq_direct_response: 0
http.ingress_http.rq_redirect: 0
http.ingress_http.rq_total: 1
http.ingress_http.rs_too_large: 0
http.ingress_http.tracing.client_enabled: 0
http.ingress_http.tracing.health_check: 0
http.ingress_http.tracing.not_traceable: 0
http.ingress_http.tracing.random_sampling: 0
http.ingress_http.tracing.service_forced: 0
listener.0.0.0.0_9211.downstream_cx_active: 1
listener.0.0.0.0_9211.downstream_cx_destroy: 0
listener.0.0.0.0_9211.downstream_cx_total: 1
listener.0.0.0.0_9211.http.ingress_http.downstream_rq_1xx: 0
listener.0.0.0.0_9211.http.ingress_http.downstream_rq_2xx: 1
listener.0.0.0.0_9211.http.ingress_http.downstream_rq_3xx: 0
listener.0.0.0.0_9211.http.ingress_http.downstream_rq_4xx: 0
listener.0.0.0.0_9211.http.ingress_http.downstream_rq_5xx: 0
listener.0.0.0.0_9211.http.ingress_http.downstream_rq_completed: 1
listener.0.0.0.0_9211.no_filter_chain_match: 0
listener.admin.downstream_cx_active: 1
listener.admin.downstream_cx_destroy: 0
listener.admin.downstream_cx_total: 1
listener.admin.http.admin.downstream_rq_1xx: 0
listener.admin.http.admin.downstream_rq_2xx: 0
listener.admin.http.admin.downstream_rq_3xx: 0
listener.admin.http.admin.downstream_rq_4xx: 0
listener.admin.http.admin.downstream_rq_5xx: 0
listener.admin.http.admin.downstream_rq_completed: 0
listener.admin.no_filter_chain_match: 0
listener_manager.listener_added: 1
listener_manager.listener_create_failure: 0
listener_manager.listener_create_success: 2
listener_manager.listener_modified: 0
listener_manager.listener_removed: 0
listener_manager.total_listeners_active: 1
listener_manager.total_listeners_draining: 0
listener_manager.total_listeners_warming: 0
runtime.admin_overrides_active: 0
runtime.load_error: 0
runtime.load_success: 0
runtime.num_keys: 0
runtime.override_dir_exists: 0
runtime.override_dir_not_exists: 0
server.concurrency: 2
server.days_until_first_cert_expiring: 2147483647
server.hot_restart_epoch: 0
server.live: 1
server.memory_allocated: 3360032
server.memory_heap_size: 4194304
server.parent_connections: 0
server.total_connections: 1
server.uptime: 15
server.version: 6124312
server.watchdog_mega_miss: 0
server.watchdog_miss: 0
stats.overflow: 0
cluster.local_service_grpc.upstream_cx_connect_ms: P0(nan,2) P25(nan,2.025) P50(nan,2.05) P75(nan,2.075) P90(nan,2.09) P95(nan,2.095) P99(nan,2.099) P99.5(nan,2.0995) P99.9(nan,2.0999) P100(nan,2.1)
cluster.local_service_grpc.upstream_cx_length_ms: No recorded values
http.admin.downstream_cx_length_ms: No recorded values
http.admin.downstream_rq_time: No recorded values
http.ingress_http.downstream_cx_length_ms: No recorded values
http.ingress_http.downstream_rq_time: P0(nan,8) P25(nan,8.025) P50(nan,8.05) P75(nan,8.075) P90(nan,8.09) P95(nan,8.095) P99(nan,8.099) P99.5(nan,8.0995) P99.9(nan,8.0999) P100(nan,8.1)
listener.0.0.0.0_9211.downstream_cx_length_ms: No recorded values
listener.admin.downstream_cx_length_ms: No recorded
local_service_grpc::default_priority::max_connections::1024
local_service_grpc::default_priority::max_pending_requests::1024
local_service_grpc::default_priority::max_requests::1024
local_service_grpc::default_priority::max_retries::3
local_service_grpc::high_priority::max_connections::1024
local_service_grpc::high_priority::max_pending_requests::1024
local_service_grpc::high_priority::max_requests::1024
local_service_grpc::high_priority::max_retries::3
local_service_grpc::added_via_api::false
local_service_grpc::127.0.0.1:8081::cx_active::1
local_service_grpc::127.0.0.1:8081::cx_connect_fail::0
local_service_grpc::127.0.0.1:8081::cx_total::1
local_service_grpc::127.0.0.1:8081::rq_active::0
local_service_grpc::127.0.0.1:8081::rq_error::1
local_service_grpc::127.0.0.1:8081::rq_success::0
local_service_grpc::127.0.0.1:8081::rq_timeout::0
local_service_grpc::127.0.0.1:8081::rq_total::1
local_service_grpc::127.0.0.1:8081::health_flags::healthy
local_service_grpc::127.0.0.1:8081::weight::1
local_service_grpc::127.0.0.1:8081::region::
local_service_grpc::127.0.0.1:8081::zone::
local_service_grpc::127.0.0.1:8081::sub_zone::
local_service_grpc::127.0.0.1:8081::canary::false
local_service_grpc::127.0.0.1:8081::success_rate::-1
envoy 5d731878fd0134ca15d5904450a64dab0ff577a9/1.8.0-dev/Clean/RELEASE live 352 352 0

Config:

{
 "configs": [
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.BootstrapConfigDump",
   "bootstrap": {
    "node": {
     "build_version": "5d731878fd0134ca15d5904450a64dab0ff577a9/1.8.0-dev/Clean/RELEASE"
    },
    "static_resources": {
     "listeners": [
      {
       "address": {
        "socket_address": {
         "address": "0.0.0.0",
         "port_value": 9211
        }
       },
       "filter_chains": [
        {
         "filters": [
          {
           "name": "envoy.http_connection_manager",
           "config": {
            "route_config": {
             "virtual_hosts": [
              {
               "routes": [
                {
                 "route": {
                  "cluster": "local_service_grpc"
                 },
                 "match": {
                  "headers": [
                   {
                    "name": "content-type",
                    "exact_match": "application/grpc"
                   }
                  ],
                  "prefix": "/"
                 }
                }
               ],
               "domains": [
                "*"
               ],
               "name": "local_service"
              }
             ],
             "name": "local_route"
            },
            "codec_type": "auto",
            "http_filters": [
             {
              "name": "envoy.router",
              "config": {}
             }
            ],
            "stat_prefix": "ingress_http"
           }
          }
         ]
        }
       ]
      }
     ],
     "clusters": [
      {
       "name": "local_service_grpc",
       "connect_timeout": "0.250s",
       "hosts": [
        {
         "socket_address": {
          "address": "127.0.0.1",
          "port_value": 8081
         }
        }
       ],
       "http2_protocol_options": {}
      }
     ]
    },
    "admin": {
     "access_log_path": "/var/log/envoy/admin_access.log",
     "address": {
      "socket_address": {
       "address": "0.0.0.0",
       "port_value": 9901
      }
     }
    }
   },
   "last_updated": "2018-09-05T07:13:58.283Z"
  },
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.ClustersConfigDump",
   "static_clusters": [
    {
     "cluster": {
      "name": "local_service_grpc",
      "connect_timeout": "0.250s",
      "hosts": [
       {
        "socket_address": {
         "address": "127.0.0.1",
         "port_value": 8081
        }
       }
      ],
      "http2_protocol_options": {}
     },
     "last_updated": "2018-09-05T07:13:58.287Z"
    }
   ]
  },
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.ListenersConfigDump",
   "static_listeners": [
    {
     "listener": {
      "address": {
       "socket_address": {
        "address": "0.0.0.0",
        "port_value": 9211
       }
      },
      "filter_chains": [
       {
        "filters": [
         {
          "name": "envoy.http_connection_manager",
          "config": {
           "http_filters": [
            {
             "name": "envoy.router",
             "config": {}
            }
           ],
           "stat_prefix": "ingress_http",
           "route_config": {
            "name": "local_route",
            "virtual_hosts": [
             {
              "routes": [
               {
                "match": {
                 "headers": [
                  {
                   "name": "content-type",
                   "exact_match": "application/grpc"
                  }
                 ],
                 "prefix": "/"
                },
                "route": {
                 "cluster": "local_service_grpc"
                }
               }
              ],
              "domains": [
               "*"
              ],
              "name": "local_service"
             }
            ]
           },
           "codec_type": "auto"
          }
         }
        ]
       }
      ]
     },
     "last_updated": "2018-09-05T07:13:58.294Z"
    }
   ]
  },
  {
   "@type": "type.googleapis.com/envoy.admin.v2alpha.RoutesConfigDump",
   "static_route_configs": [
    {
     "route_config": {
      "name": "local_route",
      "virtual_hosts": [
       {
        "name": "local_service",
        "domains": [
         "*"
        ],
        "routes": [
         {
          "match": {
           "prefix": "/",
           "headers": [
            {
             "name": "content-type",
             "exact_match": "application/grpc"
            }
           ]
          },
          "route": {
           "cluster": "local_service_grpc"
          }
         }
        ]
       }
      ]
     },
     "last_updated": "2018-09-05T07:13:58.293Z"
    }
   ]
  }
 ]
}

Logs:

grpc_1    | I0905 07:22:11.309527924      20 call.cc:773]                  2: {"created":"@1536132131.309243212","description":"Missing :authority or :path","file":"src/core/lib/surface/server.cc","file_line":726,"referenced_errors":[{"created":"@1536132131.309241561","description":"Failed processing incoming headers","file":"src/core/ext/filters/http/server/http_server_filter.cc","file_line":87,"referenced_errors":[{"created":"@1536132131.309240138","description":"Missing header","file":"src/core/ext/filters/http/server/http_server_filter.cc","file_line":141,"key":"te"}]}]}
ivasonn commented 6 years ago

Adding TE header for request solved this problem

lizan commented 6 years ago

@ivasonn That makes sense, thanks for investigation. gRPC requires TE header (value trailers).

I think grpc-bridge should add that header, how does this handled at Lyft? @junr03 @ccaraman @danielhochman?

mattklein123 commented 6 years ago

Hmm I guess we aren't hitting this issue for some reason, I'm not sure why.

stale[bot] commented 6 years ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

gnagel commented 5 years ago

I just bumped into this issue too ... for anyone else that hits this issue here's what I added to the http post to resolve the issue:

url = "http://{host}:{port}/kv.KV/Get".format(host=self._host, port=self._port)

response = requests.post(url, data=data, headers={
    'Host': self._host,
    'content-type': 'application/grpc',
    # Add this header to fix the issue:
    'te': 'trailers',  
    # Additional headers for controlling timeouts and retries:
    'x-envoy-retry-on': '5xx',
    'x-envoy-upstream-rq-per-try-timeout-ms': '1500',
    'x-envoy-upstream-rq-timeout-ms': '30000',
})

Mind if I make a bug fix PR for this one @mattklein123?

Thx, Glenn

mattklein123 commented 5 years ago

@gnagel sure please fix.