ERR_INCOMPLETE_CHUNKED_ENCODING 200 (OK) on idle stream

[jonno317]

I have a few streams running through an envoy proxy and am seeing the same behavior mentioned in #309 .

Specifically after 5 minutes streams timeout with a POST net::ERR_INCOMPLETE_CHUNKED_ENCODING 200 (OK) error. I do have the max_grpc_timeout set in the envoy config file in the way mentioned in the documentation from #309.

I don't see any other known bugs around server side streaming. Are there potentially other settings or is maybe there's a gotcha of an idle timeout of 5 minutes even with the grpc timeout turned off (I'm fairly certain the streams are only dying after 5 minutes of inactivity)?

[SebDE]

I ran into a similar problem. I would like to use the server-stream for a listener-pattern. In detail I would like to register the client via a server-stream to a listener on a grpc-server and get continuous updates. But after some time of inactivity the stream died like @jonno317 described. Are there any options for long running streams, or better for infinitely running server-streams?

[jonno317]

@SebDE I did come across a setting on the envoy side of things that seems to be mitigating it. I haven't thoroughly tested yet, but I also haven't seen the issue again since I added it.

I believe the config from the example can be modified as follows to support long-lived streams.

listeners:
  - name: listener_0
    address:
      socket_address: { address: 0.0.0.0, port_value: 8080 }
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          stream_idle_timeout: 0s  #<-- ADD THIS
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match: { prefix: "/" }
                route:
                  cluster: echo_service
                  max_grpc_timeout: 0s
          http_filters:
          - name: envoy.grpc_web
          - name: envoy.router
  clusters:
  - name: echo_service
    connect_timeout: 0.25s
    type: logical_dns
    http2_protocol_options: {}
    lb_policy: round_robin
    hosts: [{ socket_address: { address: node-server, port_value: 9090 }}]

Hopefully someone else can chime in on whether there's a downside to that combination of settings.

[chiamtc]

mine doesn't work in such format - array in hosts cluster. Instead, the format has to be like

  clusters:
  - name: greeter_service
    connect_timeout: 0.25s
    type: logical_dns
    common_http_protocol_options:
        idle_timeout: 7200s
    http2_protocol_options: {}
    lb_policy: round_robin
    hosts:
    - socket_address:
        address: host.docker.internal
        port_value: 9090
    upstream_connection_options:
        tcp_keepalive:
          keepalive_probes: 1
          keepalive_time: 10
          keepalive_interval: 10

stream_idle_timeout: 0s works

[doublerr]

I posted on #557 but updating here as well. I had this issue and it simply came down to the deadline being set on the client when connecting the stream RPC. The challenge was that if the stream connected by no data sent, the grpc status updates were thrown. However if a single message went down the stream, then the deadline manifested itself as ERR_INCOMPLETE_CHUNKED_ENCODING 200 (OK). Upping the deadline set by the client solved my immediate problem and then I manage to reconnect the stream otherwise.

[davidmontoyago]

I'm consistently experiencing this issue with the grpcwebproxy (instead of envoy) and setting a deadline of hours doesn't seem to help.

grpc-web: 1.0.7 grpc: 1.24.2

Log entry on the proxy:

WARN[0050] finished streaming call with code Unavailable  error="rpc error: code = Unavailable desc = transport is closing" grpc.code=Unavailable grpc.method=GetEventStream grpc.request.deadline="2019-12-
14T17:06:02-06:00" grpc.service=api.EventStreamingService grpc.start_time="2019-12-13T17:06:02-06:00" grpc.time_ms=10003.504 span.kind=server system=grpc

[supermihi]

I just ran into the same issue (streams disconnecting after five minutes). Using the current version of grpcwebproxy (the wrapper, not standalone version) did fix the problem for me. Hopefully this helps anyone!

// https://rogchap.com/2019/07/26/in-process-grpc-web-proxy/
func WrapServer(grpcServer *grpc.Server) *http.Server {
    grpcWebServer := grpcweb.WrapServer(grpcServer)

    httpServer := &http.Server{
        Handler: h2c.NewHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            if r.ProtoMajor == 2 {
                grpcWebServer.ServeHTTP(w, r)
            } else {
                w.Header().Set("Access-Control-Allow-Origin", "*")
                w.Header().Set("Access-Control-Allow-Methods", "POST, GET, OPTIONS, PUT, DELETE")
                w.Header().Set("Access-Control-Allow-Headers", "Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization, X-User-Agent, X-Grpc-Web")
                w.Header().Set("grpc-status", "")
                w.Header().Set("grpc-message", "")
                if grpcWebServer.IsGrpcWebRequest(r) {
                    grpcWebServer.ServeHTTP(w, r)
                }
            }
        }), &http2.Server{}),
    }
    return httpServer
}

[supermihi]

... as does @jonno317's answer above. Can work with both proxies again - thanks for the input folks!

[yoichiro]

When using the HTTP Connection Manager v3, I faced the same issue. And, the max_grpc_timeout is deprecated. Instead, when adding the following snippet, I was able to fix this issue:

...
- filters:
    - name: envoy.filters.network.http_connection_manager
        typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
        codec_type: auto
        stat_prefix: ingress_http
        stream_idle_timeout: 0s
        route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
                domains: [ "*" ]
                routes:
                - match:
                    prefix: "/"
                    route:
                    cluster: message_service
                    max_stream_duration:
                        grpc_timeout_header_max: 0s
...