How to disable Stream timeout

[dasois]

I have open streams from server to client, which shall be idle for a long time. Currently they get terminated exactly after 10 min:

• HTTP 200
• grpc-message: upstream connect error or disconnect/reset before headers. reset reason: connection termination
• grpc-status: 14

I found out that there is a GRPC_ARG_KEEPALIVE_PERMIT_WITHOUT_CALLS, but I did not manage to set this on the client. How can I do this?

[dasois]

I did try the solution suggested in https://github.com/grpc/grpc-web/issues/361#issuecomment-437384471 without success.

[z0mb1ek]

may be its envoy timeout?

[dasois]

in envoy config i have: max_grpc_timeout: 0s and stream_idle_timeout: 0s.

What else would i need?

[kwomackcodes]

Having the same issue. Have you found a solution? @dasois

[dasois]

No, apparently there is no way to do this with grpc-web.

Am 25. Juli 2019 16:45:15 MESZ schrieb KevinWomack0318 notifications@github.com:

Having the same issue. Have you found a solution? @dasois

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/grpc/grpc-web/issues/557#issuecomment-515073486

-- Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

[kwomackcodes]

No, apparently there is no way to do this with grpc-web. Am 25. Juli 2019 16:45:15 MESZ schrieb KevinWomack0318 notifications@github.com: … Having the same issue. Have you found a solution? @dasois -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #557 (comment) -- Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

Yeah I figured...something tells me (like the intuition above) that this might be an issue with Envoy as a proxy. Some else seemed to have luck here in this (still open) issue when they switched proxies. I hope they do something about this soon.

[doublerr]

FWIW, I was having a similar issue but my timeouts were exactly 1 minute. I thought it was the grpcwebproxy so I switched to envoy which had the same issue but gave me better logging. It was then clear that my issue was a deadline being hit which was betting set be the client. Envoy and the server pay attention to these deadlines. My challenge was that the error manifested differently if data had been streamed (ERR_INCOMPLETE_CHUNKED_ENCODING 200 (OK)) then if the stream was connected but no data sent (status messages). Simply upping the deadline set by the client and handling deadline timeout on the client solved my issue.

[ansraliant]

Hi @doublerr, I came across your comment as I am experiencing the same issue with envoy proxy. You said your solution was to upping the deadline set by the client and handling deadline timeout on the client could you explain a bit more how you can achieve that? Thank you.

[doublerr]

@ansraliant - Basic example (likely a bit dated now):

var deadline = new Date();
deadline.setSeconds(deadline.getSeconds() + 900); //deadline set for 15min which envoy will respect
var stream = this.client.streamArticles(request, {
  deadline: deadline.getTime()
});

For handling the actual timeout, look into the setTimeout function to create a timer and then handle it in stream.on("err", ...

stream.on("err", function(err) {
        console.log("err: ", err);
        clearTimeout(timer);
});

[kkweon]

I'm also getting 1 minute timeout but the above did not work. In fact, setting deadline to 0 or not setting is the same as setting the maximum deadline according to the spec.

In my case, it was nginx ingress controller. nginx ingress has the default proxy_read_timeout: 60s fixing this has resolved the issue for me.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: chat-server-ingress
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: 'nginx'
    nginx.ingress.kubernetes.io/proxy-read-timeout: "86400"
    nginx.ingress.kubernetes.io/proxy-write-timeout: "86400"
    nginx.ingress.kubernetes.io/rewrite-target: /$1

spec:
  tls:
    - hosts:
        - k8s.kkweon.dev
      secretName: kkweon-tls
  rules:
    - host: k8s.kkweon.dev
      http:
        paths:
          - path: /grpc/?(.*)
            backend:
              serviceName: server
              servicePort: 8080

[emil14]

@doublerr is there any way not to use setTimeout on every call?

As far as I see we'll get our callback triggered with error when deadline will be reached. So the problem is to distinguish timeout error from any other error?

If I don't care - is there timeout error or something else, I can not use setTimeout, am I right? :)

[GeorgeTsagk]

We had the same issue, while communicating with grpc-web to our backend server through an envoy proxy. The following rule (stream_idle_timeout: 0s) seemed to solve the problem. I will post a small part of the yaml configuration in case your rule was being placed at a different scope, possibly not having an effect.

    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: auto
          stat_prefix: ingress_http
          stream_idle_timeout: 0s

Rule max_grpc_timeout: 0s is placed at another scope, specifically under the matching route for our service.

Regarding the grpc-web side, you can try either setting a timeout of 0 or a huge timeout of 1 year for example. Both are expected to work (1st one is the "clean" solution).

(cc @arvchristos)

[girafferiel]

We had the same issue, while communicating with grpc-web to our backend server through an envoy proxy. The following rule (stream_idle_timeout: 0s) seemed to solve the problem. I will post a small part of the yaml configuration in case your rule was being placed at a different scope, possibly not having an effect.

    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: auto
          stat_prefix: ingress_http
          stream_idle_timeout: 0s

Rule max_grpc_timeout: 0s is placed at another scope, specifically under the matching route for our service.

Regarding the grpc-web side, you can try either setting a timeout of 0 or a huge timeout of 1 year for example. Both are expected to work (1st one is the "clean" solution).

(cc @arvchristos)

which version is ur envoy? mine is v15 and it doesnt work

[doctorpangloss]

For others, this issue manifests itself with errors about base64 decoding.

FWIW, I was having a similar issue but my timeouts were exactly 1 minute. I thought it was the grpcwebproxy so I switched to envoy which had the same issue but gave me better logging. It was then clear that my issue was a deadline being hit which was betting set be the client. Envoy and the server pay attention to these deadlines. My challenge was that the error manifested differently if data had been streamed (ERR_INCOMPLETE_CHUNKED_ENCODING 200 (OK)) then if the stream was connected but no data sent (status messages). Simply upping the deadline set by the client and handling deadline timeout on the client solved my issue.

grpcwebproxy has --server_http_max_write_timeout and --server_http_max_read_timeout args.

[jalr4ever]

🥲 Does anyone know about disabling idle bidi stream timeout in Nginx reverse proxy?

For now: grpc_read_timeout、grpc_send_timeout``client_header_timeout、client_body_timeout、grpc_socket_keepalive is not work for it.

I have used tcpdump for this issue, and I found that Nginx send FIN packet to the server in a time, it's about 6m30s every time.

[mikibella]

@Jalr4ever Did you find a solution for this problem?

[henriSedjame]

I found a solution by setting max_grpc_timeout to 0s.

virtual_hosts:

• name: local_service domains: ["*"] routes:
  • match: { prefix: "/" } route: cluster: tracer_server max_grpc_timeout: 0s

[raphaelzulliger]

In my case, it was grpcwebproxy's --server_http_max_write_timeout that did the trick, as suggested by @doctorpangloss.

E.g. just provide --server_http_max_write_timeout=900s. it seems that --server_http_max_write_timeout=0 completely disables the timeout. At least, my streaming connection is up for quite a while now. Without settings this option, my connection was dropped after exactly 10.0 seconds.

My situation

service xyzEvents {
  rpc OnChanged(OnChangedRequest) returns (stream OnChangedEvent);
}

[Angular] ---> [Angular devServer.proxy] ---> [grpcwebproxy] ---> [C++ backend (gRPC server)]

(it's just a POC, that's my excuse for using Angular's devServer.proxy)

[minoic]

in envoy config, timeout and idle_timeout may help:

            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match: { prefix: "/" }
                route: 
                 cluster: echo_service 
                 timeout: 0s
                 idle_timeout: 8s

https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto#envoy-v3-api-field-config-route-v3-routeaction-timeout

envoy would terminate my stream connection after 15s timeout in default without timeout: 0s option.