Open ykorzikowski opened 1 year ago
This issue is caused by grpc and its native keepalive not being supported by revere-proxy. We are implementing a application keepalive tin #771
Just for reference: Found this: https://github.com/camunda-community-hub/zeebe-client-node-js/issues/101
@ykorzikowski, we notice that running Nginx v1.25.1 solved the issue. Can you test it?
Hellooo,
I dont know. What helped is setting grpc_read_timeout 3600s;
to 1 hour.
I am still using nginx version: nginx/1.18.0
and did not notice this issue since tweaking my config with above parameter.
Hello,
I'm trying to deploy the netbird management into my k8s cluster v.1.25.12 with the nginx ingress controller 1.19.10
Have specified the annotations in the netbird management and netbird signal ingresses:
nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
nginx.ingress.kubernetes.io/configuration-snippet: |
allow all;
grpc_read_timeout 3600s;
grpc_send_timeout 3600s;
grpc_socket_keepalive on;
Peers are registered and the netbird client connected but vpn doesn't work:
$ netbird status
Daemon version: 0.22.7
CLI version: 0.22.7
Management: Connected
Signal: Connected
FQDN: test.hidden.tech
NetBird IP: 100.77.11.161/16
Interface type: Kernel
Peers count: 0/5 Connected
sudo tail -n 30 /var/log/netbird/client.log
2023-09-27T12:08:47+03:00 WARN client/server/server.go:226: canceling previous waiting execution
2023-09-27T12:09:34+03:00 INFO client/internal/login.go:130: peer has been successfully registered on Management Service
2023-09-27T12:09:35+03:00 INFO client/internal/wgproxy/proxy_ebpf.go:79: local wg proxy listening on: 3128
2023-09-27T12:09:35+03:00 INFO iface/tun_linux.go:15: create tun interface with kernel WireGuard support: wt0
2023-09-27T12:09:50+03:00 INFO signal/client/grpc.go:157: connected to the Signal Service stream
2023-09-27T12:09:50+03:00 INFO client/internal/connect.go:179: Netbird engine started, my IP is: 100.77.11.161/16
2023-09-27T12:09:50+03:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2023-09-27T12:09:50+03:00 INFO client/internal/dns/systemd_linux.go:135: adding 1 search domains and 0 match domains. Search list: [vpn.stage.heddin.tech] , Match list: []
2023-09-27T12:09:50+03:00 INFO client/internal/acl/manager.go:67: ACL rules processed in: 1.365082ms, total rules count: 2
2023-09-27T12:10:35+03:00 WARN signal/client/grpc.go:170: disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR
2023-09-27T12:10:50+03:00 INFO signal/client/grpc.go:157: connected to the Signal Service stream
2023-09-27T12:11:35+03:00 WARN signal/client/grpc.go:170: disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR
2023-09-27T12:11:48+03:00 INFO signal/client/grpc.go:157: connected to the Signal Service stream
2023-09-27T12:12:01+03:00 INFO client/internal/dns/systemd_linux.go:135: adding 1 search domains and 0 match domains. Search list: [vpn.stage.heddin.tech] , Match list: []
2023-09-27T12:12:01+03:00 INFO client/internal/acl/manager.go:67: ACL rules processed in: 461.906µs, total rules count: 2
2023-09-27T12:12:36+03:00 WARN signal/client/grpc.go:170: disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR
2023-09-27T12:12:40+03:00 WARN signal/client/grpc.go:151: disconnected from the Signal Exchange due to an error: rpc error: code = Unavailable desc = connection error: desc = "error reading from server: read tcp 192.168.1.100:59684->hidden:443: read: connection timed out"
2023-09-27T12:12:41+03:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within `timeout`
How should I additionally configure my nginx ingress controller and\or ingress resources to solve this issue?
PS. I can't add additionaly:
grpc_pass grpc://$service_name:$service_port;
because nginx config already has:
grpc_pass grpc://$upstream_balancer;
and I get error when try:
Error: cannot patch "netbird-management" with kind Ingress: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request:
nginx: [emerg] "grpc_pass" directive is duplicate in /tmp/nginx/nginx-cfg1131325741:4731
nginx: configuration file /tmp/nginx/nginx-cfg1131325741 test failed
Any news?
Can you post some nginx access / error logs? Maybe this is some ingress configuration issue and noting regarding netbird.
Nginx ingress controllers' logs don't contain any error:
~ % stern nginx -i netbird
+ nginx-ingress-ingress-nginx-controller-ssvsd › controller
+ nginx-ingress-ingress-nginx-controller-kc929 › controller
+ nginx-ingress-ingress-nginx-controller-znnks › controller
+ nginx-ingress-ingress-nginx-controller-dsphr › controller
+ nginx-ingress-ingress-nginx-controller-qlglz › controller
+ nginx-ingress-ingress-nginx-controller-6jqht › controller
nginx-ingress-ingress-nginx-controller-qlglz controller ip-hidden - - [06/Oct/2023:07:43:22 +0000] "POST /management.ManagementService/GetServerKey HTTP/2.0" 200 61 "-" "grpc-go/1.55.0" 118 0.002 [netbird-netbird-management-80] [] 10.244.3.224:80 91 0.004 200 d93bf7b778f62315eddbce6d13e36540
nginx-ingress-ingress-nginx-controller-qlglz controller ip-hidden - - [06/Oct/2023:07:43:22 +0000] "POST /management.ManagementService/GetPKCEAuthorizationFlow HTTP/2.0" 200 0 "-" "grpc-go/1.55.0" 149 0.002 [netbird-netbird-management-80] [] 10.244.3.224:80 67 0.000 200 55a5a3f501b501a66b96cd41b7a64dec
nginx-ingress-ingress-nginx-controller-qlglz controller ip-hidden - - [06/Oct/2023:07:43:27 +0000] "POST /management.ManagementService/GetServerKey HTTP/2.0" 200 61 "-" "grpc-go/1.55.0" 118 0.002 [netbird-netbird-management-80] [] 10.244.3.224:80 91 0.004 200 2f1504ba52578c47c42aa625646c24b5
nginx-ingress-ingress-nginx-controller-qlglz controller ip-hidden - - [06/Oct/2023:07:43:27 +0000] "POST /management.ManagementService/Login HTTP/2.0" 200 0 "-" "grpc-go/1.55.0" 352 0.002 [netbird-netbird-management-80] [] 10.244.3.224:80 88 0.000 200 3739e538f1cc0de4a30661be8e30c8e8
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:43:56 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 254 0.012 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.008 200 320af0b7fa14deb408df7a9655d6894e
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:43:56 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.004 200 c9b0464002cf4a0c99090ae3e4dd965b
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:43:57 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.000 200 8ce398045d021141db56fe2d7ab0a40b
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:44:36 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.004 200 89850818ca10896a4cd86d91ca4b0111
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:44:36 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.000 200 97ec32beb45da1830ee71e7062260acf
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:44:40 +0000] "POST /signalexchange.SignalExchange/ConnectStream HTTP/2.0" 200 0 "-" "grpc-go/1.55.0" 154 60.000 [netbird-netbird-signal-80] [] 10.244.3.157:80 187 60.000 200 33cd1401fdfed21bbafd556ef35317db
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:45:06 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.006 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.008 200 5a77a24d9402d063a6a4ecc4a2c99929
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:45:15 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.007 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.004 200 1d285753adebe4f1647259ea2d60020c
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:45:15 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.000 200 9fbd16a3312440bcc847bc6f2c93be4e
Any news?
Hey there,
I am successfully running netbird since couple of months. Some clients cant speak with each other. I am investigating this issue right now and found, that I have errors regarding the signal-service in my client-logs (all clients, also the working ones):
Is there a recommendation how to run the netbird service? Like my config or everythin on port 443 like https://github.com/netbirdio/netbird/issues/536 ?
Thank you for any help :)