Closed alexmasi closed 1 year ago
After using the container with the debug fixes I am seeing most of the pods come up (306/337). Now only 31 are stuck in an Init state.
Seeing the following potential issues:
Many of these messages across all workers:
e.g.
meshnet-nq2n6 (worker-2):
time="2022-12-13T22:24:56Z" level=info msg="SendToOnce (wire id - 77): Could not find local handle. err:interface 77 is not active"
time="2022-12-13T22:24:56Z" level=error msg="finished unary call with code Unknown" error="interface 77 is not active" grpc.code=Unknown grpc.method=SendToOnce grpc.service=meshnet.v1beta1.WireProtocol grpc.start_time="2022-12-13T22:24:56Z" grpc.time_ms=0.093 peer.address="10.240.0.24:37018" span.kind=server system=grpc
coming from here: https://github.com/networkop/meshnet-cni/blob/d3ae64833f4710c0ba1810737df0bd5de414844f/daemon/grpcwire/grpcwire.go#L254
Seeing errors like the following, reading through the source code maybe this should just be info instead of error?
time="2022-12-13T22:23:34Z" level=error msg="Appending new element bx02cbf40"
Seeing a single occurrence of this failure:
meshnet-njd5r (worker-1):
time="2022-12-13T22:22:01Z" level=info msg="Error creating vEth pair (in:eth14 <--> out:bx02deth14-0003). Error-> failed to Statfs \"/proc/20948/ns/net\": no such file or directory"
time="2022-12-13T22:22:01Z" level=error msg="AddWireRemote err: failed to Statfs \"/proc/20948/ns/net\": no such file or directory"
time="2022-12-13T22:22:01Z" level=error msg="finished unary call with code Unknown" error="failed to Statfs \"/proc/20948/ns/net\": no such file or directory" grpc.code=Unknown grpc.method=AddGRPCWireRemote grpc.service=meshnet.v1beta1.Remote grpc.start_time="2022-12-13T22:22:01Z" grpc.time_ms=106.531 peer.address="10.240.0.23:35042" span.kind=server system=grpc
coming from this chain:
Another thing I observed is two reboots of meshnet pods, one due to OOM and the other was Error status. My concern is that the "missing" interfaces were stored in local memory and did not persist across the reboot: https://github.com/networkop/meshnet-cni/blob/d3ae64833f4710c0ba1810737df0bd5de414844f/daemon/grpcwire/grpcwire.go#L143
After using the container with the debug fixes I am seeing most of the pods come up (306/337). Now only 31 are stuck in an Init state.
That's good news. I can merge the PR or wait until you add more to it? let me know
Seeing errors like the following, reading through the source code maybe this should just be info instead of error?
agreed
Seeing a single occurrence of this failure:
this looks weird. is there a chance a pod got deleted while the CNI plugin was still processing it?
Another thing I observed is two reboots of meshnet pods, one due to OOM and the other was Error status. My concern is that the "missing" interfaces were stored in local memory and did not persist across the reboot:.
You're most likely right. Looks like if meshnetd is restarted it would have no prior knowledge of existing grpc wires. Should this somehow be persisted on disk or api-server? I guess the pcap.Handles still need to be properly cleaned up and re-opened on pod restart.
I added the one more change to downgrade the log from error to info. I think this should be merged as is.
I spoke with Keysight and they have a branch fix-race
on their fork that combined with memory resource limit increase (200Mi to 1000Mi) for the meshnet pod I was able to get it to work.
Hopefully they can get that merged into this repo.
I will open a separate issue as an FR to add proper reconciliation for grpc-wires (whether that be through the topology CRD or one of the methods you mention)
@kingshukdev for visibility
Closing in favor of reconciliation FR: https://github.com/networkop/meshnet-cni/issues/57
I am having some issues when using meshnet with grpc-wire.
I have a >300 node topology (running in KNE) spread across a 5 node k8 cluster (4-workers).
Many of the router pods get stuck in
Init:0/1
state and some of the meshnet pods get stuck in a crash loop:I see that the failed pod:eth (bx04r:eth36) is reverse-skipped by the peer pod on that link (bx03s:eth1):
Reverse-skipping of pod bx04rno10 by pod bx03sfo03
excerpt from KNE topology:
However I do not see an entry in the
ip link
table for bx03s* even though bx03 is stuck in Init state on a worker with a healthy meshnet pod.On another note, even the running pods report strange errors (which seem like simple no-op bugs upon inspection):
In the worker nodes with the failed meshnet pods I see many koko links waiting to be renamed:
Any help in debugging this further would be appreciated