RunPodSandbox() fails due to network error.

[kunalkushwaha]

Sandbox is created successfully but RunPodSandBox() fails with below error.

I tried separately with sample code too, same error occurs. It is reproducible with latest containerd and cri-containerd

rpc error: code = 2 desc = failed to get network status for sandbox "fb58bf5ab96b7e1197616d5406eda1a5845b39adfc322f1efbd49d6e36472ef7": Unexpected command output Device "eth0" does not exist.
 with error: exit status 1

[abhi]

@kunalkushwaha which cni plugin are you using ? Can you post your CNI network config file ?

[abhi]

If you do not have the network config in the default location prior to starting the CRI service, you will need to use the Status() to ensure the networkstatus is ready and loaded. We moved to https://github.com/containerd/go-cni for the cni library. A Status() check is needed to load the cni config. There is no background watch being done. So: 1) If cni config is added prior to starting CRI , it will be loaded. 2) If cni config is added after CRI is started, It will be loaded on the next Status() check.

[Random-Liu]

@kunalkushwaha Does this resolve your problem?

[kunalkushwaha]

@abhi @Random-Liu below are my config files for CNI

$ cat /etc/cni/net.d/99-loopback.conf 
{
        "cniVersion":"0.3.1",
        "type":"loopback"
}

~/go/src/github.com/moby/buildkit/examples/buildkit4 on  cni-network-worker! ⌚ 9:25:46
$ cat /etc/cni/net.d/crio-bridge.conf 
{
    "cniVersion": "0.2.0",
    "name": "crio-bridge",
    "type": "bridge",
    "bridge": "cni0",
    "isGateway": true,
    "ipMasq": true,
    "ipam": {
        "type": "host-local",
        "subnet": "10.88.0.0/16",
        "routes": [
            { "dst": "0.0.0.0/0" }
        ]
    }
}

Also, I am using go-cni for one of my WIP PR https://github.com/moby/buildkit/pull/309 and it works fine with these CNI configuration.

Will try to debug more on this.

[Random-Liu]

@abhi I see a similar issue in containerd test:

E0327 02:43:26.722169    3440 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to setup network for sandbox "ba2e941b7dae2f7dcb17439e673a1a14147757154119c3698b5548191a78914d": failed to set bridge addr: could not add IP address to "cni0": file exists
Mar 27 02:43:26.723: INFO: Unexpected error occurred: rpc error: code = Unknown desc = failed to setup network for sandbox "ba2e941b7dae2f7dcb17439e673a1a14147757154119c3698b5548191a78914d": failed to set bridge addr: could not add IP address to "cni0": file exists

[abhi]

@kunalkushwaha may be you are running into this ? https://github.com/kubernetes/kubernetes/issues/25281

[abhi]

@Random-Liu your issue is different from @kunalkushwaha issue. This is an error from bridge plugin. Its a netlink error. I will checkout the possibilites for bridge plugin.

[kunalkushwaha]

@abhi @Random-Liu Thanks for looking into this.

This error got fixed at my setup. I guess it was due to outdated vendor code. I checkout the latest 1.1.0-rc code for containerd and build new binaries. It got fixed with that.

If @Random-Liu 's issue is fixed, you may close this issue.

[abhi]

Thanks ! Closing the issue.