Support retrieve FDs

[keyingliu]

There is such case: virtlet gets exited for some reasons, and during the restart, recover is executed without the TAP or other fds, everything is fine now. But later the qemu process may quit, kubelet will try to start a new one, but vmwrapper will fail to read the fds.

Can we extend the FDSource to add new method? If GetFDs is called, that means the VM is going to start, if FDServer found the fds is nil, it can call this method to retrieve the fds.

RetrieveFDs(key string) ([]int, error)

The code snippet look like:

// Retrieve retrieve the FDs
// It is only the case if VM exited but recover didn't populate the FDs
func (s *TapFDSource) RetrieveFDs(key string) ([]int, error) {
    var podNet *podNetwork
    func() {
        s.Lock()
        defer s.Unlock()
        podNet = s.fdMap[key]
    }()
    if podNet == nil {
        return nil, fmt.Errorf("bad key (%s) to retrieve FDs", key)
    }

    netNSPath := cni.PodNetNSPath(podNet.pnd.PodID)
    vmNS, err := ns.GetNS(netNSPath)
    if err != nil {
        return nil, fmt.Errorf("failed to open network namespace at %q: %v", netNSPath, err)
    }

    if err := utils.CallInNetNSWithSysfsRemounted(vmNS, func(hostNS ns.NetNS) error {
        allLinks, err := netlink.LinkList()
        if err != nil {
            return fmt.Errorf("error listing the links: %v", err)
        }

        if err := nettools.RecoverContainerSideNetwork(podNet.csn, netNSPath, allLinks, hostNS, true); err != nil {
            return err
        }
        return nil
    }); err != nil {
        return nil, err
    }
    var fds []int
    s.Lock()
    defer s.Unlock()
    for _, ifDesc := range podNet.csn.Interfaces {
        fds = append(fds, int(ifDesc.Fo.Fd()))
    }
    return fds, nil
}

[keyingliu]

@ivan4th @jellonek any comments about this?

[jellonek]

This case should be handled by https://github.com/Mirantis/virtlet/blob/ea3fd68647ef79d82b834d864a707871096f0cd8/pkg/tapmanager/tapfdsource.go#L312 but i'll take a deeper look on that soon if you are facing an issue in this place actually.

[keyingliu]

@jellonek there is such case I met: virtlet gets exited for some reasons, and during the restart, recover is executed without the TAP or other fds, everything is fine now. But later the qemu process may quit, kubelet will try to start a new one, but vmwrapper will fail to read the fds because during the Recover the TAP's FD is not opened.

[keyingliu]

You may look: https://github.com/Mirantis/virtlet/blob/master/pkg/nettools/nettools.go#L706, it just got the ifaceType, after the flow, the qemu cannot get started: (virtlet/qemu running) -> (virtlet restarted) -> (qemu quits for some reasons) -> (the quited qemu should be restarted but it fails with error), the error is: Failed to obtain tap fds for key "xxxxx-xxx": bad oob data size: 0 instead of 16

[keyingliu]

close as PR has been merged: https://github.com/Mirantis/virtlet/pull/864