[Network] Add support for MacVTap interfaces

[Vesnica]

I'm following the getting started guide, and trying to bind a macvtap to guest, which always give me this:

2020-05-28T14:03:41.904918179 [anonymous-instance:ERROR:src/api_server/src/parsed_request.rs:90] Received Error. Status code: 400 Bad Request. Message: Cannot open TAP device. Invalid name/permissions. CreateTap(Os { code: 22, kind: InvalidInput, message: Invalid argument })

I use following command to create macvtap and try to setup network for guest:

sudo ip link add link eno1 name macvtap0 type macvtap mode bridge
curl --unix-socket /tmp/firecracker.socket -i \
         -X PUT 'http://localhost/network-interfaces/eth0' \
         -H 'Accept: application/json'           \
         -H 'Content-Type: application/json'     \
         -d "{
                \"iface_id\": \"eth0\",
                \"host_dev_name\": \"macvtap0\"
            }"

firecracker version: 0.21.1

[sandreim]

Hi @Vesnica! Firecracker does not support macvtap yet. What is your exact usecase ? Macvtap simplifies the networking setup, but otherwise you can still use tap + bridge.

[Vesnica]

@sandreim Using a bridge requires me to reconfig my network (move ip to bridge etc), using macvtap can avoid these troubles.

I'm looking forward to this feature.

[pothos]

To accepting macvtap devices you can either require them to be in the format /dev/tapXY and then opening here https://github.com/firecracker-microvm/firecracker/blob/49003d0f2f8573f6e1af6d22232c0917a0253189/src/devices/src/virtio/net/tap.rs#L113 and skipping that step https://github.com/firecracker-microvm/firecracker/blob/49003d0f2f8573f6e1af6d22232c0917a0253189/src/devices/src/virtio/net/tap.rs#L123 Actually it would be nicer if the XY number for /dev/tapXY is detected from its interface name via something like let mut ifreq = ifreq_for(name);; ifreq_ioctl(socketfd, &mut ifreq, SIOCGIFINDEX).map(|ifindex| ifindex as usize);

[pothos]

Here is a small helper script for testing during development. It uses the passthru mode to forward the current default NIC to the VM (temporarily blocking access of the kernel network stack). This also works when on a WiFi (but not for too long since wpa_supplicant can't renew the authentification while the VM uses the NIC).

#!/bin/bash
set -euo pipefail

if [ $# = 1 ]; then
  IFN="$1"
else
  IFN="$(ip route get 1 | grep -o 'dev.*' | cut -d ' ' -f 2)"
  echo "Detected $IFN based on default route"
fi
TAP="$IFN"pass
echo "Creating macvtap is passthru mode for $IFN (normal kernel network access will be blocked)"
echo "When done using it, delete with:"
echo "  sudo ip link delete $TAP"
sudo ip link add link "$IFN" name "$TAP" type macvtap mode passthru
MAC="$(cat "/sys/class/net/$IFN/address")"
sudo ip link set "$TAP" address "$MAC" up
DEV="/dev/tap$(cat "/sys/class/net/$TAP/ifindex")"
sudo chown "$USER:$USER" "$DEV"
echo "Ready! MAC: $MAC DEV: $DEV IFACE: $TAP"

[raduiliescu]

I am renaming this item and I'm adding it on the Firecracker roadmap as a research item.

[richardcase]

@raduweiss - just following up on our slack chat. This is essential for the solution for DT and it would be good if this could be released before next year as we are planning a poc in Jan 22.

[ghost]

I'm really hoping to see macvtap supported in firecracker. The business reason is simplicity.

[richardcase]

I'm really hoping to see macvtap supported in firecracker. The business reason is simplicity.

@bootrino - we currently use this feature from the feature branch. As its no released, we have a fork where we've merged 1.0.0 with the macvtap branch so that we aren't blocked.

[majek]

@richardcase Do you have benchmarks? Also, would you mind rebasing the branch? (it conflicts with main)

[ccrvlh]

Hi, we've been looking for a firecracker manager and the work being made by weaveworks with flintlock and @richardcase is looking great. Is there any perspective on having MacVTap supported and merged? Would be happy to contribute.

BTW, @majek weaveworks have been maintaining a fork with macvtap here https://weaveworks-liquidmetal.github.io/flintlock/docs/getting-started/firecracker/

[dzenc]

If you want, you can comment on https://github.com/firecracker-microvm/firecracker/issues/1933 with your usecase, to see if it can gain some traction again :)

Following up from #2217 - I can explain my specific use case, if needed (which involves nested KVM firecrackers), but I actually think the best argument for implementing macvtap support is that it greatly lowers the barrier-to-entry for someone new to try out firecracker, because it greatly simplifies the initial networking configuration. If a goal is get more people using firecracker, then lowering the barrier-to-entry is a "big deal." (From personal experience, I put firecracker on the back burner several times before actually getting it working because the network config was frustrating, and I didn't want to make global changes to my system - e.g. enabling forwarding, just to try it out.)

[xmarcalx]

Hi @dzenc and @ccrvlh ,

Thanks for your messages and sorry for the late reply. I am the manager of Firecracker so feel free to reach me out on slack if you need anything. In mid 2022 we tried to resume this work and try to find a way to merge it on main with @richardcase considering weaveworks was having it on a feature branch from a while, but i think the communication channel broke because ended up de prioritized by both parties. This work require someone to mainly:

1. rebase the feature on top of main
2. show some benchmark data to showcase any improvement compared to our current solution
3. firecracker team members helping to assess security posture of the device, make sure there is way that we can test regularly this device and setup too so things do not get stale.

So if there are people interested in 1 and 2, we can sync with item 3 and see the best path forward. We were also considering the opportunity to add a new network device in Firecracker to improve performances and flexibility overall (if that was a drive). @majek posted an issue on vhost-net #3707 where there is a bit of chat on the topic.

[richardcase]

@xmarcalx - i'm no longer at Weaveworks but i'd be interested in helping to get this feature in and could help with 1 & 2.

[xmarcalx]

Hi @richardcase (et all),

thanks for the reply, and sorry for the late reply, it has been holiday and therefore OOO season.

It is great that we count on your help for 1 & 2, thanks for that.

As i was citing in my previous message, seems there is a lot of interest in modifying the networking backed of Firecracker among all of us. There are multiple initiatives in this direction: this macVtap, the proposal on vhost-net, and our idea to use vhost-user-net in line with the recent effort to use that technology for the block device. I think we need to stay true to our tenets and try to understand if we can catch all the needs and usecases with just one implementation of the three. This will keep the code simpler and also unify our effort and cycles spent in this area, hopefully satisfying everyone. If not, then kick off the streams of work previously cited.

To better understand the usecase (also because some time passed from when the issue was opened 😢 ), I have some questions:

1. are you aware of someone still actively using macVtap?
2. Will macVtap help "just" with simplicity of setup or you saw also some performance gain?
3. Would a new vhost-net/vhost-user-net solution help you for example to address the problem you were trying to solve with macVtap? Which would be your concerns with those solutions?

Anyway, we decided to track the effort to rethink about Firecracker network backed in our roadmap such that we are going to make a decision and implementation hopefully within 2024 and we do not delay this or any other networking decision further.

Thanks a lot as usual for your contribution.

[xmarcalx]

Removing tracking from roadmap since we are tracking network improvements already as part of #4364 and we still do not have clear datapoint on previous questions