search

kata-containers / runtime

Kata Containers version 1.x runtime (for version 2.x see https://github.com/kata-containers/kata-containers).
https://katacontainers.io/
Apache License 2.0
2.1k stars 377 forks source link

Network hotplug issues with firecracker #3165

Closed networkop closed 3 years ago

networkop commented 3 years ago

I'm trying to test the network hotplug functionality with kata-fc with netmon enabled. When adding a new interface to the container's namespace, it's not showing up inside the VM. At the same time the following can be observed in the logs:

May 03 10:27:12 xps kata-runtime[300067]: time="2021-05-03T10:27:12.948107948+01:00" level=warning msg="hotplugAddDevice: unsupported device" arch=amd64 command=kata-network container=b4b374a2855da7b1e2353ba80bb21ef9091e9519e4253a17dc5d91b7a7577af0 devInfo="&{{{85daa5b4-adcc-44bc-9fd0-18891843ea6b br1_kata {tap1_kata 52:8f:75:ca:29:11 []} [] []} {eth1 46:83:0c:45:88:df []} 2} {{{0 65000 0 eth1 52:8f:75:ca:29:11 0 0 0 0 <nil>  <nil> 0 <nil>  <nil> unknown 0 0 0 0 0 []} veth} [] [] {[]  [] []} []} virtual }" deviceType=2 name=kata-runtime pid=300067 sandbox=b4b374a2855da7b1e2353ba80bb21ef9091e9519e4253a17dc5d91b7a7577af0 source=virtcontainers subsystem=firecracker
May 03 10:27:12 xps kata-runtime[300067]: time="2021-05-03T10:27:12.948250174+01:00" level=error msg="Error attach virtual ep" arch=amd64 command=kata-network container=b4b374a2855da7b1e2353ba80bb21ef9091e9519e4253a17dc5d91b7a7577af0 error="Could not hot add device: unsupported device: &{{{85daa5b4-adcc-44bc-9fd0-18891843ea6b br1_kata {tap1_kata 52:8f:75:ca:29:11 []} [] []} {eth1 46:83:0c:45:88:df []} 2} {{{0 65000 0 eth1 52:8f:75:ca:29:11 0 0 0 0 <nil>  <nil> 0 <nil>  <nil> unknown 0 0 0 0 0 []} veth} [] [] {[]  [] []} []} virtual }, type: 2" name=kata-runtime pid=300067 sandbox=b4b374a2855da7b1e2353ba80bb21ef9091e9519e4253a17dc5d91b7a7577af0 source=virtcontainers subsystem=network

Inside the conainer's namespace, I can see both the new interface (eth1) and that corresponding tap interface (tap1_kata)

p -br link show
lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP> 
tap0_kata        UP             fa:4a:06:3e:48:d7 <BROADCAST,MULTICAST,UP,LOWER_UP> 
tap1_kata        DOWN           ce:9d:9a:bd:24:fa <NO-CARRIER,BROADCAST,MULTICAST,UP> 
eth0@if427       UP             02:42:ac:14:14:02 <BROADCAST,MULTICAST,UP,LOWER_UP> 
eth1@if431       UP             ae:a6:4b:71:35:4b <BROADCAST,MULTICAST,UP,LOWER_UP>

The same functionality works with QEMU VMM just fine. Is this supposed to work with FC and/or CLH?

networkop commented 3 years ago

When running with CLH as the VMM, the error log looks like this:

May 03 11:00:48 xps kata-runtime[307307]: time="2021-05-03T11:00:48.402362277+01:00" level=error msg="add interface failed" arch=amd64 command=kata-network container=dc15d5bb0e3a3d9274227d329e818d2039a3f4c4f07d26a8c0d4b294e595ea72 error="cannot hotplug device: unsupported device type '2'" name=kata-runtime pid=307307 resulting-interface="<nil>" sandbox=dc15d5bb0e3a3d9274227d329e818d2039a3f4c4f07d26a8c0d4b294e595ea72 source=runtime
May 03 11:00:48 xps kata-runtime[307307]: time="2021-05-03T11:00:48.402420499+01:00" level=error msg="cannot hotplug device: unsupported device type '2'" arch=amd64 command=kata-network container=dc15d5bb0e3a3d9274227d329e818d2039a3f4c4f07d26a8c0d4b294e595ea72 name=kata-runtime pid=307307 sandbox=dc15d5bb0e3a3d9274227d329e818d2039a3f4c4f07d26a8c0d4b294e595ea72 source=runtime
May 03 11:00:48 xps kata-netmon[306894]: time="2021-05-03T11:00:48.404259786+01:00" level=fatal msg="handleEvents()" error="exit status 1" name=kata-netmon pid=306894 sandbox=dc15d5bb0e3a3d9274227d329e818d2039a3f4c4f07d26a8c0d4b294e595ea72 source=netmon
networkop commented 3 years ago

Looks like CLH doesn't support hotplugging of network devices: https://github.com/kata-containers/runtime/blob/59e227336903383fcb04e0075e0b55cbd98c42bb/virtcontainers/clh.go#L471

networkop commented 3 years ago

Ha, looks like FC doesn't support it either https://github.com/kata-containers/runtime/blob/59e227336903383fcb04e0075e0b55cbd98c42bb/virtcontainers/fc.go#L1042

networkop commented 3 years ago

So, I guess this is by design, so feel free to close this issue. I'm just curious if it's possible to avoid hotplugging and connect multiple network interfaces before FC VM starts.. Is this possible?

egernst commented 3 years ago

It’s been a while… but I believe you can create a runc container, add a second interface to it, and then create a kata container that uses the net of the first runc container. This’ll result in a kata container running with two interfaces at boot, rather than relying in hotplug.

egernst commented 3 years ago

See https://gist.github.com/egernst/0c8acf60d5aa4b6ab9d36517580c760a#launching-clear-container-with-two-vhost-user-interfaces

networkop commented 3 years ago

yep, perfect, just as I was hoping. thanks again @egernst :+1: