search

NetSys / bess

BESS: Berkeley Extensible Software Switch
Other
313 stars 156 forks source link

Run BESS inside a container #942

Open jianfenw opened 5 years ago

jianfenw commented 5 years ago

I wonder if there is a way of running BESS inside a container. The scenario is as following (1) multiple BESS processes running at a single host (2) a vSwitch (e.g. BESS) that connects all BESS processes with the physical NIC. Is it feasible to do so? I am quite curious and really appreciate if you could provide guides (or possible hints) for me to try out this scheme.

trozet commented 5 years ago

Hi @jwangee. Yes it is possible. We already do it in the Nimbess project. The docker images are here: https://cloud.docker.com/u/nimbess/repository/docker/nimbess/bess

The only caveat is that the kmod needs to be compiled and loaded on the host first.

jianfenw commented 5 years ago

Nice. This looks interesting. Nimbess seems to be a fast-path Kubernetes CNI option. I could run BESS on my local host. Also, I've compiled and loaded kmod on the host, which I guess is necessary for creating VPort and forwarding packets for containers. I could also run BESS on host and creating VPort which is visible inside a specific container. Then, I tried to run BESS insdie the Nimbess/bess container, but failed with the following messages. I wonder if you've gone through this.

$ docker run --privileged -it --rm nimbess/bess bash [root@0684b28bb899 bess]# ./bessctl/bessctl Type "help" for more information. Connection to localhost:10514 failed Perhaps bessd daemon is not running locally? Try "daemon start".

$ daemon start E1025 18:33:43.111024 38 bessd.cc:356] Failed to launch a daemon process *** Error: Cannot start BESS daemon Followup: I figured it out. We still have to mount the Hugepage host directory to the container. $ docker run -v /dev/hugepages/:/dev/hugepages/ --privileged -it --rm nimbess/bess bash
trozet commented 5 years ago

Right, unless you use "-m 0" which I think will disable hugepages and bessd will start. Here is the k8s yaml we use to launch bess:

https://github.com/nimbess/nimbess-agent/blob/master/k8s/nimbess.yaml#L55

jianfenw commented 5 years ago

Thanks a lot for confirming this. It helps. I could run BESS as a vSwitch for containers. One more thing that I am curious (I didn't succeed in doing so in my local machine): is it possible to run multiple such BESS containers together at the same machine (and forwarding packets between VPorts)? (I doubt it because these BESS containers would share the same set of Hugepages.)

trozet commented 5 years ago

You might be able to do it if you have multiple numa nodes and you isolate hugepage mem allocation per BESS instance to a single node (also configure lcores to the right socket). That's something that is missing in BESS right now. Today if you specify a huge page size with -m arg, it will automatically try to allocate that size from all sockets.

trozet commented 5 years ago

I started on: https://github.com/trozet/bess/commit/6f558f77e6e4954ca63fbc32aa79ce5f12f9d3b9

It will let you start BESS and allocate hugepages from only 1 socket. However it is crashing when the workers are started. Even when I specify worker core it is still using core 0. Looking into it.

trozet commented 5 years ago

Looks like the worker 0 was being used by Nimbess controller by default and that was causing bessd to crash. I added a check during AddWorker to ensure the core requested is valid for the socket being used. Works for me. Give it a try and see if you can launch multiple bessd containers with hugepages on different sockets:

946