search

hashicorp / nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
https://www.nomadproject.io/
Other
14.95k stars 1.96k forks source link

`mode = "bridge"` support for FreeBSD #17311

Open mayakerostasia opened 1 year ago

mayakerostasia commented 1 year ago

I don't know how possible this might be, but I'm willing to spend some time trying to get it to work. If anybody knows where I might start that would be amazing.

Proposal

FreeBSD support for Bridge networking mode

Use-cases

Some work has been done to make nomad work with pot the jail driver and the only downside is that FreeBSD doesn't seem to be able to support mode = "bridge" so that Consul Service Mesh can be taken advantage of when using Pot as the task driver on FreeBSD systems.

Attempted Solutions

Installed FreeBSD specific CNI plugins and installed them in the default directories. Tried a custom cni/mynet specification but the error throws that it must use "bridge" and not "cni/mynet"

lgfa29 commented 1 year ago

Hi @nwisemanII 👋

I think this would be great, and we really appreciate the desire to contribute some code for this 😄

I don't have much experience with BSD, so I'm afraid I can't help you much here. But I will check with my colleagues if someone would be able to provide better guidance.

From a Nomad perspective, I imagine you would spend most of your time in the allocrunner and, more specifically, in the network hook and network manager (which has a linux and non-linux version).

Maybe just allowing the network manager to run on FreeBSD would enough, but who knows 😅

https://github.com/hashicorp/nomad/tree/main/contributing has some general guidance on building and testing Nomad. It would also be nice to have a FreeBSD host in our end-to-end test suite, but that can be worked out in the future.

I hope this gives you a place to start. Feel free to reach out if you get stuck 🙂

tgross commented 1 year ago

@nwisemanII one challenge that comes to mind is that Nomad rummages around looking for the kernel module for bridge networking, which probably won't work at all on FreeBSD. Unfortunately it does it in a way that's terrible and buggy :blush: (ref https://github.com/hashicorp/nomad/issues/10902 for just one example), which would probably be improved by https://github.com/hashicorp/nomad/issues/6618 and that'd presumably work fine for FreeBSD too.

lgfa29 commented 1 year ago

Another piece of information I received from the team is that FreeBSD doesn't have network namespaces, like in Linux. The closest option would be jail VNETs (https://klarasystems.com/articles/virtualize-your-network-on-freebsd-with-vnet/).