search

slackhq / nebula

A scalable overlay networking tool with a focus on performance, simplicity and security
MIT License
14.52k stars 975 forks source link

Container Networking Thoughts #175

Closed keverw closed 7 months ago

keverw commented 4 years ago

I was originally looking at container networking, so wonder if this could be used to give each container their own IP address or in the future? Pretty sure others would love this for that use case too. So you could have container hosts running in different locations or providers, and have your own internal network between them, so then the only stuff exposed to the public internet IPs would be your entry points to the system like your load balancers, etc.

I also do like how you can manually set IPs too, it seems some other systems handle this all for you magically so being able to give a server you know going to be around a long time a static IP is cool! I know I heard someone was running a cluster in their containers using another solution and the thing reassigned the same IP to another similar node during reboots and caused different clusters to join each other causing issues - so they had to customize the IPAM for their use case so being able to manually control is a great feature already.

Just not too sure how you would handle this with containers or if the same host could use multiple IPs(even as separate certs so don't have to allocate IPs together per node in case you later wanted to move a container). I know a very early project though, and I know other networking projects seems to use their own CNI plugin for this sort of thing, to use with Docker, Podman, etc.

I think getting this working with one private IP per host would work for sure and then just give each private container a different port on that interface but I really would love giving each container their own IP treating them like their own mini virtual machine in a way so one big network no matter where you are running your workloads. So not sure the whole every container getting their own part or possible yet with this, but I do believe this could be a very popular use case.

nbrownus commented 4 years ago

CNI support is likely in the future. Nebula is in use today in kubernetes as a sidecar with multiple pods having their own unique address per host. I'm personally not very familiar with the k8 ecosystem but it would seem there could be some manual work to booting nebula in a network namespace that gets handed to a pod/container later.

keverw commented 4 years ago

Neat to know it's possible then! Not sure if I plan to use Kubernetes yet or not, was thinking of using it directly with a container runtime. Playing around with an idea virtual world related, so planning to spin up chunks of the world as needed and network them together but still very early.

nbrownus commented 4 years ago

Ah, that's a bit easier depending on how you set it all up. I personally run a number of LXD containers with veths on the host as NICs in the containers then just boot up nebula inside the container and away you go.

keverw commented 4 years ago

Interesting, I guess each container runs their own copy of nebula then? or you run one container with nebula and then connect your app container to the other kinda? I personally haven't played with LXD though but heard of it, I believe Chrome OS Linux app support implementation uses that. Wonder if that'd be simpler maybe, I was planning to use Node and Typescript as the agent on each machine to kinda manage and glue everything together but so far not really committed to any specific container tech yet but I know I really like the idea of using containers.

nbrownus commented 4 years ago

Yes, each container gets its own certificate and runs its own copy of nebula in my setup.

keverw commented 4 years ago

Neat, kinda goes a against the idea of one process per container but sounds do able...

Then I guess if I made a load balancer I'd have a list of IPs as part of the nebula network and expose say port 80, 443 to the host's IP then to the public internet. Haven't played with veths yet, was reading some other tutorial for another project talking about creating virtual ethernet cable pairs, so wonder if that's similar... I guess then you could just use a generic small container with nebula and then plug it into your other container with your app's own code.

nbrownus commented 4 years ago

That’s where network namespaces would come into play to share a network amongst a group of containers.

On Sat, Jan 25, 2020 at 2:58 PM Kevin Whitman notifications@github.com wrote:

Neat, kinda goes a against the idea of one process per container but sounds do able...

Then I guess if I made a load balancer I'd have a list of IPs as part of the nebula network and expose say port 80, 443 to the host's IP then to the public internet. Haven't played with veths yet, was reading some other tutorial for another project talking about creating virtual ethernet cable pairs, so wonder if that's similar... I guess then you could just use a generic small container with nebula and then plug it into your other container with your app's own code.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/slackhq/nebula/issues/175?email_source=notifications&email_token=AAHJXBZIQGLTG5D7FPXKX6DQ7S7Y3A5CNFSM4KLSU4UKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ5HLVY#issuecomment-578450903, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHJXB4R7KQ7S63252TCUZ3Q7S7Y3ANCNFSM4KLSU4UA .

keverw commented 4 years ago

Oh neat, It sounds like I need to study up and learn more about LXD. I know someone else posted an issue here talking about them using LXD, seems like multiple different competing runtimes but that one sounds more direct maybe to use for my use case.

bmullan commented 4 years ago

Great thread! I do something similar with LXD!

Each container still gets its own IP address because I use LXD to create a separate "custom" LXD bridge for each tenant, set each Bridge to provide dnsmasq/dhcp ranges for the containers created & running behind each of those bridges.

So far in initial tests seems to work. Each LXD container for each Customer could be running whatever you might normally run on a "linux server" if it was a VM ... except there is no VM overhead here and each LXD container is sharing the Host's Kernel.

In my testing case the lighthouse was on a Hetzner Cloud instance in Germany. I didn't use AWS but had 2 cloud instances on Digital Ocean's Cloud here in the U.S. plus 3 KVM VMs running on a Server here locally.

The cloud instance "servers" were basically a "mirror" of the bottom of the diagram (a server & 3 LXD bridges each connected to 1 or more TenantN LXD containers.

My Use-Case diagram is below. Click on it to enlarge

CIAB Overview Diagram (generic)

To help understand/explain my diagram...

By "multi-tenant"...

By "multi-node"...

By "multi-cloud/hybrid"...

Some "tenant" might have their own Data Center(s) but they may also have some of their computer services in a Cloud Provider like Amazon's AWS, or Microsoft's Azure or Digital Ocean etc.

Full-mesh means...

any Node/Server can reach any other Node/Server directly as long as that Node/Server is connected to the Mesh VPN

To ROUTE TenantN data i've configured BGP on each "NodeN".

On Node0 I also configure BGP Route Reflectors

Each TenantN data traffic is "tagged" with a different Private BGP AS number on each NodeN but the TenantA AS number will be the same all Nodes, TenantB AS will be the same on all nodes etc

For BGP the following AS numbers are reserved for private use by IANA.

Number Range ................... Description 64512-65534 ....................... Reserved for Private Use 4200000000-4294967294 ... Reserved for Private Use [RFC6996

example:

TenantA might be tagged with AS 65001

TenantB might be tagged with AS 65002

TenantC might be tagged with AS 65003

This has BGP routing each Tenant's traffic separately from every other Tenant's traffic. but one Company can't access another Companies data even though they are on the same Mesh Network.

Auto-Learning means...

to add a New computer Node/Server I only have to configure it to "peer" (ie connect) to ANY existing Member Node/Server of the VPN Mesh network and that New

Node/Server will automatically learn the "best route" for a direct connection to every other existing Node/Server in the Mesh Network.

Also, when adding a New Node/Server, every other existing Member Node/Server will learn how to connect directly to the New Server.

In essence the VPN Mesh Network of Nodes is self-configuring...

Not sure if that helps or makes things worse to understand... wink

Nathan & Kevin... if you have an interest in some collaboration on using Nebula with LXD I've got another LXD user I know that also is a fan of Nebula and perhaps we could collaborate and come up with a "generic" design w/some documentation that could be replicated/used by Nebula users ?

Let me know if this might be interesting and I can contact my friend.

Kevin... for LXD info here are a couple great links to keep handy:

LXD Users forum - devs answer questions daily here

Simos Xenitellis' awesome LXD Blog Series

The LXD Sub-Reddit

Brian Raleigh North Carolina

keverw commented 4 years ago

Yeah the grouping part is cool too, I personally just wanted something simpler to network with but I know corporations and other uses like to the idea of isolating things, say sensitive PCI data to limit the scope of access, so probably only the admin nodes could talk to those nodes instead of the entire application being able to access it, just in case an exploit ever. I know for example if Facebook thinks you used a fake name with their real name policy, they require to upload your photo ID. So don't see a reason the chat edge nodes should need access to photo ID scans. So groups are a bonus.

Also, IPV6 looks promising too Unique Local Addresses #6 - which would give more flexibility and probably would need to worry about conflicting with other local IP addresses and things especially if you wanted to run larger clusters. Then I know some compaines have said they only use IPV6 internally now too, only thing IPv4 is public facing servers since ISPs are lagging with upgrading and switching over.

I feel like researching server stuff I could spend days and days just reading things, watching tech talks, debating with myself without getting anything done.

I am wanting to build a multiple zone aware systems that in a few clicks could spin up database clusters like MongoDB or some of the newSQL ones, file servers, etc primitives to build an app on + app servers like your API or WebSocket servers(Gaming, chat, etc). So my ideal dream is to build something similar to a control panel but instead of a single box for shared hosting, it's one that's powerful enough to build large apps on and scale mostly just by throwing more nodes at it. Services support with load balancing options or for some apps like databases/games you'd manually place them somewhere in the cluster instead of automatically scaling things. Want security and easy to use, reliability to be the focus. I know there's another clustering tech out there and seen an article talking about how complex it is, there are worries about security holes too. So I guess I plan to build some sort of self-hosted PaaS that could run on VPSes or Dedicated servers even would be a way to explain it, but goal is to try to make it powerful and simple. Probably should set a rule to write documentation as you write code too, so it's understandable.

Then use that PaaS to power my products but the PaaS itself I was thinking of making it open source since I feel this sort of thing is very powerful and useful to society, but thinking maybe just require in the license that if a cloud wants to take and run with it, they'd have to pay a small amount per gig of ram to help contribute back to the project(I figured that's fairer than per CPU too since more and more stuff is running in the cloud anyways), which could pay contributors that way. Idk though, I know funding in open source is tricky though so unsure. Kinda make it be the Linux for server apps. Just something I've been thinking about on and off over the years when it comes to server management stuff. Then use only computing power and attached storage on the clouds too for volumes. So no cloud specific services to be locked into, since I know some providers offer all these extra value added proprietary services but might have to rewrite your app to switch providers in that case.

bmullan commented 4 years ago

@nbrownus @keverw

Primarily directed at Nathan. Nathan I didn't want to "assume" you know this but with LXD v3.19 which was just released last week... they have added an alternative to the traditional Layer 2 Bridge (lxdbr0) that connected the Host/Server & Internet to the LXD containers.

But insofar as VPNs go that Layer 2 Bridge (lxdbr0) makes it easy to integrate TAP based VPNs but TUN based such as Nebula and WireGuard then require extra work to interconnect the Layer 3 TUN (nebula1) with the Layer 2 lxdbr LXD Bridge. It works, its just extra work.

In LXD v3.19 they have introduced the Routed NIC interface.

I've been wanting to use this with Nebula's TUN interface but haven't had time "yet".

A couple days ago I'd posted an "issue" on the LXD GitHub asking if someone could post some example/reference information on how to implement LXD containers using a Routed NIC (ie Layer3) versus the alternative Layer2 (lxdbr0 bridge).

Thomas Parrott, one of the LXD development contributors did an extensive reply which I summarized and _put into a PDF to make it easier to read_.

I think adopting the Routed NIC for Nebula Container use is the way to go now (note: haven't tried it yet :-) myself.)

So in my thinking... install Nebula on the "Host/Server".

See "Example Usage #2" in the PDF.

@nbrownus You said you "run veth" on the hosts as NICs in the LXD Containers.

From my reading on the new Routed NIC I "think" that is what LXD is now doing for you!

So it can simplify your design.

Maybe it was one of Thomas Parrott's comments.

Brian

nbrownus commented 4 years ago

I'll have to look into it, thanks for letting us know!

bodleytunes commented 4 years ago

@keverw

Just to chime in with some of my thoughts, even though I love LXD and use it for all my server needs, if your wanting to build something that scales sideways where you are throwing more umph at it and spinning up more containers/services, you really need to look more at Kubernetes as its been designed from the ground up for this and people are already building cni's that create the fabric between the pods and the physical nodes, and its all automated, weave, calico, canal and more - thats the thing (the killer feature). There is no concept of any ip address that you would need to configure or even know about as its abstracted away, everything uses its own internal dns and you refer to the application names or fqdn. This means it scales perfectly.

If this (kubernetes) could be integrated with the encryption advantage and firewalling of nebula (say as was mentioned above using side cars) then this would be what I would be heading towards trying to use and get my head around.

For quicker and more static setups (like my own meagre needs) then LXD most definitely would be better for that.

Cheers!

Jon.

keverw commented 4 years ago

@bodleytunes Yeah, I thought about kubernetes but sounds like running in multiple cloud there's mixed stuff, I guess you should federate but not sure the progress with that. So personally wanting to build something super lightweight and specific stuff I want where I can understand what's going on more, so not really a blackbox. I know though it's hugely popular though.

keverw commented 4 years ago

I’ve been kinda out of following the whole containers news and stuff. Seen that Podman is going to have a API more like Docker instead of the varlink protocol (which no Node lib for). So that sounds exciting :slightly_smiling_face: https://podman.io/blogs/2020/01/17/podman-new-api.html

I did look at LXC more, but the whole idea of treating them as VMs and didn't seem to have a dockerfile equivalent, so I'm still aiming for something like Podman since I can wrap my head around it more.

I think a CNI thing at some point would be cool but no clue where to start with something like that. but I guess Podman is like a Docker alternative but security people I guess like it. Image format is compatible too. and also has a CNI interface. Right now I kinda shifted my focus though to working on parts of my apps idea itself, but the PaaS and infrastructure stuff I dream about seems will be able to get it going some day as this space progresses, and plus some things I want the app and platform itself to communicate for some things too but not that far along yet.

raqbit commented 3 years ago

I'm running a couple of plain-old servers in a Nebula network and I would like to make some Kubernetes-based workloads join this same network for easy & secure communcation between services. As far as I can tell there currently is no way built-in to make this work. Is container networking still planned for the future of Nebula? How should this work with the dynamic nature of containers? (Dynamic IPs? IP-per-service? Via "sidecars" or at a lower level?)

mzehrer commented 1 year ago

@nbrownus Can you explain how you use nebula with k8s?