How to establish L2 connection between 2 user applications?

[MarkMckessock]

Question

I am looking to implement a simple routing demo like this using NSM:

host <-- (172.16.1.0/31) --> router <-- (172.16.2.0/31) --> host

Essentially, I'm trying to establish L2 connections between each of the pods either by inserting a sidecar into the router or using an L2 forwarder NSE between each pair of devices. ex: client[NSC] <-- [NSE] l2_forwarder [NSE] --> [NSC]router[NSC] <-- [NSE] l2_forwarder [NSE] --> [NSC]client

My current strategy has been to deploy the cmd-nse-icmp-responder as a sidecar next to the router which allows me to create a topology that looks like this: client <-- (172.16.1.0/31) --> router

However, my second client remains 'Pending'. I also tried deploying two instances of the cmd-nse-icmp-responder as sidecars with different names which resulted in two kernel interfaces being created in my NSE pod, however both of my clients were assigned 2 IP addresses, one from each of the subnets configured on the two cmd-nse-icmp-responder pods.

Any advice on the best way to approach this would be appreciated.

Examples

I believe my use-case is similar to #5247. I attempted to implement the suggestion provided here by @denis-tingaikin but I'm fairly new to Go and I couldn't find much documentation for the SDK.

[glazychev-art]

Hi @MarkMckessock , Would a wider mask work for you? If I understand correctly, you want to try:

client[NSC(172.16.1.1/30)] <-- [NSE(172.16.1.0/30)] l2_forwarder[NSE(172.16.1.2/30)] --> [NSC (172.16.1.3/30)]router[NSC...] <-- ...

If so, then set this env to one of the cmd-nse-icmp-responder sidecar:

            - name: NSM_CIDR_PREFIX
              value: 172.16.1.0/30

[MarkMckessock]

I guess my ideal scenario would be to have an NSE that acts like a virtual switch and would effectively forward Ethernet frames coming from one connected NSC to any other attached NSCs.

I'd like the clients in my example above to be on two separate networks and have the router in the middle perform routing between them. If I were to use the 172.16.1.0/30 mask, both of the clients would be placed on the same network.

[denis-tingaikin]

@MarkMckessock At this moment, we don't plan to add L2 router.

But if you want, you can start work on it and we'll love to help you with implementation and questions :)

[edwarnicke]

@MarkMckessock It sounds like what you really want is a bridge domain network service. This should actually be pretty easy to build.

First, remember that your BridgeDomain network service would be 'payload' Ethernet, not the default 'payload' IP. From the client side, this is actually pretty easy, as you'd simply want to define your NetworkService as taking payload IP.

Where things get interesting is on the NSE side. What you would want to build is a BridgeDomain NSE to provide that Network Service. The good news is that should be pretty simple to do if you are comfortable with writing code in Golang.

Network Service Mesh builds NSEs using small reusable components chained together. The good news is that this is actually pretty simple.

I'd recommend for a first pass that you do a single localized bridge domain in your NSE. Later, once that is working well, you can look at building out a 'distributed' bridge domain across multiple NSEs if you'd like.

I know that @denis-tingaikin has a tool for making it easy to generate a new NSE from a template he could point you to :)

Outside of various boiler plates for things like Spiffe identities that the template should handle for you, the core of what you'd need to do is create an NSE object, and server it via GRPC. Network Service Mesh uses a 'chain of responsibility pattern' internally, where you construct NSEs of pieces that do various parts of the work. So for your bridge domain you'd basically need something that

1. Creates an interface for the incoming connection.
2. Plugs that interface into a bridge domain.

The good news is... we have existing components for the first one, and very close to the chain element you are looking for for the second one.

For 'plug in an interface' I'd recommend having a look at:

https://github.com/networkservicemesh/cmd-nse-vl3-vpp/blob/12cfa4a8278050bef8c5824ed6e4d8937dc39301/main.go#L546-L553

which simply provides the logic for plugging the incoming connection in as a memif interface (your fastest choice)

We have an existing l2bridgedomain chain element that was constructed for a vlan related use case that does almost what you need. It plugs the created interface into an l2bridgedomain (creating it if it doesn't already exist). There's a little bit of logic you'd need to strip out around plugging that into vlan tagging elsewhere... but it should be pretty easy.

In terms of the chaining, logically, NSM chains make decisions down the call chain, and take actions up the call chain. For that reason, if some action has to happen before another, it needs to be later in the call chain, so logically you'd have something like:

1. bridgedomain
2. memif

as an ordering.

@MarkMckessock So my question to you: is this something you'd be interested in doing? Because we'd love to help you along :)

[MarkMckessock]

Thanks for the very detailed response @edwarnicke. I'll take a look at what you linked and try to brush-up on my Go. What would be the best place for questions about NSE developpment? Github? Slack?

[edwarnicke]

@MarkMckessock

What would be the best place for questions about NSE development? Github? Slack?

That depends entirely on what you are comfortable with. I'm happy to continue the conversation here if that works for you (and it would also keep everything in context). I'm also fine continuing it on slack with one small caveat: if you @ me on slack then my client notifies me... otherwise it may take longer to notice.

[denis-tingaikin]

What would be the best place for questions about NSE development? Github? Slack?

I'd be recommended to use Github, #nsm-dev for questions and for history. These questions may be super useful for other folks who will implement their own NSE

[edwarnicke]

@denis-tingaikin Is your nsmctl generate in a state that @MarkMckessock could use it yet?

[denis-tingaikin]

@edwarnicke yeah, it's a great idea to test nsmctl ;)

@MarkMckessock To start implementing ur own vpp NSE you could use nsmctl tool.

Steps

1) go install github.com/networkservicemesh/nsmctl@latest 2) cd $GOPATH/src 3) Use help to see generating options

nsmctl gen nse --help

4 Gen vpp nse

nsmctl get nse vpp --name 'any nse name' --path 'my-nse-vpp'

5. Open my-vpp-nse folder with editor and it should be a great point to start ;)

[edwarnicke]

@ljkiraly Wrote the l2bridgedomain chain element and may also have thoughts :)

[ljkiraly]

I checked the l2bridgedomain chain element and seems that is easy to modify for generic usage. @MarkMckessock I can help understanding/using this part if needed and do the refactor if needed.