Custom interface and IP address for the client request

[bilgehan-erman]

With the recent releases, the NSM improvements, stability, and the new features have been impressive. Thank you for all those contributing to this impressive work.

However, as you may agree, user guide-type documentation to help with advanced SDK use will take some time to catch up.

Back in around the 0.2.0 release we had a way to create complex topologies with per-link custom IP assignments and custom interface names. But of course none of that works anymore. Something like in the following diagram:

Now with all the updates, it is not clear how and if we can do these kind of configurations (SDK, coding custom NSCs and NSEs).

Where we are stuck is figuring out how to make the NSE act on a set of labels in different client requests such as “… peerintf = intf2 & source=1200::1:1/96 & destination = 1200::1:2/96” to assign custom IP to the custom interfaces (as indicated in the client request context).

Any guidance will be highly appreciated.

[glazychev-art]

Hello @U24543426, Thank you for appreciating!

I think you are right that it is convenient to be able to configure this.

1. So, as you can see in the examples, in order to configure the client, you can use an NSM annotation like this:

   annotations:
   networkservicemesh.io: kernel://icmp-responder/nsm-1

   nsm-1 is the client interface name.To set it as intf4 - just specify networkservicemesh.io: .../intf4

2. I think it's not possible to set 1200::1:1/96, because we mostly use point2point IPAM and it allocates addresses with /32 for IPv4 and /128 for IPv6 prefix length. So now we can for example: nsc1 (1200::1:1/128) ---- (1200::1:0/128) nseA (1200::1:2/128) ---- (1200::1:3/128) nsc2 To do this, we need to specify NSM_CIDR_PREFIX env for the NSE:

           - name: NSM_CIDR_PREFIX
             value: 1200::1:0/126

3. Custom NSE interface name - unfortunately this is not currently available, it is generated automatically. But I think we can consider it.

[glazychev-art]

@edwarnicke Could you please share your thoughts?

[glazychev-art]

Also I guess we need to fix the prefix setting. For example, if we use:

    - name: NSM_CIDR_PREFIX
      value: 1200::1:0/127, 1200::2:0/127

that means:

...
    point2pointipam.NewServer(1200::1:0/127),
    point2pointipam.NewServer(1200::2:0/127)
...

This is what we use for dual stack

I propose to make it possible to combine several prefixes into one ipam. For example:

    - name: NSM_CIDR_PREFIX
      value: [1200::1:0/127, 1200::2:0/127], 1200::3:0/127

that means:

...
    point2pointipam.NewServer(1200::1:0/127, 1200::2:0/127),
    point2pointipam.NewServer(1200::3:0/127)
...

[glazychev-art]

/cc @denis-tingaikin

[bilgehan-erman]

Thank you for the detailed info @glazychev-art Given the IPAM constraints on the NSE, I'm also considering setting the payload to ETHERNET and manage the IP binding by the non-NSM processes within the POD -- once we get the individual interfaces/links setup between the nodes.

Not sure if this makes any sense but for the custom interface name, I'm considering catching the Request in the NSE and use the Mechanism helper func (m *Mechanism) SetInterfaceName(interfaceName string) to assign the name (the name may get there somehow, like with labels from the client in the Request context). If that doesn't work, we may have to live with the random interface names but it is not going to be pretty ... that will require more surrounding effort to sort things out.

Thanks again.

[glazychev-art]

For the custom interface name, you are absolutely right - I think this can be done quite simply. I can briefly explain: For the NSE (for example cmd-nse-icmp-responder) we set the interface name here - https://github.com/networkservicemesh/cmd-nse-icmp-responder/blob/main/main.go#L221 And here we can catch the Request - https://github.com/networkservicemesh/sdk/blob/main/pkg/networkservice/common/mechanisms/kernel/server.go#L49 - and call mechanism.SetInterfaceName(). You can try to get the interface name from the labels.

By the way, as you can see, this element has an option WithInterfaceName that sets interfaceName. But I don't think it will work for multiple clients because interfaceName is "static" (there will be an error when creating the second interface).

[bilgehan-erman]

That is good to know. Thank you @glazychev-art!

[glazychev-art]

Thanks for reporting! So, we have the following problems:

1. Custom IP address for NSC and NSE
2. Custom interface name for NSE
3. Grouping CIDR prefixes

Is that correct?

@bilgehan-erman Could you also tell a little more about your use case?

[bilgehan-erman]

Switching to ETHERNET PAYLOAD rather than using IP will relax the addressing part of the problem. As was mentioned earlier, it seems like we will be able to set a custom interface name even at the NSE side -- catching the Request, etc. (However, we will need to inspect and understand the static interface name declaration aspect you had mentioned.)

Unfortunately, we could not get to the point to try all that out. We've been hopelessly stuck on an rpc authorization problem the NSC keeps getting from monitorConnection. We weren't been able understand the reason for the error or find a way to work around it. I've attached a snipped of the NSC log, but I'll open a separate Issue to explain it with full context.

Regarding the use case ... What we are trying out is a simple layout but a little off the typical intended track of the NSM model. It is like the Topology Example that was in the old archived NSM. The topology examples are usually implemented with distinct NSC and NSE nodes or with sidecars. What are trying to do is to strip it down to a generic pod that functions as both NSC and NSE together with all other processes in the pod.

nsc-rpc-auth-error.txt

[edwarnicke]

@bilgehan-erman

However, as you may agree, user guide-type documentation to help with advanced SDK use will take some time to catch up.

You are correct, and your patience is appreciated.

From your diagram it looks like you want to set the IP address from the NSC side rather than the NSE side. Is that correct? If so, could you help me understand why ? And also how you plan to have multiple NSCs connecting to your NSE not have IP address collisions without having to manually manage each of their Pod specs?

Please note: I am constantly finding folks who have interesting needs I would have never anticipated. Understanding the underlying why helps us get to a more elegant solution :)

[bilgehan-erman]

The motivation is to build a pre-set topology. What we are trying out is a little outside of the main NSM model where NSE is providing a specific network service and all the surrounding conveniences that goes with it. Even though the full service automation capabilities may not be used, NSM still provides a powerful framework to create an overlay network. Primary benefit of this may be more for simulation scenarios rather than deployment considerations. In the context of what we are trying out, the NSE’s role is just to be a routing hub. The “services” (such as routing) are provided by external entities in the same container/pod. Therefore, each node’s NSC can connect to any other node’s NSE. If the NSEs are modeled as routers, then some of the NSC-NSE links are expected to be in different subnets. IP address assignment doesn’t have to be by NSC (that was just an implementation related convenience), however, unless the NSE incorporates the notion of IP-groups and their conditional allocation, then the desired model falls apart.

But now since the payload can be set to ETHERNET, the IP related problem goes away. Rather than setting the IPs in the scripts, we can set the IPs on the interfaces.

There have been Topology Examples in the NSM repos. This is just a dynamic, uniform realization of those scenarios with more realistic functional roles — without needing to figure out how to line up NSCs and NSEs to get the desired routing domains.

[edwarnicke]

@bilgehan-erman Ah.. got it :)

What we are trying out is a little outside of the main NSM model where NSE is providing a specific network service and all the surrounding conveniences that goes with it.

I tend to think of it a bit more positively: you are getting creative :)

Primary benefit of this may be more for simulation scenarios rather than deployment considerations.

I could also see this for educational situations (a subset of simulations in my mind). Being able to stand up a 'real network' virtually like this has lots of interesting applications :)

I think you've probably landed on the right solution: Ethernet payload. It sounds like you are really modeling 'routers' connected via ethernet, and Ethernet payload is great for that :)

Are you satisfied with the 'Ethernet' Payload solution?

Do you have any interest in contributing a Topology example like the one you are building out?

Please note, our examples are documented in https://github.com/networkservicemesh/deployments-k8s ... so I'm not so much talking about writing code (I noted your SDK comment) but rather about documenting that style of example (which we then compile to go tests and run as part of our CI across various environments).

[bilgehan-erman]

@edwarnicke Thank you for the positive take on the scenario.

Are you satisfied with the 'Ethernet' Payload solution?

I strongly think we will be, but as I'd documented with #7062, we had to stop the work because of the PermissionDenied error we are getting when NSC and NSE operate from within the same container.

Do you have any interest in contributing a Topology example like the one you are building out?

Thank you for the suggestion. Once we are able to sort things out at our end, we will be glad to contribute a clean set of scripts to demonstrate the concept.

[edwarnicke]

@bilgehan-erman Thanks for bringing attention to #7062 . Again: creative approach :)

We'll see if we can get to the bottom of it :)