How to attach pods to MAC-layer network segments available on only some nodes of a cluster?

I have a cluster where there's a meaningful topological distinction between some of the network segments available to different nodes in the network.

In my case, I'm trying to connect to some external devices via an application-specific custom-ethertype protocol that doesn't get routed between network segments, in a kubernetes cluster with a heterogenous topology -- that is, only some of its nodes have the additional physical interfaces connecting to these network segments, in addition to their main one connected to the common cluster backplane segment.

In simpler toy homogeneous clusters, I can successfully attach pods to local interfaces and speak the required MAC-layer protocol using a definition like:

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: network-5
spec:
  config: '{
      "cniVersion": "1.0.0",
      "type": "macvlan",
      "mode": "vepa",
      "master": "eth5"
    }'

This almost seems to meet my needs for the heterogenous case too, but I'm running into several issues:

There's nothing by default to indicate to the scheduler where a network is or isn't available; it's easy to end up with a permanent failure if a pod gets scheduled for a node that lacks an interface with the same "master" name.
Even when a node does have an interface by that name, that doesn't mean it's attached to the same network segment, so it's easy to end up with a pod attached to the wrong network if you don't specifically rename the system network interfaces.
Marking nodes and pods with taints that correspond to network segment availability solves the permanent failure issue, but makes it a hard scheduling constraint rather than just a performance suggestion.

What I really want is to be able to attach a single pod to network segments that aren't all available in one place. But it feels like I'm probably even missing something about the conceptual model here, since there doesn't seem to be a way to "spell" the concept either, of connecting to NodeA's eth6 and NodeB's eth11.

How do I set up a NetworkAttachmentDefinition to specify an attachment to a network segment only available on some nodes, that allows pods on other nodes to use it.

If there's no such way, is there a way to at least inform the scheduler of the location more automatically (rather than just with my own ad-hoc implementation)?

k8snetworkplumbingwg / multus-cni

How to attach pods to MAC-layer network segments available on only some nodes of a cluster? #1244