Add a read-only data api for external and cluster.local tools

[Cerebus]

This is an initial implementation, but wanted to expose it here for comment and expansion.

The need I had was to dynamically visualize the meshnet topology, and since I already had Grafana deployed I wanted to use the Node Graph panel.

The quick and dirty way to approach this without writing my own complete datasource was to expose the list of topologies (by namespace) at a service API. The fastest way to do that was to just dump the JSON over a REST API, and use the JSON datasource to read and reshape it into something the Node Graph panel can read.

The examples/grafana/ directory has a simple dashboard that massages the TopologyList data and visualizes the result. The datasource simply taps http://meshnet:51112/v1/meshnet. The dashboard variable $namespace selects the namespace from which topos will be read. The "nodes" and "edges" queries provide the two required data frames, using path /topos/${namespace}.

Currently the only thing being visualized is the topology and the Pod status (via containerId; red for no container assigned; green for container assigned). Data is not cached (otherwise the colors don't update right).

A couple screenshots. A 4-node network (in a line, don't ask) at initial deployment: And again, after deleting pod lf1 (but not the topology):

This can be merged as-is, or hidden behind a feature flag, or refactored (see below) depending how you want to approach it.

I can also take this a different way; there's a generic gRPC data source. Using this would be more efficient (grpc-gateway not required) but has a bit more code to implement.

[Cerebus]

Another thing I'd love to be able to do is to put prometheus data on nodes and edges, but not sure how to approach that. Frex, I currently run SNMP on nodes and scrape with prometheus-snmp-exporter. It would be way cool to put SNMP stats on edges.

I'm open to ideas.

[kingshukdev]

Very interesting. Will take a deeper look next week. For the edges, we can indicate the link type - veth/vxlan/grpc, may be by using different edge color. For edges MTU could be another info to indicate. ( I will start working on user configurable MTU soon).

[Cerebus]

I was thinking about relocating to /meshnet/v1beta1 and adding endpoints for /nodes/{namespace} and /edges/{namespace} to return a pre-shaped dataframe so there's less work on the grafana server. Adding link type, MTU, and color IDs would make sense there. I still like having the /topos endpoint with the raw resources b/c it's simpler than going through kube-apiserver, but I don't have a fleshed out use case for that either.

[networkop]

would it make sense to split it into a standalone application instead of having every agent serve this information? I'd imagine a separate single-pod deployment that would just have enough permissions to read topology CRDs. wdyt?

[Cerebus]

A fair point, but that's another thread running and another controller to write, build, and deploy, so I'm hesitant. The Service is going to load balance all the Endpoints, and if it's just a read-only API it doesn't matter which DS instance answers.

OTOH, maybe this would be a good idea, since I still want to put runtime dynamism into meshnet (e.g., add/remove Pod interfaces, change Pod interface state, use tc-netem on Pod interfaces, manage routes, maybe a few other things), and that might be a place to put it. IOW, the controller watches the Topologies and calls an API on the associated DS instance; the DS instance then twiddles with the host netns.

ETA:

Of course, this would mean that the meshnetd DS would have to run privileged, but if it starts taking on the roles of a service mesh, then that's to be expected. Theoretically with this approach the CNI plugin could be made obsolete, but I kinda like having it manage pod creation time. Without it things start looking like NSM, which (IMHO) is too complex for the core use case of emulating an arbitrary network topology as an overlay of point-to-point links.

I could do without dynamic interface creation/deletion; but changing state, setting up routes, and shaping traffic would be very useful. I currently do these things with an injected sidecar and Pod annotations, but it feels clunky and it's more overhead (inotify limits, particularly).

[networkop]

yeah, makes sense. and what's the point of doing this with the GRPC gateway? why not just start a simple http server? in the future it could serve all sorts of data, including prometheus-style metrics.

[kingshukdev]

As far as "shaping up traffic" is concern, this is in my todo list. High level though is, in the topology CR, one can describe what kind of shaping the user wants on the link i.e tc-netem capabilities. As the pods comes up, links with desired tc characteristics are set up by meshnet.

[Cerebus]

yeah, makes sense. and what's the point of doing this with the GRPC gateway? why not just start a simple http server? in the future it could serve all sorts of data, including prometheus-style metrics.

Honestly? B/c I was hacking. Originally I thought I'd need to write a Grafana datasource plugin and call meshnetd as the backend, so the service implementation came first, and adding a gRPC method is trivial. When I poked around in the datasource SDK, that started to look like waay too much trouble. The only gRPC datasource would have been even more work, so I settled on JSON+REST just to get something that works for my internal use.

[Cerebus]

As far as "shaping up traffic" is concern, this is in my todo list. High level though is, in the topology CR, one can describe what kind of shaping the user wants on the link i.e tc-netem capabilities. As the pods comes up, links with desired tc characteristics are set up by meshnet.

That's where I'm headed, but I need to be able to do this during runtime, not just at Pod init. CNI plugins are great but they only execute once. I want to be able to /change/ the link characteristics while the Pod is running. That's what I meant by "dynamism."

[networkop]

have been even more work, so I settled on JSON+REST just to get something that works for my internal use.

Ack. I think making this a pure httpServer then would make it cleaner. Otherwise I think it's a good feature, would be a great addition.

[Cerebus]

The more I think on it, if I move forward with adding tc-netem to meshnetd (still working on internal corp paperwork), I'll have to sort resource partitioning. Since meshnetd runs as a ds, you don't want two instances updating the same resources. This doesn't come up with the CNI plugin b/c it's calling the host-local instance directly, but adding actual reconciliation actions to will require the controller to be aware of this. It's not a hard thing to do and the data is already there in the CR.

Further, for things like metrics, meshnetd needs a Service anyway, so prometheus et.al. can scrape all the instances (it uses SRV record lookups, which are provided by the svc controller).

So if it's already going to be plumbed out, and the instances of the ds are coordinating anyway, there's no reason to run a separate controller just to provide a datasource backend. Whether this is gRPC-gateway or not is debatable, but TBH I like gRPC-gateway b/c it's more flexible; the API is available either way, and we can easily expose additional REST endpoints from the other gRPC services if needed.