Proposal for plugin support.

[emil-palm]

Background

I was listning to Roman's presentation at DKNog 2023 and was smitten by gNMIc. My employeer (netnod) has been looking for a gNMI system that we can utilize for our ST needs and gNMI fit our purpose fast.

Some context

My first task was to reimplment our statistics solution which currently use a SNMP Exporter for Prometheus to use gNMIc's prometheus output. Just basic prometheus output works really good but we have one more feature that would be good which is to append a customer specific identifier to a metric as a prometheus label. This identifier has to be fetched from a GraphQL API that contains the device + interface mapping to customer id. This is where i got stuck.

To get around this issue i wrote a custom formatter which does the thing i wanted and that was not very complex. But then we come into the issue with maintaining our fork of gNMIc which is something we dont really like to do if we dont really have to, we are a small company with few devs and SRE's. So thats why im proposing this.

I was initially looking at the GoLang plugin module (https://pkg.go.dev/plugin) but this has its drawbacks that it can get abit wonky when you run a plugin compiled by another version of go etc.

And after talking quickly with Roman on Slack about this he pointed me to https://github.com/hashicorp/go-plugin which also does the same thing but utilizing RPC calls over local sockets.

The idea i have the current formatters, actions, inputs and formatters the same way and have them included in the current code base. But also allow users of gNMIc to code their own modules to be loaded at runtime. This way one could keep their highly specific implementations to their modules but without maintaining a separate fork with all of those issues.

I have not started writing any code for plugin support there for its abit vague how its to be implemented. I'm also very happy to develop a working example if the maintainers deem this to be worthy path to pursue and think it has any chance of beeing accepted into mainline. I dont want to waste my time implementing this if there is no intrested from the maintainers for this feature.

[karimra]

Thanks for your interest in gNMIc. The ability to add tags/labels dynamically has been one of the most requested features lately, my plan was to implement something using the existing http action but couldn't come up with something generic enough.

I like the idea of plugin for processors and actions, the Golang plugin has a major advantage over the HC one which is performance. While the HC plugin may be slower but allows you to write the plugin in any language you want. I wouldn't mind seeing both getting implemented with a knob that controls the plugin type. Ideally this would be done using the existing interface definitions but as plugins.

[emil-palm]

Yeah i think its super hard to make a genric solution that fits all, and i think if you want to do such specific things as im trying todo that you need more complex logic or that you might wanna do something other then just HTTP requests (donno what it would be but hey) its better to have a plugin solution.

I agree that Golang plugins are faster but they do give the problems with modules can have problems when complied using different toolchains between the plugin and the main code. I havent done any testing on it.

The biggest thing with the HC one over Golang plugin is the separation of memory so you cant crash the main process with the HC plugins but you can with the Golang ones. But i also thinks that something that can be looked aside.

I also think the issue with compile toolchains can be ignored because i dont like the idea of having "pre-compiled-ready-to-download" plugins in a software like gNMIc i think it should be that you might have to package your own gNMIc packages and have the same toolchain enviroment for your plugins thus having complete control of the software.

If there isnt any big vocal voices saying this is a shit idea within the next few days i will start looking into building a framework for plugins i might start with formatters and then that can be used as a proof of concept and then ported over to the other types of plugins we might wanna have.

[Sparc0]

We would love this feature also.

[spolack]

Others solve that problem by generating labels based on the description node of a object (interface, bgp session). IMO it would be a good idea to have that use-case in mind while designing a plugin ecosystem.

References:

https://www.denog.de/media/DENOG10/day2_1240%20Streaming%20Telemetry.pdf - Slide 8|9 https://github.com/aristanetworks/goarista/tree/master/cmd/ocprometheus#dynamic-label-extraction https://github.com/czerwonk/junos_exporter#dynamic-interface-labels https://github.com/exaring/openconfig-streaming-telemetry-exporter

[hellt]

@spolack I think gnmic already has everything DENOG deck explains in slides 8/9. @karimra can keep me honest here, but exporting XPATH key/vals as prom labels is a feature that was added since the beginning.

[karimra]

Getting labels from XPATH k/v is done automatically when using prometheus or influxdb as outputs.

It's also possible to automatically make any value of type string a label when using prometheus as output by setting strings-as-labels: true. With other outputs the same can be done with the event-to-tag processor.

Labels (or tags) can be added "statically" to any outgoing message using the event-add-tag processor. Or by setting tags under the target config, see here

Labels can also be "extracted" from the received message using regular expressions using event-extract-tag

To add tags from a message to other messages, there are a few options:

• Use the event-value-tag processor. This requires the event messages to be grouped together beforehand. The grouping can be done by using a a cache on the output.
• Use the event-starlark processor, example here

Adding a plugin processor would allow to get values from remote sources. I believe the plugin should simply be a process that takes a list of Event messages as input and returns another list of Event messages that have been modified in some way or another.

[github-actions[bot]]

This issue is stale because it has been open for 12 months with no activity.