Add a gRPC based plugin system

[danielnelson]

Feature Request

Proposal:

Add a plugin system, similar to #1717 but using gRPC.

Current behavior:

~Plugins must be compiled into Telegraf directly. The only option for external code is the exec plugin.~ Update: Options for external code are now: inputs.exec, inputs.execd, processors.execd, and outputs.execd.

Desired behavior:

Be able to create plugins that can be compiled and maintained separately from Telegraf, and have improved performance with support for event driven ServiceInput's.

Use case:

Some plugins cannot be submitted to, is not general purpose enough for, or for other reasons cannot be included in the main Telegraf build. Allowing the plugin to be compiled separately and loaded at runtime into any supporting Telegraf version will reduce the overhead of maintaining these plugins.

[kerams]

Do you eventually want to move all current "plugins" out so as to keep the binary size to minimum and let users manually drop in what they need, a la the config directory?

[phemmer]

2 questions to start off with, more to follow depending on the answers:

1. Does this mean that each plugin is going to have to spin up an HTTP/2 server (to be able to receive calls from telegraf), as well as telegraf running an HTTP/2 server itself (for service inputs).
2. Will telegraf launch the plugins itself, or will they be standalone services that the admin has to manage?

[danielnelson]

We would probably move some of the plugins into an optional package, perhaps on Debian we would have multiple packages: telegraf.deb and telegraf-extras.deb.

I think that only the gRPC server needs to run a HTTP/2 server while the client just makes HTTP/2 calls. Due to the size overhead, we probably will want to allow including multiple plugins to be in each gRPC process.

I would sure like it if Telegraf launched the plugins and managed their lifecycles itself.

We could also investigate basing it on https://github.com/hashicorp/go-plugin

[phemmer]

I think that only the gRPC server needs to run a HTTP/2 server while the client just makes HTTP/2 calls.

Do you mean that telegraf isn't going to do the scheduled polling of each plugin? It'll be up to the plugin to push its metrics on the configured interval?

[danielnelson]

That's a good point, we would want to trigger the polling. It seems like switching around the server and client roles wouldn't entirely help us either because service plugins need to send on their own schedule.

[jeanlucmongrain]

there is a standard API call for healthcheck https://github.com/grpc/grpc/blob/master/doc/health-checking.md

[m82labs]

Do we need to "trigger" the polling? Or could the plugins manage it themselves and just check in with telegraf on start to get the polling interval? I really like the general idea of this method of managing plugins.

[danielnelson]

I don't see why the plugin couldn't handle the interval itself. However, one goal we should have is to keep grpc plugin writing as simple as possible. The more we logic we push off to the client the more complicated it will be, and while we could provide a helper library, this somewhat reduces the ease of writing in unsupported languages. It does seems pretty likely we would need a helper library anyway, so maybe its not a big concern.

[alexpantyukhin]

For now I'm required to send telegraph data on one grpc call. For the current situation I see only one way: write some kind of service which recieves data from telegraph with http output and transform it as I need. Is it good idea or not? Do such services exist already?

[danielnelson]

If you are sending to a common service then perhaps it should be a new output plugin, but if you are sending to a custom service then this would be a good solution. Another option you have is to write a custom output plugin for it and maintain it separately but I don't think it would make sense to try to generalize a gRPC output because of how open ended the protocol is. I'm not aware of any such services that could do this in general.

Overall though your idea is a good solution and since it doesn't require any modification to Telegraf you wouldn't be on the hook for compiling new versions.

[wz2b]

is this still alive? I like this approach ... it matches what Grafana is doing with their plugins. It is also safer - if a plugin crashes, it's its own process. Telegraf can just try to restart it a few times. I think this makes a lot of sense - more agent based than telegraf being a monolith.

[wz2b]

Do you eventually want to move all current "plugins" out so as to keep the binary size to minimum and let users manually drop in what they need, a la the config directory?

I would.

[wz2b]

Do we need to "trigger" the polling? Or could the plugins manage it themselves and just check in with telegraf on start to get the polling interval? I really like the general idea of this method of managing plugins.

you could have telegraf "long poll" for data so the plugin could set its own timing ... that's easier then some kind of callback arrangement.

[ssoroka]

I'm not sure if this is still relevant. There are new execd plugins that essentially allow you to write any kind of plugin you want, compiled (or interpreted) in its own process, apart from Telegraf, including gRPC if you wanted. These interface with Telegraf via Influx line protocol over stdin/stdout.

Does anyone see any incremental value to specifically supporting some sort of gRPC here? Would be interested in hearing from someone with a use case in mind.

[wz2b]

Does anyone see any incremental value to specifically supporting some sort of gRPC here? Would be interested in hearing from someone with a use case in mind.

Maybe.

• Ability to gather data in a more structured format than stdout text
• Ability to implement a more robust plugin lifecycle
• Standardize semantics for passing configuration information to the plugin (from the same toml file)
• Faster to parse than text (it's protobuf) so it should scale up much better

[Hipska]

With the various execd plugins available, only last part required to fulfil this request is a protobuf parser/serializer.

[srebhan]

Related to #9492.