A simplier backend plugin approach

[hubo1016]

Besides #568 there may be simplier ways to provide backend plugin functions.

We can use external binaries (or scripts) to provide backend functions. This is the same approach with kubernetes plugins. The external binaries receives parameters from environment values / commandline arguments /stdin, and write the output to stdout in JSON format.

Then we can add a "external" backend to the backends, which executes external binaries for WatchPrefix and GetValues.

A possible standard:

./plugin watch --prefix <prefix> --waitindex <index> [keys] ...
# OUTPUT: {"waitIndex": <nextIndex>}

./plugin get [keys] ...
# OUTPUT: {"values": <key-value map>}

Binaries should return non-zero when there are errors. More error information could be returned from stderr and / or stdout.

A lot of backends have CLI tools and client SDKs, so some backends can easily be implemented with shell/python scripts. More complex plugins may be implemented with a standalone service, and use the binary to call RPC methods on the service.

[okushchenko]

@hubo1016 Thanks for a suggestion! When I was first investigating a solution for a plugin system there were 2 dominant solutions to the problem: RPC based plugins or exec based plugins. I've chosen the RPC based plugin system because it allows the plugin to be a long-lived process, therefore preventing constant reconnections to the datastore, removing the need to pass a waitIndex, which is not even available in some backends and allowing to use caching in the plugin. Also, the plugin system itself is not a core of Confd, so I thought it would be beneficial to use an existent solution and the https://github.com/hashicorp/go-plugin is the best one in the Golang ecosystem at the moment.

What are your thoughts?

[hubo1016]

@okushchenko An exec-based plugin can also rely on a long-lived process, simply moving the RPC calls to the binary. This gives flexibility to the plugin designer, and remove the complex dependency from confd. Plugin designers can also choose their preferred protocols for RPC (HTTP, JSON-RPC, UNIX socket). And the long-live process can be managed separately, not necessary to be started by confd. And designers are also free to use existing CLI tools if they do not care about reconnecting (some backends may only have HTTP REST APIs, it is rather impossible to use only long-live connections).

Besides watch mode, confd supports -interval too. It is not very meaningful for a backend which does not support watch, to have a long-lived plugin process.

Also an exec-based plugin is more friendly to other languages. For many backends, Go is neither the most "native" language, nor the most familiar language for the plugin designer. Although GRPC supports many languages, it is not easy to learn. On the other side, every programmer can develop an executable in minutes.

Most importantly, go-plugin is far more complex than confd. It might not be suitable to introduce a plugin system so complex into this simple tool. Open source libraries are not free, they need a lot of efforts to trace the developing and issues of the upstream project.

[hubo1016]

@okushchenko I think waitIndex can not be removed currently. There can be multiple template resources for a single confd to process, each has its own watch process, so the WatchPrefix interface must have some parameters to mark the last watch position.

A customized JSON structure can be more useful than the uint64, but at least the plugin could save the position internally, and use a uint64 as an identifier.

[okushchenko]

@hubo1016 I hope that you will see my answer as a way to continue this discussion, your opinion is very valuable. I'll address your points one by one.

go-plugin is far more complex that confd.

That's the whole point of using an external library. Doing raw file io or encryption is complex too, but that's not a point to not use the libraries that do it for you. I see no point in developing even a simple exec plugin solution and supporting it besides confd, that adds no value. go-plugin is well maintained, solid library. It has a lot of common dependencies (the biggest one, gRPC is the same one that is used by etcdv3 backend).

I think waitIndex can not be removed currently.

It can and should be removed. If you write a backend as a long-lived process (goroutine) that is spawned once on a WatchPrefix call and sends updates back on a channel then there is no need to pass the waitIndex around. Also, other backends that have no such checkpointing mechanism can benefit.

On language support and plugin development. To not have a possibility to write a plugin in any language is definitely a downside, but on the other hand, it immensely improves the productivity you have while writing plugins in Go. The whole plugin is an interface implementation with some boilerplate hidden by a library. It makes it easy to write a plugin even for people who have no experience with Go. For example, you don't need to think about the way your plugin should interact with the host and how it should be executed.

Can you suggest a library that implements exec plugins, so I can take a closer look?

[hubo1016]

@okushchenko CNI plugin system https://github.com/containernetworking/cni/blob/master/SPEC.md uses an external binary. It is used by kubernetes. You may notice that the referenced implementations are also written in Go.

Like what is done in CNI, we could also provide a plugin library with a function PluginMain(plugin BackendInterface), so the plugin designer working with Go is able to implement the interface and finish the job.

Use a channel instead of calling WaitPrefix in a loop is a good idea. But it can be converted to / from the old interface, so we do not need to modify all the backends. For example, there is a plugin which returns a channel. When adapting it to the old interface, we start a goroutine to process all the events from the channel, at each time increase an internal counter. When the old interface WatchPrefix is called with a waitIndex, the internal counter is used as a revision. In this way we can use the old interface for the streaming backends. Streaming API is great, but keep the old interface makes the project stable.

It is not really necessary to choose one and give up another. We can have both of them implemented.

[okushchenko]

@hubo1016 Right now there is no plugin system, so changes are not visible externally. It's a good time to apply breaking changes now before the plugin system is introduced and the interface is locked to preserve compatibility. Also, I think that having both plugins systems is too complex.

CNI doesn't look like a plugin system library that can be reused outside the actual CNI use-case. Are there any examples for that?

[hubo1016]

@okushchenko Yes, CNI is not a plugin system library, it is a special purpose plugin spec. But it is not a plugin system library that we are looking for, we are looking for some specical purpose plugin spec just as the ones in CNI. So I think we could consider the pattern.

https://github.com/hashicorp/go-plugin/issues/45 I'm worried about this.

[okushchenko]

@hubo1016 I've double checked the #568 and it works just fine in a Docker container.

You still need to write an implementation of a spec/pattern and I just don't see any value added to Confd project by maintaining this implementation.