Spire plugins in their own containers

[kfox1111]

Spire plugins currently require being in the same container as the spire service itself so it can fork/exec the plugin. Since all communications happen via grpc, could support be added so that a container orchestrator can launch the plugin in its own container and spire/the plugin can communicate via shared unix socket? This would allow containers for plugins to be shipped by the plugin writers and easily added at runtime instead of having to be munged together into the same image.

[azdagron]

I think this is definitely an interesting feature request. If we did go down this route, we'd have to be careful not to make assumptions about the state of the "plugin" when SPIRE launched. There is also a question about who controls the configuration, e.g. does SPIRE still configure these plugins (i.e. invoke Configure on the Config service) or are they configured separately.

[azdagron]

If configuration isn't something SPIRE is in charge of, then it becomes easier to not make assumptions about the state.

[kfox1111]

I could see it go either way. Having it all configured via settings injected into the container makes it easier on spire.

But, if it works the same way as now, just the binary is not execed directly, then the plugin could work inside and outside of containers with no changes. Which may be beneficial too.

[azdagron]

Some other questions came up during our contributor sync, particularly in how some features currently provided by the plugin SDK would work with these types of plugins, namely:

• Logging: Plugin logs are emitted by SPIRE core's logging facility, which helps debug issues since its easier to correlate with core actions.
• Host Services: Some plugins use services provided by SPIRE core to do their job. We'd probably want to figure out how this would work. One solution could be yamux sessions.

The plugin SDK can hide some of these details, but we need to come up with a plan for them, or limit the functionality of these kind of plugins until we do.

[azdagron]

Another thing to consider. Since SPIRE would no longer be in charge of the lifetime of the plugin, we may need to add some retry behavior around plugin RPCC issuance that we currently haven't needed. For example, if the plugin is updated and gets kicked by the scheduler, there may be temporary blips where it is not available. Some operations SPIRE performs would be impacted by too much latency here....

[azdagron]

We talked and are ok with this feature, but before it can be moved forward, there needs to be answers about:

1. host service support
2. logging (which could be provided by a host service, if we can figure out (1).
3. retrying and other issues related to no longer managing the lifetime of the plugin

I'll go ahead and backlog this issue, since we think its a viable feature, but mark it unscoped until someone can put in some effort to figure out the answers to these questions.