Shared Store across multiple plugins.Manager

[mjungsbluth]

What is the underlying problem you're trying to solve?

When using OPA close to the edge, i.e. integrated with an Ingress or API gateway and with sufficiently large data portions, you might want to have either multiple plugins.Manager reuse the same shared bundles or you might want to have on OPA process as a side car that can download multiple discovery documents and attach them to different API rules. This can be compensated by publishing one discovery document and share policies / data across teams but what this enforces is a sort of mono repo for policies and data. Especially with a lot of independent teams this becomes tricky quick. If you however want more team autonomy, you can have the following situation: per application policies with small data sizes attached, shared policies that are for example maintained by security engineers and global data from directories (10MBs). In a canonical mapping that also reflects how commercial control planes work, you'd have one discovery document per application and would used those in one central place. When using OPA as a library you end up having multiple plugins.Managers with a storage.Store each, effectively wasting memory and risking OOM kills.

Describe the ideal solution

What I'd like to have is a Store that can be used across multiple plugins.Managers and that shares memory for the same bundles.

I did a stab at this last year and I think it is possible to make it work without interface changes but I would like to get some input on this before investing time on implementing and testing this:

• The Store is currently agnostic to bundles, i.e. it does not understand this concept completely.
• To pass this information from the bundle plugin where activation and deactivation is happening, the transaction parameters could be used to pass bundle name and version as context to smaller operations
• A decorating Store per plugin.Manager would keep track of which bundles with which revisions it uses. Operations to a delegate Store are transparently prefixed with a bundle revision
• The tricky part is ref-counting and cleanup as unused bundles need to be unused by all plugin.Managers

WDYT?

[ashutosh-narkar]

Hey @mjungsbluth thanks for filing this. Can you please help me understand a little more about the use-case we're trying to solve here.

If you however want more team autonomy, you can have the following situation: per application policies with small data sizes attached, shared policies that are for example maintained by security engineers and global data from directories (10MBs).

From what I understand, we have several OPA instances running and each of them need to be loaded with some policies that are common to all (ie. security team authored) and some that are specific per OPA based on the application its making policy decisions for. Is there a reason the discovery feature with some dynamic configuration will not work in this case? For example, I imagine the service bundle for each OPA pointing to the same bundle source for the common policies and the application specific bundle source would be generated by evaluating the discovery config.

I would think that in-memory sharing between OPA instances as you pointed out would be tricky and we'll have to deal with issues like contention, cleanup etc. So if we can leverage discovery for this use-case then we should look into that.

[mjungsbluth]

Let me try to unpack this a bit:

• We have routes in each k8s Ingress configuration that can be contributed to by the teams owning the application, we have no central control over which rules target which application.
• All Ingress resources in one k8s cluster are consumed by one elastically scaled deployment.
• Due to a design decision in Styra DAS, the lowest grain bundles (systems/application) have overlapping bundle roots, so we cannot mix multiple of those bundles in a single discovery doc
• As the number of rules that need authZ provided by an OPA instance is dynamic, the number of differently configured OPAs (triggered by different discovery documents) is basically unbounded, a problem that would require writing very custom code to start those on a k8s cluster.
• What we are doing at the moment is to embed OPA as a golang library that instantiates a plugin.Manager per route that needs authZ. This works quite well with the only caveat that the biggest memory waste comes from the shared global bundles (user directories for example).

To be clear, I am asking if you'd accept a contribution that makes it possible to share an in-memory store across multiple plugin.Managers and with passing bundle metadata as transaction context for activation / deactivation. Most of the work probably would go into the tests and I'd really like to avoid spending the effort only to discard the PR then.

[ashutosh-narkar]

Thanks for the context @mjungsbluth. Few questions to understand this better.

As the number of rules that need authZ provided by an OPA instance is dynamic, the number of differently configured OPAs (triggered by different discovery documents)

In this case are you saying you have one OPA per route? If yes, then for one application whenever a team owner adds a new route in the Ingres config, you'd spin-up a new OPA instance just for that route? I would have imagined an approach where you have one OPA per application and when new routes are created that OPA would get a new bundle to take into account those routes. But maybe I'm missing something here and/or this is not feasible.

we have no central control over which rules target which application.

Hmm so you as the admin have no visibility whatsoever in routes added to the ingress and hence you have a setup that programmatically spins up a new plugin.Manager basically acting as a new OPA instance. Alternatively, you could have one OPA instance and a new bundle for the new route 🤔

I did a stab at this last year and I think it is possible to make it work without interface changes but I would like to get some input on this before investing time on implementing and testing this.

Do you have a branch somewhere which has your initial changes?

[stale[bot]]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days.