Closed AlexsJones closed 2 years ago
beeme1mr
commented
2 years ago In the mutating admission webhook section, you mention that the agent could perform a "micro reload". Could you elaborate on that a bit? In that scenario, would it be possible for the agent to be aware of what changed?
AlexsJones
commented
2 years ago In the
mutating admission webhooksection, you mention that the agent could perform a "micro reload". Could you elaborate on that a bit? In that scenario, would it be possible for the agent to be aware of what changed?
I think I actually missed a heading, so I'll fix that and clarify.
The thinking is we need to update the configmap from the controller-manager when there is a CRD change or possibly a labelling update on the deployment ( thinking direct set and standing orders examples ) .
Typically this is just updating a file and that's the end of it. We need the agent to have an open watch on the mounted configmap to force an update of the presentation of flags available to the host container. This could be as simple as a value change or it might be more complex structured data and possibly new endpoints.. the reason that's the case is I don't want to query all global flags if I am just checking a subset
e.g.
/v1/getfruits
/v1/apples
/v1Very large differences in the amount of information pulled back into the host-container to be updated. However, I won't say more on this as the agent design also needs caching consideration and other small nuances.
AlexsJones
commented
2 years ago 
We will reconvene in two weeks after Kubecon to look at the Agent and operator a little more.
AlexsJones
commented
2 years ago We had a meet up at Kubecon that touched on a few key issues that have helped to improve and inform this design pattern.
Given that we want to accommodate vendors and enable them within this ecosystem, we are going to introduce a concept that allows for the Flag Custom Resource to indicate the desire for a remote endpoint point. To that end, it will enable a completely new set of capabilities from the host vendor to interact at the pod level with processes for the cloud-native provider. It serves as a mechanism to instigate a remote fetch capability that would merge or override the local configuration within the custom resource.
It could possibly have some of these types of fields:
remoteFlagProvider:
type: <implementation name>
strategy: merge
credentials:
secret: <secret name> In order to enable host containers to consume the sidecar then there should be multiple protocols to do so.
There was an initial proposal to incorporate the AF_LOCAL/AF_UNIX socket family and within that family, we should decide whether is a need to support SOCK_STREAM and SOCK_DGRAM, I would initially suggest only supporting SOCK_STREAM.
This would enable us to further layer HTTP protocol support on top where required.
The below illustration has been updated also to reflect the current thinking around the initialisation flow of the flagging system.
That said there are some learnings from Istio and concerns around side car overhead - namely around upgrading and maintenance. As such it is worth exploring a pattern for rolling or upgrading sidecars, as the implication is that this will force a deployment rollout due to the change on the deployment object ( and other resource types sts/ds)
AlexsJones
commented
2 years ago @thschue @beeme1mr are you able to transfer this issue to https://github.com/open-feature/research/issues ?
beeme1mr
commented
2 years ago @AlexsJones, thanks for the summary from our meeting. The update to the agent diagram is exactly what I had in mind! I would also agree that SOCK_STREAM would be preferred over SOCK_DGRAM.
A couple other actions items from our sync in Valencia:
@therealmitchconnors, you had some concerns about using a sidecar. Would you be able to add them to this thread when you have a moment?
therealmitchconnors
commented
2 years ago Thanks for the detailed architecture proposal, @AlexsJones ! If I'm reading this right, one of the key motivators for using the sidecar and configmap is to decrease the latency of fetching individual flags, while minimizing the load on the API Server. Do we have some idea of what amount of scale we expect this architecture to handle? For instance, how many application instances might run against a single Flag Store? How frequently would each application need to check a flag? How many flags does the average application have?
Here is an alternate diagram which proposes packing all the logic which gives an application access to a flag into the SDK (note that validation would still need to occur elsewhere). This is a very simple architecture, and I suspect it does not fit with some of our requirements or assumptions about the project, but I'd like to use this diagram as a way of identifying those gaps.
In this model, an informer cache in the Go SDK keeps a cached copy of each flag which may be relevant to this application. When the application asks the SDK for a flag value, CEL evaluation is performed in process, and a result is returned. Because this is part of the Go SDK, all interactions happen at the library level, and no marshaling or sockets are required. Additionally, there is no need for a mutating webhook mounting a configmap, or for a sidecar running in each pod. Like I said, I suspect this is a naive architecture, but I'd like to understand which of our requirements or constraints make this unsuitable.
Thanks in advance for your input!
AlexsJones
commented
2 years ago Thanks for the input and ideas! To address your questions around scale; I am working on the assumption that typically folks would prefer to manage less and make it do more - having perhaps a single feature flag configuration that is shared across a few workloads, whilst that assumption isn't critical, it lets us extrapolate some rough ideas around workloads.
We could ball-park that this N:1 relationship might be in the order of 1-10 workloads with 1-50 pods per workload. This finger in the air guess at least gives us some numbers that give us a pause for thought and force us to have good answers to scalability and overhead.
--
From your design there are a few things that stand out that we should probably think about:
--
I am not sure if it was made clear but in the current design, there is a single configmap per namespace that would be created and managed by the operator; this in turn would be mounted by the desired pods and the flagd service used to evaluate requests from the host container.
Let me know your thoughts, I hope this is useful and I'll try to contribute more comments as we go.
therealmitchconnors
commented
2 years ago Hi @AlexsJones thanks for the clarification! I think this explains pretty well why we should use a local file in lieu of an in-memory k8s cache. Would it make sense to read that file directly from the SDK, rather than using a sidecar?
Also, I seem to recall some surprising nuance to the way FS mounted configmaps get updated in a pod, depending on the way the configmap is loaded. Do we have a preferred mechanism?
beeme1mr
commented
2 years ago Reading the file directly in the SDK would certainly work as well. The main reason a separate process was so intriguing was that we could keep all the flag config and evaluation engine logic in a single place. It would also allow support nearly any runtime as long as they can community over a socket.
I seem to recall some surprising nuance to the way FS mounted configmaps get updated in a pod, depending on the way the configmap is loaded.
It looks like a subPath volume doesn't receive ConfigMap updates. The refresh can also take up to two minutes if using the default configuration. However, it looks like we could trigger an immediate refresh by updating an annotation on the pod.
Can you think of any other nuances we should be aware of? This is an important part of the proposed architecture.
AlexsJones
commented
2 years ago I have been testing this and am seeing updates in less than ten seconds in cluster. As an example I had in the pod a loop running...
Every 2.0s: curl localhost:8080 nginx-deployment-2-58bc59df49-j9bqb: Tue Jun 7 11:22:28 2022
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 763 100 763 0 0 110k 0 --:--:-- --:--
:-- --:--:-- 124k
{ }I then applied a new FeatureFlagConfiguration object
apiVersion: core.openfeature.dev/v1alpha1
kind: FeatureFlagConfiguration
metadata:
name: featureflagconfiguration-sample
spec:
featureFlagSpec: |
{
"newWelcomeMessage": {
"state": "disabled"
},
"hexColor": {
"returnType": "string",
"variants": {
"red": "CC0000",
"green": "00CC00",
"blue": "0000CC",
"yellow": "yellow"
},
"defaultVariant": "blue",
"state": "enabled"
},
"fibAlgo": {
"returnType": "string",
"variants": {
"recursive": "recursive",
"memo": "memo",
"loop": "loop",
"binet": "binet"
},
"defaultVariant": "recursive",
"state": "enabled",
"rules": [
{
"action": {
"variant": "binet"
},
"conditions": [
{
"context": "email",
"op": "ends_with",
"value": "@faas.com"
}
]
}
]
}
}
I saw the change reflected very quickly
Every 2.0s: curl localhost:8080 nginx-deployment-2-58bc59df49-j9bqb: Tue Jun 7 11:22:59 2022
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 763 100 763 0 0 107k 0 --:--:-- --:--
:-- --:--:-- 124k
{
"newWelcomeMessage": {
"state": "disabled"
},
"hexColor": {
"returnType": "string",
"variants": {
"red": "CC0000",
"green": "00CC00",
"blue": "0000CC",
"yellow": "yellow"
},
"defaultVariant": "blue",
"state": "enabled"
},
"fibAlgo": {
"returnType": "string",
"variants": {
"recursive": "recursive",
"memo": "memo",
"loop": "loop",
"binet": "binet"
},
"defaultVariant": "recursive",
"state": "enabled",
"rules": [
{
"action": {
"variant": "binet"
},
"conditions": [
{
"context": "email",I really prefer working with native Kubernetes mechanics rather than against them.
therealmitchconnors
commented
2 years ago That's great news! We should probably test this at scale to see how latency changes. To that end, what is acceptable latency?
Also, as a side note, can you point me to the Definition for the FeatureFlagConfiguration type? I hadn't found one last time I looked...
beeme1mr
commented
2 years ago You can see the flag configuration here. It's specific to flagd and will likely change a bit based on feedback.
toddbaert
commented
2 years ago Ya, it's worth noting this schema isn't ideal. Were using it at the moment because there was no golang parser available for 3.1, which allowed us to avoid having "top level" groupings for types. We'd like to avoid that, but since this is all experimental at this point, we decided to revisit the problem later.
UPDATE: the schema has been updated, but is not final.
Hello folks,
I wanted to share some thoughts on an initial architectural design for the open feature project from the Kubernetes layer. From my initial engagement and involvement with the group, this will be primarily focused on the server-side capability of presenting feature flags to container workloads.
Assumptions
We initially assume that workloads will be some sort of web server, however, I would like to incorporate into the design the support of local AF_UNIX sockets for processes to manage their control flow based on external flags. To that end, we are also basing much of our overarching narrative on the ability to perform some sort of A/B testing; at this point not considering the use of flags to manage logic based on some sort of event activity - horizontal pod autoscaling as an example.
In my following example and illustrations, I have created the latter example of a flask based web server that is presented on the public internet. The flag system however is server-side and should not be programmable from the world wide web. At this time to me, it seems illogical to need to worry about RBAC/ACL and TLS concerns when making a highly sensitive API like this available to the public.
Operator pattern
The typical Operator pattern in this context is appropriate over a stand-alone web service for the previously mentioned reasons but also because of the stateless and scalable nature of the design. For example; a multi-tenant environment is constrained by namespacing and service accounts; something which we would have to overcome if writing a centralised cluster-wide API.
In addition to this, there is a "feel" factor in using this pattern as it is easy to design, manage and augment. I suggest that in conjunction with reconciliation on a primary custom resource, there is also the use of a mutating and validating admission webhook to enable key functionality.
Workflow
In a nutshell, there are two types of a control flow for setting a feature flag. Direct set and standing orders are the pet names for them in this illustration.
These combined should cover the majority of use cases and get the project into a rapidly usable state given they are fairly well-trodden patterns for interacting with services.
Direct set
This pattern uses labels/annotations ( I am undecided currently as there are penalties on the size of the YAML file for storing large histories here ) directly for the open feature labelling system or a reference to an object that does; for example:
The key point here is that the deployment encapsulates where it wants to derive its feature flags from - this makes the issue of the owner referencing and mapping the configuration to the agent trivial.
Once a deployment has been configured with the appropriate annotations/labels then a cluster with OpenFeature Operator running will employ an optional namespace scoped set of webhooks.
validating admission webhook
The purpose of this webhook is to validate any inline JSON that has been configured with a manifest and to provide any additional context that needs to be encoded. This might well be appropriate in a multi-tenant environment to write remarks from an authoritative source.
For example:
mutating admission webhook
The job of this webhook is to run after the validating admission component and inject the OpenFeature agent into the configuration object and complete the required setup to present it to the host container ( the container running the desired workload for feature flagging ) within the pod.
This webhook will deal with configuration such as open port, transport type and configuration path locations ( possibly expanding to backing type such as PVC vs configmap ).
Configuration reloading
In the scenario of a feature flag being altered, the configuration would be modified directly by the controller-manager and the agent would micro reload to present to the host container ( perhaps using the confd workflow ).
Standing orders
This flow is supplementary to writing annotation directly to the deployment and can coexist within the same ecosystem. The idea here is that you have a custom resource that can be programmable but might not necessarily be immediately associated with an application. It would be possible for an OpenFeature agent to fetch from a standing order custom resource rather than it's locally scoped configmap in this scenario.
This is important as it gives a known and persistent custom resource for programming feature flag states but also encourages a subscribe mechanic from a deployment. It might look like the following:
An argument against an API server
I believe there was an initial idea to create a cluster-wide API server, which I would discourage. The problems presented with RBAC/ACL/TLS and other features are not-intractable but they are secondary to wider architectural design issues. I have laid out a few here.
High-scale stateful configuration management
Because having state kept in memory is just a bad idea, especially with many engineers attempting to program against the backend API for OpenFeature, my thinking is we would persist this into state files.
In a large cluster with a centralised server, the thinking is that the configuration state files would need to be persisted to disk, this isn't alone a problem but a single file would create a complex nested object construct which would scale inversely to the number of workloads using open feature flags. This could be compensated by distributing to a state file per workload but at the expense of complexity. Managing orphaned files and alternative formats then compound this issue.
Security
All workloads requiring access to a centralised server will need both access to the kubernetes control plane, the API server and the host overlay network. There are also risks with turning off someone else's feature flags as discussed which means a lot of machinery around security needs to be built and managed here - how do we check service accounts are valid? How do we check the authority level? How do we map a service account to feature flag permissions?
As you can see this design is an invitation to reinvent the wheel.
Performance
Network calls will increase as workloads increase. Neither design ( operator vs api ) are immune to this, however, the distance of calls will be increasing across nodes on the host overlay network unless there is a separate tenant network for calls to the OpenFeature API server.
In addition, when the API server fails or restarts all calls will start timing out to it unless there is behaviour introduced into the agents ( which is completely possible ). However, the remark about a single point of failure holds true.
Let me know your thoughts