Closed neilkakkar closed 1 year ago
cc: @liyiy @alexkim205
You might want to have a look at the OpenFeature
specification www.open-feature.dev
On initialization, the library makes one request:
/api/feature_flag/local_evaluation
to get the flag definitions of all feature flags. This request returns an object with two keys:
flags
, which is a list of flags with their definition.
group_type_mapping
, which is a mapping from group type index to the group type.If a flag has experience continuity enabled, then it's never evaluated locally.
Consider this case today:
local_evaluation
, so now you’re falling back on /decide
to evaluate it.Thus, when local evaluating, get non-active flags as well from /local_evaluation
, and check if the flag is active or not. If not active, return False.
To make it easier for developers to debug when a flag is locally vs remotely evaluated, log this.
/decide
, log that as well.For local evaluation, we expect the client to provide person and group properties alongside any isFeatureEnabled
, getFeatureFlag
call.
This changes the interface from:
def get_feature_flag(self, key, distinct_id, groups={}):
to
def get_feature_flag(self, key, distinct_id, groups={}, person_properties={}, group_properties={}):
For the flag in question, if all required property keys are present, and the operator is not is_not_set
, then the flag is evaluated locally.
In every other case, we fall back to a /decide
API call.
Note: Whenever we fallback to /decide
, we should pass all group_properties
and person_properties
that were called with the get_feature_flag
call to `/decide.
For cohorts, we fall back to /decide
.
If flag evaluation conditions above are met, we then determine if rollout percentage conditions are met. To do this, we generate a hash as follows:
The hash key is
hash_key = f"{feature_flag.key}.{distinctID or groupKey}"
and the value is
hash_val = int(hashlib.sha1(hash_key.encode("utf-8")).hexdigest()[:15], 16)
divided by the largest float, float(0xFFFFFFFFFFFFFFF)
, to give a number between 0 and 1.
That is, the first 16 characters of a sha1 hash, parsed as an integer in base 16.
If this value is less than or equal to the rollout %, then the flag evaluates to True.
For multivariate flags, the value is calculated the same way.
The only two differences are the hash key, and which variant is assigned based on rollout %s.
The hash key is
hash_key = f"{feature_flag.key}.{distinctID or groupKey}variant"
That is, there is a salt added to the key, called variant
.
To determine which variant the flag belongs to, we generate a variant lookup table that looks like so:
value_min = 0
for variant in feature_flag.variants:
value_max = value_min + variant["rollout_percentage"] / 100
lookup_table.append({"value_min": value_min, "value_max": value_max, "key": variant["key"]})
value_min = value_max
then to compute the variant, if hash_value >= value_min and hash_value < value_max
, then it's the right variant.
Since it's easy to make mistakes here, we also want a consistent test suite across all libraries. Here are some expected tests and the subsequent results:
get_feature_flag
and is_feature_enabled
should follow the exact same codepaths. This ensures that we don't have to double the amount of tests we write, testing for both.is_simple_flag
, and try and calculate all flags locally. This can fail whenever the properties we're looking for are not present, which then fall backs to /decide
, failing which, it falls back to the default value.personal_api_key
to call /decide/
, in any case! We only need it to download flag definitions, which is via the api/feature_flag/local_evaluation/
endpoint. Always remember to pass ?token={}
param to this call, which ensures it gets the flags for the right project.$feature_flag_called
is called only once per distinctID per flagThese tests ensure that the hashes are computed correctly. These take the result from the server, and ensure that client libs match these results. Every library must have 2 of these tests, which replicates the server tests here.
/decide
when results are inconclusive: Property type is cohort, or when all properties aren't present.We expect 2 tests here as well.
For each property type, ensure that the results are what we'd expect.
8 tests (one per operator)
Feature flag definitions look something like:
groups: [
{ properties: [], rollout_percentage: k},
{ properties: [], rollout_percentage: k},
]
Here, all groups are OR'ed, while all properties in the arrays are AND'ed. Have a complex test to test all permutations here.
And some special edge cases:
1 test each for simple person and group property matching.
If flag has continuity enabled, never try to calculate locally.
$feature_flag_called
capture events are firedFor long running libraries, the caching we do to check if we've sent a $feature_flag_called
for this user can explode. Thus we need to make sure there's a size limit on this cache.
This event follows atleast once semantics, so it's fine if we send the event again for a single user, so just clear the cache when it reaches a certain size. Python library has this at 100,000 people right now.
To enable client-side libs to not wait for flags to parse, we can have an option to initialise the library with a distinctID, and corresponding flags. To make this seamless though, our server side libs need another method: get_all_flags()
. I think we should add this now!
Spec looks like this:
Only for server-side libraries.
Function signature is: get_all_flags(distinctID, groups, person_properties, group_properties)
, which is very similar to the signature of get_feature_flag
.
It computes locally what it can, and then sends a request to /decide
for the remaining flags.
The final return type is a dict of flag_key and values.
After playing around a bit, I'd like to add a few more things ( and noticed a few more missing things):
It can get very stressful to ensure that your flag is always computed locally (for example, maybe you made an error, updated feature flag definitions, etc. etc.). Specially in latency sensitive applications, adding a backend delay of upto 500ms because you changed a feature flag in PostHog can be disastrous. Thus, I want to give an option to only compute a given flag locally. That is, get_feature_flag(**existing_args, only_evaluate_locally = False)
.
$feature_flag_called
should get a locally_evaluated
property. Also, should be sent not just once per person, but once per person per response value (i.e. if it returned true
, and then false
, we should send another event)
$feature_flag_called
should also pass the groups
property coming to feature flags.
Ensure that poll interval is customisable by client.
And one more questionable thing:
$feature_flag_called
events should be sentDefault results parameter in the spec has been confusing users, specially around when it applies. So:
All of our client and server side libraries should support experience continuity and multivariate feature flags. This is the spec for functions we'd want to see:
Client Side Spec
A mechanism to send a POST request to
/decide/?v=2
with the following data:The response will contain an object at
response['featureFlags']
with the flags that are enabled for this user, and the variant value, if any. The distinct ID should be already available from existing methods implemented in this lib. The response should be cached and refreshed periodically (for simple flags only. Doesn't apply (yet) to multivariate support.Experience continuity: New feature to "persist flag across authentication steps"
Whenever an
$identify
call is made for the first time for an anonymous user, we also want to send$anon_distinct_id
as part of the decide request. See the posthog-js implementationA method
getFeatureFlag
This method takes a feature flag key as an input and returns the variant value if multivariate flag, or otherwise a boolean indicating if the given flag is on or off for the user based on the response from
/decide
. See posthog-js. Whenever this method is called, we also send a$feature_flag_called
event.A method
isFeatureEnabled
This method takes a feature flag key as an input and returns a boolean indicating if the given flag is on or off for the user based on the response from/decide
. See posthog-js. Whenever this method is called, we also send a$feature_flag_called
event.Set feature flag values on capture Whenever an event is captured, the library also sets the following two properties on every event:
$feature/<feature-flag-name>
, for each feature flag, with the value as the output ofgetFeatureFlag
.$active_feature_flags
, which is a list of all active feature flags.Server Side Spec
On the server-side, the template to follow is
posthog-python
. The biggest difference is that server side libraries are stateless. We don't store distinctIDs, nor feature flags, because the same instance is often called for multiple users, depending on what requests are coming in.Requests to
/decide/?v=2
to get feature flags, the API contract is the same.No experience continuity, since there's no anon_distinct_ids
A method
getFeatureFlag
, same as client side libraries, which also sends a$feature_flag_called
event.A method
isFeatureEnabled
, same as client side libraries, which also sends a$feature_flag_called
event.An option
send_feature_flags
on capture calls, which gets all feature flags for a given distinct ID, and adds them to the event, just like in client side libraries. See this PRNew addition: See comment https://github.com/PostHog/posthog/issues/10459#issuecomment-1197922172