This PR does a few things, all related to using a new seed for ensuring a fair distribution for variants.
Update the client spec tests. The commit has been changed to one that has backported the changes related to variant hashing as described in the backgrund section.
Removed the override tests from the client spec tests we run. There is no logic to handle overrides in this SDKs, so those tests would always fail.
Update the way we test variant tests for the client spec (we previously only checked the enabled state, which is not sufficient).
Fix bugs in distribution algorithms for gradual rollout and variant selection (we were using > instead of >=).
Background
After a customer reported that variant distribution seemed skewed we performed some testing and found that since we use the same hash string for both gradual rollout and variant allocation we'd reduced the set of groups we could get to whatever percentage our gradual rollout was set.
Example
Take a gradualRollout of 10%, this will select normalized hashes between 1 and 10, when we then again hash the same string that gave us between 1 and 10, but with modulo 1000 for variants, this will only give us 100 possible groups, instead of the expected 1000.
Fix
Force the normalization to accept a seed, and make sure to use a new seed when normalizing the variant distribution hash.
Worth noting
This will require releasing a new minor version, since we are changing how hashing works on a pre-1.0 package.
(Mostly copied from the same fix for the Go SDK)
What
This PR does a few things, all related to using a new seed for ensuring a fair distribution for variants.
>
instead of>=
).Background
After a customer reported that variant distribution seemed skewed we performed some testing and found that since we use the same hash string for both gradual rollout and variant allocation we'd reduced the set of groups we could get to whatever percentage our gradual rollout was set.
Example
Take a gradualRollout of 10%, this will select normalized hashes between 1 and 10, when we then again hash the same string that gave us between 1 and 10, but with modulo 1000 for variants, this will only give us 100 possible groups, instead of the expected 1000.
Fix
Force the normalization to accept a seed, and make sure to use a new seed when normalizing the variant distribution hash.
Worth noting
This will require releasing a new minor version, since we are changing how hashing works on a pre-1.0 package.