It's observed that while an experiment is running it provides same value for feature flag for different users in a set of 10 users

[harish-suthar]

Describe the bug I created an experiment to provide two different values of a feature flag for example 50% of users should receive variant 1 which is true and the remaining 50% of users should receive variant 2 which is false, but when I started the experiment and used the feature flag with 10 different users all the users were getting the same value for feature flag.

To reproduce Create a feature flag with a rule where if the context attribute has a certain value then it returns the value true to 50% of audience and it returns the value false to the remaining 50% of the audience. Create experiment using this rule And try fetching the feature flag value wile this experiment is running for 10 users.

Expected behavior I expect that out of 10 users at least 3-4 users should have received variant 2 value for feature flag to be able to verify the development work using launch darkly

SDK version 8.0.1

OS/platform iOS 15

Video reference https://user-images.githubusercontent.com/91060750/225291119-9d1ba20b-e299-4933-964d-33e1d7a5250b.mp4

[keelerm84]

Good afternoon! I appreciate your write-up and the video clip as well. Thank you!

In the iOS SDK, flag evaluations do not take place on the device. Rather, when you identify a user with the SDK, the LaunchDarkly APIs are used to evaluate ALL flags against the specified user, and then those values are returned to the mobile device for fast evaluation.

In your application, when you click on a particular user, are you calling the identify method on the SDK? Do you have anything waiting to ensure the new flag evaluations have been received before calling the variation method again?

[harish-suthar]

@keelerm84 Yes when I click on a particular user I do call the identify method on the SDK. Regarding the waiting thing I start feature flag observation on the start experiment button only, so nothing is waiting till the user tap on the Start experiment button.

Do let me know if this is wrong or I am missing something here.

[harish-suthar]

https://user-images.githubusercontent.com/91060750/226269613-ed67861e-d2bc-487c-853c-33a4e284d91e.mp4

@keelerm84 There is one new observation regarding this issue, when I created this experiment the feature flag type was number and kept two variants for it, this time I created a new experiment and feature flag associated with this I kept of value type boolean, and as you can see in the attached video this time the 50%-50% split is working fine.

So my question is, will the 50%-50% split rule only apply to feature flags of boolean value type only?

[keelerm84]

Yes when I click on a particular user I do call the identify method on the SDK.

When you trigger an identify call, the SDK has to notify the LD API and wait for flags to be evaluated again. If your application isn't taking into account the delay between identifying and evaluating a feature flag, then you may see unexpected results. There are multiple ways to account for this.

Regarding the waiting thing I start feature flag observation on the start experiment button only, so nothing is waiting till the user tap on the Start experiment button.

Flag change observers are definitely 1 approach you can take. You don't need to register and deregister them each time someone clicks the Start button though. You can register them once for the lifetime of the SDK and take action from that.

Additionally you can provide a callback to the identify method which will complete when the SDK comes back online.

There is one new observation regarding this issue, when I created this experiment the feature flag type was number and kept two variants for it, this time I created a new experiment and feature flag associated with this I kept of value type boolean, and as you can see in the attached video this time the 50%-50% split is working fine.

So my question is, will the 50%-50% split rule only apply to feature flags of boolean value type only?

From my understanding, it should not matter which type of flag you are evaluating. I did some testing locally and I have a numeric flag that seems to be giving about 50/50 results.

By default, the bucketing strategy is based on the user key. In your sample, are each of the user keys random and different, or are they similar? Experiments have different seeds, so that might explain the behavior difference between the two types of flags.

Can you provide sample user configurations and the details on your flag / experiment configurations so I can try to reproduce this locally? Please include both the numeric and boolean based setup.

You can also provide that information via a support ticket if you would prefer to not post it here.

[github-actions[bot]]

This issue is marked as stale because it has been open for 30 days without activity. Remove the stale label or comment, or this will be closed in 7 days.