search

PostHog / posthog-js-lite

Reimplementation of posthog-js to be as light and modular as possible.
https://posthog.com/docs/libraries
MIT License
70 stars 36 forks source link

$feature_flag_called only reports bootstrap values if bootstrapping is used #279

Closed danbim closed 1 month ago

danbim commented 1 month ago

Bug description

Without bootstrapping, the system works as expected: the client retrieves a feature flag value from the server and reports accordingly, using the $feature_flag_called event with the value previously retrieved from the server.

With bootstrapping, the value reported with the $feature_flag_called event equals the bootstrapped value. There is however no second $feature_flag_called event sent to PostHog containing the actual value later on retrieved from the server. As a result, if you're using bootstrapping in your application you will only ever see the bootstrap values being reported in the usages tab of the PostHog dashboard. AFAIU this will also cause wrong (exposure) numbers in A/B test experiments.

How to reproduce

I have created a "minimal" reproducible example for verification in form of an executable React Native application with activated PostHog debug logging. Please go to https://gitlab.holi.team/danbim/posthog-bug-demo and follow the instructions in the README.

Related sub-libraries

Our real application uses posthog-react-native and sees this problem. We also have a web version of our application which uses posthog-js for which we have not observed (nor validated) this problem.

I have previously contacted PostHog support (reference e776bf96-ec43-4d4f-8ec9-94a61b730592) but after initial back and forth I haven't received an answer for quite some time. To support the effort on your side, I have created the reproducible example.

Additional context

I'm suspecting https://github.com/PostHog/posthog-js-lite/blob/ca271b82ff58377c5118dddb6cbf1a38808c798f/posthog-core/src/index.ts#L1186 to be the root cause of the issue. Once a $feature_flag_called event has been emitted for a given feature flag key, it will never be emitted again. However, I believe that in presence of bootstrapping, this is exactly what should happen. Therefore I guess that resetting flagCallReported after receiving a response from the server could be a potential solution.

Thank you for your bug report – we love squashing them!

marandaneto commented 1 month ago

Hello @danbim This is a known issue, See PRs https://github.com/PostHog/posthog-js-lite/pull/259 and https://github.com/PostHog/posthog-js-lite/pull/260 I am still waiting for @Phanatic and @neilkakkar since they are the ones who should know how this should work, comment here.

danbim commented 1 month ago

Hi @marandaneto , thanks for the response. If I understand the MRs correctly and your message, then this means that the issue I see is caused by bootstrap values overwriting previously persisted (on device) values that were originally retrieved from PostHogs API. Is that correct?

If that's the case I fear there could still be an issue if the device has no values persisted yet, e.g. on fresh app installations. In this case I would guess that the $feature_flag_called event would still be emitted with the bootstrap value and not again within the same app session. But that's theoretical, not verified...

danbim commented 1 month ago

Just to add what (I believe) this bug means to us: we can currently not run A/B tests because the insights generated for the A/B test believe that every user is in the 'control' group while in fact 50% are in a 'test' group. So for us this blocks the usage of the A/B testing feature altogether.

marandaneto commented 1 month ago

@danbim

If I understand the MRs correctly and your message, then this means that the issue I see is caused by bootstrap values overwriting previously persisted (on device) values that were originally retrieved from PostHogs API. Is that correct?

correct

f that's the case I fear there could still be an issue if the device has no values persisted yet, e.g. on fresh app installations. In this case I would guess that the $feature_flag_called event would still be emitted with the bootstrap value and not again within the same app session. But that's theoretical, not verified...

in this case $feature_flag_called would be reported with the bootstrap values, every time that the flag is read/used. Yes, there's some caching to not report the very same flag again. https://github.com/PostHog/posthog-js-lite/blob/ca271b82ff58377c5118dddb6cbf1a38808c798f/posthog-core/src/index.ts#L1186-L1192 Apparently the flagCallReported is never cleared out if the feature flags response changed, do you mean that this is the issue?

danbim commented 1 month ago

Yes that's precisely what I believe could be a (follow-up) issue if not addressed in the MRs right away :)

marandaneto commented 1 month ago

@danbim thanks for the issue, I think https://github.com/PostHog/posthog-js-lite/pull/281 should address the other issue we discussed.

danbim commented 2 weeks ago

@marandaneto We updated our clients last week and defined a first A/B test. I just checked it again for validation and everything seems to be working just fine now for us, i.e. the two fixes seem to work. Cheers, Daniel