search

mozilla / addons

☂ Umbrella repository for Mozilla Addons ✨
Other
127 stars 41 forks source link

Record experiment name and variation as GA dimensions #14625

Closed bobsilverberg closed 2 years ago

bobsilverberg commented 2 years ago

When running experiments with TrafficCop, Bedrock uses GA custom dimensions to record the experiment name and variation, which leads to a fairly simple way of generating a GA report on the results of experiments.

We should do the same, which will make reporting on simple A/B experiments much easier.

For QA:

With this test experiment landed, when a user lands on, or navigates to, a landing page (e.g., https://addons-dev.allizom.org/en-US/firefox/extensions/, https://addons-dev.allizom.org/en-US/firefox/themes/), we should see an enrollment event sent to GA which includes dimension8 and dimension9. 10% of users will not be enrolled, so it it possible that sometimes no enrollment event will be sent, but most of the time one will be sent. After some of these have been generated, GA can be checked to see if the pageView event was recorded with the correct dimension information. This needs to be checked the next day, so it could be coordinated with @bobsilverberg.

ioanarusiczki commented 2 years ago

@bobsilverberg

On AMO dev: I opened addons-dev.allizom.org and navigated on several pages. I've noticed that the new experiment enrollment is available for Extensions and Themes pages only.

enrollment

I also encountered a profile with notInExperiment -> in that case there's no enrollment event or dimension8, dimension9. For the other situation, a-branch -> every time I navigate back to https://addons-dev.allizom.org/en-US/firefox/extensions/ or https://addons-dev.allizom.org/en-US/firefox/themes/ I see the 2 GA events in the Console, dimension8 and dimension9.

There is a scenario when this is not happening, I am on https://addons-dev.allizom.org/en-US/firefox/extensions/ (I see the 2 events in the Console) and after I click on https://addons-dev.allizom.org/en-US/firefox/themes/

I've also noticed this being present on AMO stage.

bobsilverberg commented 2 years ago

On AMO dev: I opened addons-dev.allizom.org and navigated on several pages. I've noticed that the new experiment enrollment is available for Extensions and Themes pages only.

This is expected. It will only happen for the Extensions and Themes pages.

enrollment

I also encountered a profile with notInExperiment -> in that case there's no enrollment event or dimension8, dimension9.

This is expected. 10% of users will end up in the notInExperiment branch, and they should not have dimensions set.

For the other situation, a-branch -> every time I navigate back to addons-dev.allizom.org/en-US/firefox/extensions or addons-dev.allizom.org/en-US/firefox/themes I see the 2 GA events in the Console, dimension8 and dimension9.

This is expected.

There is a scenario when this is not happening, I am on addons-dev.allizom.org/en-US/firefox/extensions (I see the 2 events in the Console) and after I click on addons-dev.allizom.org/en-US/firefox/themes

This is also expected. When you navigate directly between the two pages, the LandingPage component remains mounted, and the code that sets the dimensions is only called when the component is mounted. This is fine because once the dimensions are set, for a given session, they remain in place.

I've also noticed this being present on AMO stage.

Yes, this patch was included in the tag, so everything will also be on stage now.

Please let me know if you have any questions.

ioanarusiczki commented 2 years ago

@bobsilverberg

Ok, thanks, it means it looks good.

About the last part of the description, it's a bit confusing:

After some of these have been generated, GA can be checked to see if the pageView event was recorded with the correct dimension information. This needs to be checked the next day, so it could be coordinated with @bobsilverberg.

Does this require some information from my side? Or can I mark this verified?

bobsilverberg commented 2 years ago

Ok, thanks, it means it looks good.

About the last part of the description, it's a bit confusing:

After some of these have been generated, GA can be checked to see if the pageView event was recorded with the correct dimension information. This needs to be checked the next day, so it could be coordinated with @bobsilverberg.

Does this require some information from my side? Or can I mark this verified?

You can mark it as verified. I will make a note to check GA tomorrow. I assume I will see some activity from both dev and stage?

ioanarusiczki commented 2 years ago

@bobsilverberg Yes, mainly on -dev.