Add a waffle flag to rollout the beta stats pages

[willdurand]

We'd like to rollout the beta stats for more users in the near future (like start with 5-10-25% of our user base). In order to do this, we'll need a waffle flag. This flag will replace the email address check introduced in https://github.com/mozilla/addons/issues/7620.

This flag will also add links to beta stats for unlisted add-ons.

---

For QA:

• the waffle flag is named beta-stats and it is available in the Django admin. By default, admins have access to the beta feature just like users who are eligible to the beta.
• users with a mozilla.com email can still access the beta pages, no matter the flag, but links to the stats dashboard in DevHub won't directly point to the beta and they won't see the message below when browsing the old stats (like here: https://addons-dev.allizom.org/en-US/firefox/addon/some-addon/statistics/):

• regular (authenticated) users will have access to the beta depending on the percent value set on the flag. In -dev, I manually configured it to 50% but the default is 0% as we won't enable this flag right away in prod.
• regular users should see the message above if they land on the old stats (from addons-frontend for example). Otherwise all links in DevHub pointing to the "stats dashboard" should link to the beta directly.
• it should be possible to go back to the old stats from the beta stats
• unlisted add-ons should show stats in beta
• users can force access to the beta by changing the value of the dwf_beta- stats cookie (from False to True)

Also feel free to play with the percent value to test that when it is 0, no one except admins should have fully access to the beta (fully = with the message above because users with a mozilla.com email will always have access to the beta pages via direct URLs).

When set to 99.9%, I think all users will have access to the beta. The cookie mentioned above should be set for regular users.

The most important thing to verify here is that all links to the stats are consistent: when user has access to the beta, all links should directly point to the beta, otherwise all links should point to the old stats.

Note: addons-frontend will point to the old stats.

[willdurand]

@jvillalobos thinking about it, I believe we can ship this flag next week and maybe start to rollout for 5% of the users or something and target 100% on July 2nd. WDYT?

The reason why we'd like to have a flag and a progressive rollout is that we don't know how BigQuery will behave in production. It seems fine but better safe than sorry.

[willdurand]

This flag will also add links to beta stats for unlisted add-ons.

Also, I think users who will be able to see the beta stats will only see the beta stats (and not both versions of the stats). That will make the transition easier.

[jvillalobos]

We want to give developers time to export their old stats, if they want to, so they need a clear date after which they shouldn't expect the old dashboard to work. I see a couple of options: (1) do the rollout, but start later (06/18) to give devs more time, or (2) give developers control over the rollout, maybe by having a switch in the stats page (it should be fine if it's a one way switch, as long as we are clear about it).

[willdurand]

@jvillalobos ha, right. Maybe we could use the flag to enable the beta stats by default and add a link to the "old" stats from the beta pages?

[jvillalobos]

That would work!

[willdurand]

@jvillalobos so I update the messages a bit and we have two messages now:

1. When a user lands on any of the beta stats pages (if the user did not have access to the beta, it would return a 404):

2. When a user who has access to the beta lands on any of the "old" stats pages:

I'd be happy to reword these two messages if needed.. WDYT?

[willdurand]

This flag will also add links to beta stats for unlisted add-ons.

This will be implemented too but we won't have any download stats for unlisted only (empty graphs). What should be do in that case? It's tricky to think in terms of unlisted vs listed add-ons because (and you know it) channels are set on versions and not on add-ons.

[willdurand]

@jvillalobos you can get an idea of what it will look like for a user who gets access to the beta from there: https://addons-dev.allizom.org/en-US/developers/addon/some-addon/edit

The two links to the "stats dashboard" (on this page and in "manage my submission") should point to the beta directly. There is a message explaining a bit and a link to go to the old stats pages.

[jvillalobos]

I'd be happy to reword these two messages if needed.

The messages look good. Just add a period to Take me to the new dashboard.

This will be implemented too but we won't have any download stats for unlisted only (empty graphs). What should be do in that case?

I think it's okay that the download graphs don't just any data in that case. A nicer way to tackle this would be to hide all Download charts when they have no data, which should only happen when you're unlisted.

you can get an idea of what it will look like for a user who gets access to the beta from there

Looks good!

[ioanarusiczki]

@willdurand So far I tested the following scenarios:

waffle is enabled for everyone and percent set to 0%

• A mozilla.com account and a regular account will have access to beta stats for both listed and unlisted add-ons through the links from dev hub. I don't know if that should happen since the percent is 0%.
• The frontend link will direct in all cases to the old stats and the message is displayed as mentioned above.

waffle is enabled for everyone and percent set to 50%

• I used one account and I login from new profiles and then I went to stats. 4/4 times I was able to access the beta stats from the dev hub links.
• I don't see the cookie being available on a new profile. I had it in the storage for a profile but after I cleared all cookies it did not reappear.

As a conclusion, the cookie and percentage needs further investigations.

Things that look alright to me:

• to go back from beta to old stats is possible.
• when the waffle is disabled mozilla.com users and admins still can access beta stats.
• unlisted addons show the beta stats in dev hub when the waffle is enabled.
• the frontend link always directs to the old statistics.

[willdurand]

waffle is enabled for everyone and percent set to 0%

This means there is no flag anymore, basically. The beta feature is the default for everyone. Observations make sense. The frontend won't have a link to the beta in any case because that would be too much work for little value (we won't keep the beta forever, it will become the default in a few weeks)

[willdurand]

waffle is enabled for everyone and percent set to 50%

Same as before because when the flag is enabled for everyone, other parameters aren't used. You should switch the value back to unknown. That will make the flag work for 50% of users. That's what I just did in -stage to make sure things were alright (and they are)

[ioanarusiczki]

@willdurand Thanks, I'll take a look again tomorrow.

[ioanarusiczki]

@willdurand

I tested the regular users for:

value "unknown" and percent is 50.0 -> 2 profiles had the old stats and 2 profiles had the beta stats value "unknown" and percent is 99.99 -> all 4 profiles had the beta stats value "unknown" and percent is 0.0 -> all 4 profiles had the old stats (even if the value of the cookie was "True", probably because I tested previously the 99.99 using the same profiles )

With value "unknown" and percent 50.0 I could force cookie to change from False to True (and back).

[ioanarusiczki]

@willdurand About the cookie, I tested today to see the state while also changing the beta flag percent.

Two scenarios to mention:

First scenario:

1. I created a new profile and login, beta stats flag is 0% - cookie is not available in Storage and the old stats are available.
2. Increased percent to 50% - cookie shows up and it’s value is true, beta stats are available
3. I change the flag back to 0% - old stats are back , but the value of the cookie remains “True”.

I see that the expiration date is / Max-Age:"Sun, 12 Jul 2020 12:45:10 GMT"

Probably that is why the value is still "True". Yet the user can't see the beta stats anymore.

Second scenario:

1. On a new profile, I login and I set the beta flag to 0% - cookie is not available yet in Storage and the old stats are displayed.
2. I changed the beta flag percent to 10% - I get the “False” value in Storage and I still see old stats.

3. I increased it to 99.9% - value is remains “False” and the user has access to old stats.

   In this case the expiration date is "Session" so I think it should go to "True" and show beta stats. I restarted the browser but I still get the "False" value with the old stats.

[willdurand]

Probably that is why the value is still "True". Yet the user can't see the beta stats anymore.

When the flag is 0%, the cookie is not deleted but the beta is unavailable.

In this case the expiration date is "Session" so I think it should go to "True" and show beta stats. I restarted the browser but I still get the "False" value with the old stats.

heh. The cookie expiration date (WAFFLE_MAX_AGE) is too far in the future (1 month) so we'll need to change that :(

So this works as intended (rollout mode). The fact that you got a False value again after restarting the browser is surprising but maybe it's bad luck? Edit (again): the session cookies are not always cleared by FF after restart. This is a weird behavior but it is a FF behavior. Those cookies should be cleared eventually.

[willdurand]

(I deleted a comment where I was explaining things based on a misunderstanding)

Anyway, I think everything works as expected. We're using the waffle rollout mode and it seems to work fine, even if FF's behavior around session cookies is weird.

[ioanarusiczki]

@willdurand Thanks for looking into it. I'll mark this issue verified on AMO dev with FF76(Win10).