The "Blipz" doorhanger is wrongly displayed one day after the last display

[Softvision-MariusComan]

[Affected versions]:

• Firefox 63.0.1 Build ID - 20181030165643
• Firefox 64.0b6 Build ID - 20181101155334

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Prerequisites]:

• Have a new Firefox profile.
• Have the "extensions.webcompat-blipz-experiment.variation" string set to "little-sentiment" in the "about:config" page;
• Have the latest version of the "blipz" add-on (2.0.2) installed from here.

[Steps to reproduce]:

1. Open the browser with the profile from prerequisites.
2. Navigate to "https://www.youtube.com/" and wait ~60 seconds.
3. Click anywhere outside the doorhanger and change your system time to one day into the future.
4. Navigate to "https://www.facebook.com" and wait ~60 seconds again.
5. Observe the behavior.

[Expected result]:

• Nothing happens.

[Actual result]:

• The "Blipz" doorhanger is displayed again.

[Notes]:

• Considering the fact that the doorhanger should be triggered at the same frequency as in the "Blipz v1", a user should see the doorhanger only once at three days (see Blipz: Experiment Design).
• Attached a screen recording of the issue:

[wisniewskit]

Hmm. Why should this be a blocking issue for an experiment?

@tdsmith, @miketaylr, do we care if the user messes with their system time even in the final product?

[Softvision-MariusComan]

Hi Thomas, moving the system time is the closest and the most used method to simulate the user experience in order to test that the doorhanger is displayed at the desired frequency, more exactly once per add-on and no more than once every three days. Considering the above we still consider it a blocker from QA perspective.

[wisniewskit]

@Softvision-MariusComan, I see. May I know why this wasn't considered a blocking issue for v1, but is for v2?

[SoftVision-PaulOiegas]

Hey @wisniewskit I can answer this question as the Team Lead of the Experiments QA team from which Marius is part of. Long story short, this issue wasn't caught on the v1 study. I can raise several reasons why this wasn't caught, but I don't think this is the point.

To be honest the Shield Study testing is a new process for everyone, it's continuously evolving and we steadily grew our default pool of tests for the process. This was probably missed in v1 due to the short experience we had back then with studies. Lately we have encountered multiple studies like Blipz and we diversified our tests that we run, and one of those tests caught this issue in this version.

[wisniewskit]

@SoftVision-PaulOiegas, that's fair enough. I really just wanted to see if there was something I could do to improve the flow here, as it felt like a completely different QA experience from v1 (not that I'm judging the people involved, mind you). But if we're just getting used to what QA for such experiments entails, then I understand.

[wisniewskit]

Actually, I think this is just a misunderstanding, not a bug. I haven't changed the frequency between versions, and it follow this text in the linked document:

Prompting principles:

• Frequency: no more than 1 prompt per day
• Prompts should be 1 every 3 days, no more than 5 per user. Should be evenly distributed
• 1 notification per site max

That is, it will wait at least a day before prompting again, but after 3 days it will prompt again for sure the next time the user hits a domain/site we're interested in (up to five prompts total, and only once per tracked domain/site).

@tdsmith, is that still the intended behavior? Maybe I had it wrong the whole time...

[wisniewskit]

Tim responded on Slack:

I think we talked about changing the frequency [for v2] but we never decided that we had a reason to I'm not sure how to resolve "no more than 1 prompt per day" vs "prompts should be 1 every 3 days, evenly distributed" that language predates my employment i think the current behaviour sounds reasonable and i would default to not changing what we did before in the presence of ambiguity without a clear preference

I don't personally have a clear preference, and I'm open to changing how we do this in the next iteration. But this late in the v2 cycle I don't feel it's worth changing it unless there is a truly compelling reason to do so, and we've all had a chance to discuss it.

@Softvision-MariusComan, what do you think?

[miketaylr]

Manager 2 cents: let's keep it the same.

[Softvision-MariusComan]

This means that the initial text from the PHD document leaves room for interpretation, which leads to this confusion. Because we read the first two rules as ONLY showing the doorhanger 3 days apart.

Based on the discussion, I'll close this issue since it no longer applies.