search

mozilla / addons

☂ Umbrella repository for Mozilla Addons ✨
Other
125 stars 41 forks source link

Implement a banner with a link to a survey on select AMO pages #11977

Closed jenniferdavidson closed 6 years ago

jenniferdavidson commented 6 years ago

We'd like to run a survey on AMO to learn more about our visitors.

Locales that use these languages (en-US, de, ru, zh-CN, es, fr,) account for 78% of the traffic to AMO.

More info on Top AMO Languages (thanks Jorge!)

More details:

MeridelW commented 6 years ago

@jenniferdavidson and @pwalm. I'm thinking that 5 minutes might be too big of a time commitment. I recommend we aim for 2-3 minutes. Depending on length, here are the copy options:

Thanks for visiting this site! Please take a few minutes to tell the Firefox team about your experience in a short survey[link].

Thanks for visiting this site! Please take a minute or two to tell the Firefox team about your experience in a short survey[link].

jenniferdavidson commented 6 years ago

@muffinresearch I met with the Strategy & Insights team - they recommended showing the survey on the first page of the user's session on AMO (regardless of what page that is), but no subsequent pages. Would it be possible to do that instead of just having it be persistent on 3 URLs?

jenniferdavidson commented 6 years ago

@muffinresearch another question - can we pass a parameter along with the link that will tell what page they took the survey on? The Strategy & Insights team recommended not to rely on the referer information, but rather pass in a parameter that SurveyGizmo can read. It can be a URL or utm parameter (I'm not sure I know the difference).

jenniferdavidson commented 6 years ago

Here's what the survey link will be: https://qsurvey.mozilla.com/s3/addons-mozilla-org-survey

pwalm commented 6 years ago

Invision proto here: https://mozilla.invisionapp.com/share/BYMGI2CX3P6

muffinresearch commented 6 years ago

@muffinresearch another question - can we pass a parameter along with the link that will tell what page they took the survey on? The Strategy & Insights team recommended not to rely on the referer information, but rather pass in a parameter that SurveyGizmo can read. It can be a URL or utm parameter (I'm not sure I know the difference).

@jenniferdavidson why is the referer header information a problem? A url parameter is more visible and if anything even easier for the user to manipulate. If it was a utm parameter what would that be/represent?

@muffinresearch I met with the Strategy & Insights team - they recommended showing the survey on the first page of the user's session on AMO (regardless of what page that is), but no subsequent pages. Would it be possible to do that instead of just having it be persistent on 3 URLs?

@jenniferdavidson It's possible, but it would make the whole thing more complex though. I'm not sure why we need to do that. Is there more context to the recommendation? I'm assuming you'd want the link to disappear from ever being shown again once the user navigates.

jenniferdavidson commented 6 years ago

@muffinresearch Let me see if I can get @TylerDowner in on this. I'm not sure the technical details of why he's recommending it in this way. I just want to know what page they were on when they took the survey. Since we're asking about the user's journey in the questions, it'll be helpful to put the pieces together.

And the context for the recommendation about putting it on the first page of the user's session is: Concern 1: We are asking about the user journey, and only 18% of users go to the 3 pages. While that exposure will give us plenty of respondents, it omits 80% of users who land on different pages on AMO. This puts us at risk for not answering our research questions of "How did AMO visitors get to AMO?" and "Did they find what they need?".

Concern 2: We don't want to oversample by putting the survey on every single page of AMO. Oversampling is poor practice with respect to designing surveys. It may also annoy / inadvertently interrupt users.

So, the recommendation of just putting it on the first page of the user's session was to land at a sort of compromise that will hopefully help us answer our research questions, without annoying or oversampling.

@pwalm Question for you. Which of the following do you think makes more sense from a user experience perspective?

OR, do you think the experience should be:

TylerDowner commented 6 years ago

So, quick answer, we can't use logic on HTTP referrers, which means you can't use the built in SG result sheets or do the same logic in survey that we would want to otherwise.

We run our surveys as anonymous, which strips HTTP referrers unless there is a REALLY good reason not to

User manipulation isn't much of a concern, if they are going to manipulate the URL variable than we shouldn't be trusting their responses anyway, so we'd just toss them.

pwalm commented 6 years ago

I would pick the second option. The user can always dismiss it if they don't want to see it. I'd rather it be there on the second page they visit (they might want to complete their task before engaging with the survey) than disappear.

jenniferdavidson commented 6 years ago

@muffinresearch Alrighty, let's go with Phil's suggestion (which was your suggestion!).

To summarize (and I'll edit the description on the bug to reflect the changes):

The survey localization is underway (the link will remain the same). The implementation can go on dev anytime, but should not be pushed to prod until the survey itself has been localized (I expect that to be between sometime by/before July 15). I'll let you know when that happens.

We'll work on testing the survey content outside of this bug. In terms of testing the feature on dev, what would you recommend?

muffinresearch commented 6 years ago

@kumar303 would you be able to take a look at getting this setup and behind a config flag.

@jenniferdavidson I don't think the original timeline for this is possible. Even if we are able to get this out this week (12th) (which by itself leaving very little time) there won't be any translations for the link copy until the following push.

muffinresearch commented 6 years ago

@jenniferdavidson thinking about this a bit more, I think if we get this landed as soon as possible we should be able to go live on the 19th - would that work for you? That way it'll give more time to get the patch landed, and it'll be possible to expedite the translations by doing an extraction as soon as it lands.

@kumar303 see above - as soon as this lands it'll probably make sense to run an extraction straight after (unless we're close to a tag) - that way we can give as much time as possible for the l10ners.

kumar303 commented 6 years ago

can we pass in the page the user was on when they took the survey part of a URL parameter.

What should the name of this URL parameter be? As a value, I suppose we could pass in some form of the current URL.

TylerDowner commented 6 years ago

The URL parameter can be anything we like, "source" is a good one :) As long as the values make sense and we can group them than I'm fine with whatever you send in

jenniferdavidson commented 6 years ago

@muffinresearch The 19th works for me! We should have some translations for the actual survey done by the end of the day today (Friday Pacific time).

@kumar303 & @muffinresearch - when do you think we can get this on dev to test it?

kumar303 commented 6 years ago

The final patch is currently up for review. Once it lands on dev I will update this issue.

kumar303 commented 6 years ago

@jenniferdavidson this is live on the dev server now: https://addons-dev.allizom.org/

Please test it out. We will be "freezing" code today (in 30 minutes!) for Thursday's release but it's slushy, meaning we can pick out patches if we need to make last minute changes. The fewer of those the better so that we can be sure it's in good working order.

Here are some notes about my implementation:

kumar303 commented 6 years ago

Also, if there is anything we need to fix, please file new issues. This makes it easier to track the patches needed as we go through our release process.

jenniferdavidson commented 6 years ago

@kumar303 noted! the 30 minutes is within a meeting time I have. So any needed changes would be after the code freeze. Fingers crossed it's ok!

pwalm commented 6 years ago

Everything looks and works on my end. Great work, all. 😄

jenniferdavidson commented 6 years ago

I've checked it from my end too, and it looks like the survey is passing in the source parameter in a way that surveygizmo is picking it up, so that's good! (We can tell what page of AMO they were on when they took the survey)

ioanarusiczki commented 6 years ago

@kumar303 I made some testing on AMO dev and AMO stage with FF61(Win10).

  1. The survey is available only for the mentioned languages en-US, de, ru, zh-CN, es, fr, pt-BR, pl, ja, zh-TW I clicked the survey and it opened in a new tab, I also tried to open it from different pages on AMO. The URLs look like this: ru https://qsurvey.mozilla.com/s3/addons-mozilla-org-survey?source=firefox%2Faddon%2Fwebmail-ad-blocker%2F fr https://qsurvey.mozilla.com/s3/addons-mozilla-org-survey?source=firefox%2F zh-TW https://qsurvey.mozilla.com/s3/addons-mozilla-org-survey?source=firefox%2Fusers%2Fedit en-US https://qsurvey.mozilla.com/s3/addons-mozilla-org-survey?source=firefox%2Faddon%2Fnoscript%2F

  2. I noticed that the survey is automatically displayed on a new FF profile and once it is taken if another user uses the profile the survey will not be shown (log out after the survey was taken with an account then log back in using another). Please let me know if this is the expected behaviour.

ioanarusiczki commented 6 years ago

Based on the today's discussion - the survey is not related with the sign in, it's set by a cookie, I deleted cookies and history and the survey re-appeared. Verified fixed on AMO dev and AMO stage with FF61(Win10&android 8.0).

ioanarusiczki commented 6 years ago

@jenniferdavidson FYI, I've noticed on Android that the survey form has some layout issues.
survey on android

TylerDowner commented 6 years ago

Can we only show this to Desktop users? We can do that logic on the survey side, but probably easier to just not even show mobile users the survey.

Jennifer, perhaps you can clarify, but I'd say Desktop is your main emphasis?

On Fri, Jul 20, 2018 at 3:00 AM ioanarusiczki notifications@github.com wrote:

@jenniferdavidson https://github.com/jenniferdavidson FYI, I've noticed on Android that the survey form has some layout issues. [image: survey on android] https://user-images.githubusercontent.com/33448286/42993601-7273d104-8c14-11e8-8487-c5d98a9ec7d4.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mozilla/addons/issues/11977, or mute the thread https://github.com/notifications/unsubscribe-auth/ACAbTaLsM5LUzxeEdmIkbwuH1Dv_jSh9ks5uIZwlgaJpZM4U6O_B .

-- Tyler Downer Data Analyst, User Advocate & Survey Guru Firefox Strategy & Insights

jenniferdavidson commented 6 years ago

I'm late on this, but we talked about how not many users are taking it from a mobile device. I should have done it for desktop only, but we know where the survey was taken so I can filter out those responses in case they skewed the results because of layout issues.