Pass User Emails on to our Marketing Team

johngruen commented 10 years ago

This is the engineering part of #992 @ckarlof @pmclanahan to fill in the details.

ckarlof commented 10 years ago

The idea would be that we subscribe new FxA users to our Firefox newsletter via the Basket API built by the Engagement team.

ckarlof commented 10 years ago

Python client: https://github.com/mozilla/basket-client JS client: https://github.com/mozilla-b2g/gaia/blob/master/apps/communications/ftu/js/basket-client.js

ckarlof commented 10 years ago

@pmclanahan, how do we bypass the double opt-in when we've already verified the user's email? /news/subscribe with optin = Y or N? Do we need to set validated?

pmclanahan commented 10 years ago

I'll have to double check, but I'm pretty sure that you're right. Optin = true. Validated means you're sure the email is real. It bypases DNS record checks.

Sent from my mobile. Please excuse my brevity.

shane-tomlinson commented 10 years ago

Will our TOS/PP agreements need to be updated to allow this?

Two lines from the Privacy Policy:

We do not share this data with others and we use it solely to provide you with Firefox cloud services.

Is marketing material considered a "Firefox cloud service"?

If you request to have your Firefox account deleted, we will no longer retain this information.

Does that mean if a user deletes their Firefox account, marketing will be notified and the user removed from their list?

ckarlof commented 10 years ago

Is marketing material considered a "Firefox cloud service"?

Good question. I don't know. @mikaland2?

Does that mean if a user deletes their Firefox account, marketing will be notified and the user removed from their list?

Yes.

mikaland2 commented 10 years ago

The answer is probably no, b/c the purpose of an email associated with a firefox account (to get services) is different from the purpose of an email associated with marketing (to get newsletters and updates). So if a user deletes the FxA, it will, AFAIK, only delete the data associated with the account. The user has to separately unsubscribe from emails (which can be done at anytime by selecting the unsubscribe option on a Mozilla email. There may be other ways to do this also).

Urmika Devi Legal Counsel Mozilla Corporation 2 Harrison Street San Francisco, CA 94105 Mobile: 202.549.1397

----- Original Message ----- From: "ckarlof" notifications@github.com To: "mozilla/fxa-content-server" fxa-content-server@noreply.github.com Cc: "mikaland2" udevi@mozilla.com Sent: Thursday, April 24, 2014 8:07:35 AM Subject: Re: [fxa-content-server] Pass User Emails on to our Marketing Team (#993)

Is marketing material considered a "Firefox cloud service"?

Good question. I don't know. @mikaland2?

Does that mean if a user deletes their Firefox account, marketing will be notified and the user removed from their list?

Yes.

Reply to this email directly or view it on GitHub: https://github.com/mozilla/fxa-content-server/issues/993#issuecomment-41291173

pdehaan commented 10 years ago

If I deleted my Firefox Cloud Services account, I personally wouldn't expect to be removed from a newsletter as well.

I assume I could sign up for the newsletter even if I didnt have a Firefox Account, so mentally I'd consider those two separate silos. I don't imagine if I deleted my LinkedIn account I'd ever get off their multitude of mailing lists. Or if I deleted my GitHub account, I assume I'd still get the "what's hot in GitHub today" emails.

But I guess the definitive solution would be to add a checkbox to the "Delete account" page where we say "Remove me from the Firefox newsletter" and leave it deselected by default. That gives the user the option of quickly unsubscribing if they successfully rage quit their account (or remain on the list if they so choose).

Just my $0.01 CDN.

mikaland2 commented 10 years ago

We already have email unsubscribe methods, and should stick with the existing methods as much as possible before creating a new one. You will have to check with the engagement team before implementing a second box on opting out of emails.

Urmika Devi Legal Counsel Mozilla Corporation 2 Harrison Street San Francisco, CA 94105 Mobile: 202.549.1397

----- Original Message ----- From: "Peter deHaan" notifications@github.com To: "mozilla/fxa-content-server" fxa-content-server@noreply.github.com Cc: "mikaland2" udevi@mozilla.com Sent: Thursday, April 24, 2014 8:56:52 AM Subject: Re: [fxa-content-server] Pass User Emails on to our Marketing Team (#993)

If I deleted my Firefox Cloud Services account, I personally wouldn't expect to be removed from a newsletter as well.

I assume I could sign up for the newsletter even if I didnt have a Firefox Account, so mentally I'd consider those two separate silos. I don't imagine if I deleted my LinkedIn account I'd ever get off their multitude of mailing lists. Or if I deleted my GitHub account, I assume I'd still get the "what's hot in GitHub today" emails.

But I guess the definitive solution would be to add a checkbox to the "Delete account" page where we say "Remove me from the Firefox newsletter" and leave it deselected by default. That gives the user the option of quickly unsubscribing if they successfully rage quit their account (or remain on the list if they so choose).

Just my $0.01 CDN.

Reply to this email directly or view it on GitHub: https://github.com/mozilla/fxa-content-server/issues/993#issuecomment-41297412

ckarlof commented 10 years ago

From a legal perspective, the answer I'm hearing from @mikaland2 is "no, we don't need to automatically unsubscribe the user if she deletes her FxA".

From a UX and user expectation perspective, it's not as clear cut to me. The discussion here assumes the user understands the distinction between the two. It's obviously easier if we don't unsubscribe on delete, though.

ckarlof commented 10 years ago

@pmclanahan do I understand correctly that anyone can call /user/subscribe with optin=true without an API key or token? This seems odd to me. It implies anyone could sign me up for newsletters without me having to verify my email address.

pmclanahan commented 10 years ago

It turns out that optin=true isn't working at the moment. We're going to be adding it back in shortly. The short answer is: yes, that's true. Until very recently none of the en-US newsletters required a confirmation at all. So when adding this back in we could require an API key. I'll make a note to look into this.

mikaland2 commented 10 years ago

If it doesn't apply to all newsletters, it should (some newsletters do - example is here: http://www.mozilla.org/en-US/newsletter/). The engagement team is responsible for this area. Let me know of any newsletters that you came across that don't have email verification, so we can get that fixed.

----- Original Message ----- From: "Paul McLanahan" notifications@github.com To: "mozilla/fxa-content-server" fxa-content-server@noreply.github.com Cc: "mikaland2" udevi@mozilla.com Sent: Tuesday, May 6, 2014 1:52:53 PM Subject: Re: [fxa-content-server] Pass User Emails on to our Marketing Team (#993)

It turns out that optin=true isn't working at the moment. We're going to be adding it back in shortly. The short answer is: yes, that's true. Until very recently none of the en-US newsletters required a confirmation at all. So when adding this back in we could require an API key. I'll make a note to look into this.

Reply to this email directly or view it on GitHub: https://github.com/mozilla/fxa-content-server/issues/993#issuecomment-42357315

ckarlof commented 10 years ago

Ok. I think that we'll need optin=true support for us to sign FxA users up for engagement emails without requiring them to verify their email twice in a row, which would be weird.

What's the timeline on having that turned back on?

pmclanahan commented 10 years ago

Should be in the next week or so.

Sent from my mobile. Please excuse my brevity.

mikaland2 commented 10 years ago

I just read the full thread here and got the full context (sorry, I didn't realize the initial request is that you want a way for users who create Fxa accounts to easily sign-up for engagement newsletters with minimal extra email verification, since the user just completed that process). Sign-up for engagement emails requires (1) separate opt-in to the Websites Privacy Notice (2) language selection (3) country selection and (4) email verification process. Totally fine if you can remove the last step (since it's repetitive) - so long as the user opts-into receiving engagement emails to the same email address they used for Firefox Account sign up. Is this possible?

ckarlof commented 10 years ago

@mikaland2:

1) Is there a way to unify the cloud services and websites privacy notices? I thought this was the idea of having a unified PN for all our web and cloud components.

2) Is it not sufficient to user the browser language and country as an indication of what to use there?

3) We're doing email verification, so that should be covered.

mikaland2 commented 10 years ago

I made a legal bug to track the conversations on this topic (Bug 1010630).

The websites privacy notice covers different things from the cloud services notice. Websites notice covers what data mozilla receives in connection with users interaction with our websites and communications (like emails, analytics, gifs, etc.). This is different from cloud services, which covers what data mozilla receives from users in connection with their use of cloud services like sync, accounts, etc.

pmclanahan commented 10 years ago

FYI: Bug 1011001 has been fixed and pushed to prod, which enables the optin=Y flag when providing an API Key. Contact me when you're ready to test for a key.

ckarlof commented 10 years ago

Thanks @pmclanahan. This effort is a bit blocked on legal right now.

shane-tomlinson commented 10 years ago

@ckarlof - any word on this?

ckarlof commented 10 years ago

We have no movement on a unified TOS. The options we discussed were to put it in an interstitial after login (not signup) or put it on the email verify link landing page. Either of these would require the user accepting a separate TOS/PN for Mozilla Websites.

Actions:

Get a Basket API key from @pmclanahan
Finalize UX flows. The original flows in #992 (presumably) assumed a unified TOS/PN.

ckarlof commented 10 years ago

If we put it on the email verification landing page, this would probably also require a de-hackifying of the snippets code, because I promised @johngruen we would do that after we added a third thing there. :)

shane-tomlinson commented 10 years ago

If we put it on the email verification landing page, this would probably also require a de-hackifying of the snippets code, because I promised @johngruen we would do that after we added a third thing there. :)

I extracted the marketing-snippet into its own module in #1299 so that we can hack on it in isolation.

pdehaan commented 10 years ago

@shane-tomlinson and @johngruen to collaborate.

ckarlof commented 10 years ago

@shane-tomlinson, some technical considerations:

The Basket API subscribe call requires an API key if we're asking to skip optin. This probably means that subscribing the user needs to be done on the backend.
To respect Basket's abuse mechanism, our "subscribe" proxy should probably only be available to "authenticated" users. An open question is "how authenticated" and what server verifies this authentication.
"How authenticated" will determine at what points in the user flow will the user have the opportunity to opt-in to the newsletter.

ckarlof commented 10 years ago

Let's take them in reverse order: "how authenticated?"

Some possible authentication states:

"Fully authenticated" Proven knowledge of the user's password on a verified account.
Created account, but unverified This means the user has proven knowledge of (or created) a password, but hasn't verified her email yet. Her agent can't get an auth server session token yet, which means she can't do much with any of our servers yet.
Has verified access to email address, but not necessarily proven knowledge of password This is a weird one, because email access currently allows the user to reset the password, which enables fully authenticated access, but this is subtly different than knowledge of the old password, because resetting the password will wipe the sync data. Regardless, if the user proves she can access the email address (e.g., with a confirmation link), we have no way of currently capturing this in an identity assertion.

We can keep things simple at first by requiring the the user to be in a "fully authenticated" state to display the engagement promotion. A simple proposal would be: "If the user clicks the verification link on the same machine that she created her account, we can display the promotion on the link's landing page."

There may be other opportunities, but that's a decent place to start.

ckarlof commented 10 years ago

Next: Where do we proxy the Basket API request?

Looking to the future, we'll probably want to allow the user to manage her newsletter settings from an FxA dashboard.

I need to read the Basket documentation more carefully, but I think there are two ways to manage the subscription settings on a per-user basis:

If we have an API key, we get blanket access and can reference users by their email address.
We can manage Basket tokens on a per-user basis.

I suspect we'll need an API key to skip the opt-in during subscription, so first approach is probably simpler.

Where is this proxy? It could be on the profile server. If the user's agent is "fully authenticated", it can get an Oauth token for the profile server, which would give it access to an API to proxy requests on the user's behalf.

Does it make sense to be on the profile server? If someone else asked me to put this API on the profile server, I'd maybe tell them "no", because the profile server is supposed to be user data that is meaningful across all of our services. It is a setting that's closely associated with the user's account.

Perhaps a "better" idea is to make the Basket API an Oauth relier. There are some issues I'm going to have to chew there on there.

shane-tomlinson commented 10 years ago

A simple proposal would be: "If the user clicks the verification link on the same machine that she created her account, we can display the promotion on the link's landing page."

This sounds like a reasonably light weight first step, though I am wondering if we can do better. What assumptions/restrictions/requirements are we beginning with?

FWIW, there is a TON of discussion in #992. Pasting in some highlights.

From @ckarlof and @pdehaan in https://github.com/mozilla/fxa-content-server/issues/992#issuecomment-41215609:

Would we want it on the "Sign up" page, or more on the "verify your account" page where we know we have a valid email address (since they would have in theory gotten the challenge email and clicked the shiny button)? If the user fat-fingers their email address here, I'd just wonder if we'd be signing a fake address up to a newsletter. Or even worse, we pass along a fake address to the newsletter team and THEY also send a challenge email which also bounces. Pushing the checkbox further down the stack reduces chances it will be seen, but increases the chances of encountering a valid email.

@pdehaan, yes we would actually sign the user up after verification has completed. But I argue we should capture intention at sign up. I suspect the later we push it in the flow, the fewer conversions we'll get.

Also, putting the checkbox on the "Settings" page would be nice if the user missed the checkbox on the /signup page

Profile server, baby. The Basket API uses a token to identify and auth users. Longer term, we can host a page on the accounts portal for FxA users to manage their email subscriptions, and we can shove the Basket API token in the profile server.

From @ckarlof in https://github.com/mozilla/fxa-content-server/issues/992#issuecomment-41222835

@pdehaan, yes we would actually sign the user up after verification has completed.

There are some "details".

1) Where do we store the intention to subscribe? 2) How do we authenticate our proxy to the Basket API?

I like the profile server for this, but that may be a stretch for May 19th. Maybe not.

@seanmonstar's response in https://github.com/mozilla/fxa-content-server/issues/992#issuecomment-41234503

I think that might work. That's 2 weeks after train-12?

1) Up to the content-server, I imagine. Probably storing in local/sessionStorage, and then once verification is complete, contact the lil' ol' profile server with an updated setting. 2) What's Basket?

Would it be fair to think we'd store many of a user's preferences in their Profile? Perhaps under /v1/settings, with a profile:settings and profile:settings:write scopes?

shane-tomlinson commented 10 years ago

"Fully authenticated" Proven knowledge of the user's password on a verified account.

Are we going to store any bits external to basket, either on the profile or auth server, that signifies whether a user is signed up to the newsletter, or whether they have seen the signup snippet? If so, we could use that knowledge to show the opt-in to already signed up users (some 3 million of them!) on the "you are ready!" screen.

If we do not store that information remotely, I am wondering if there is any scheme we could use to locally decide whether to show an opt-in to already existing users. A local only solution would mean some users would see the opt-in > 1 time, on different devices.

shane-tomlinson commented 10 years ago

Does it make sense to be on the profile server? If someone else asked me to put this API on the profile server, I'd maybe tell them "no", because the profile server is supposed to be user data that is meaningful across all of our services. It is a setting that's closely associated with the user's account.

Can we store the bit on the sync server, in the user's Firefox profile?

shane-tomlinson commented 10 years ago

The FxOS Basket JS Client has been moved to: https://github.com/mozilla-b2g/gaia/blob/master/shared/js/basket_client.js

shane-tomlinson commented 10 years ago

@ckarlof, @pmclanahan - I am looking at the list of newsletters available in https://basket.mozilla.org/news/newsletters/, the available newsletters are:

[ 'addon-dev',
  'marketplace',
  'drumbeat',
  'firefox-tips',
  'ambassadors',
  'mobile',
  'student-reps',
  'join-mozilla',
  'firefox-os',
  'firefox-flicks',
  'labs',
  'about-standards',
  'beta',
  'app-dev',
  'affiliates',
  'about-mozilla',
  'mozilla-and-you',
  'addons',
  'os',
  'aurora',
  'mozilla-phone' ]

Are we signing the user up for one of these, or a new one?

shane-tomlinson commented 10 years ago

firefox-tips ?

pmclanahan commented 10 years ago

We should confirm with Jessilyn, but I believe it will be mozilla-and-you which is our primary newsletter, and is actually titled "Firefox and You".

shane-tomlinson commented 10 years ago

@ckarlof, @mikaland2 - are we using http://www.mozilla.org/privacy/ as the link to the privacy policy? This is what is linked to from http://www.mozilla.org/en-US/newsletter/

mikaland2 commented 10 years ago

It should link to: http://www.mozilla.org/en-US/privacy/websites/

We're going to change the name so the context of the document is more clear to users. Instead of being called the "Websites Privacy Notice" it will probably become "Websites, Emails & Cookies Privacy Notice"

I'll file a bug with the webdev team to correct the link from http://www.mozilla.org/en-US/newsletter/

shane-tomlinson commented 10 years ago

Chris and I had a long and rambling discussion yesterday about how we could approach this problem. I am attempting to write down some of that context so that it is not lost forever.

There are four use cases we must consider when talking about email marketing: 1) The current opt-in that occurs post-signup 2) Opt-out - the idea is to sign users up to certain newsletters automatically with the knowledge they can opt-out via the email. 3) an "email list server" - an external server to us that handles a user's email subscriptions. 4) An email page as part of Firefox Accounts (where this would live is TBD)

We tossed around a number of approaches:

1) Make basket an OAuth relier. Some of the current endpoints would need an equivalent OAuth based endpoint. 2) Use a basket API key and proxy opt-in/opt-out requests via the content server, with the caveat this logic could move to another server, possibly the auth server. 3) Create a hybrid server that is an OAuth relier that uses the basket server as a service provider. This server would manage basket subscriptions on the user's behalf.

Each has its own pain points. 1 is additional work for the basket folks. 2 feels gross if not done right. 3 may be "yet another server".

Since 2 and 3 involves newsletter management via external services, they require a way of discovering a user's basket token, if one exists.

One proposed discover mechanism is to have the oauth or profile server be able to query the basket server for the information.

The basic flow for 3 would be something like: As part of the sign in/sign up flow, an OAuth code is generated and sent to the not-quite-basket OAuth relier. The relier requests a token from the oauth server, using the scope basket or similar. Since basket requires a token for the majority of its operations, and the oauth server is in the business of giving out tokens, the OAuth server could query the basket server for the user's token, then return the token to the relier. The relier would take that token, and manage subscriptions on the basket server on the user's behalf. A point to consider is that the user may not have a token on the basket server. A bit funky.

@ckarlof - feel free to edit this and fill in omissions and errors.

ckarlof commented 10 years ago

@shane-tomlinson, we have a different idea on how to do this. The idea is to create endpoints on the Basket API that accept Browser ID assertions. We presented a proposal to the engagement team, and agreed to move forward. Next action is for @warner to drill down on the details of the Basket API changes.

@shane-tomlinson, this approach will let us do everything from the client side (verification link landing page and account dashboard) via BrowserID assertions and no proxies. You can continue to get the front-end UI ready until the Basket API changes are ready. One additional ask from the engagement team is that we only solicit the user to sign up for the newsletter if her preferred language is one that the newsletter supports. You can see that list here, but we should double check with @pmclanahan.

pdehaan commented 10 years ago

@shane-tomlinson, if it helps, I scraped all the countries in the <select> field, ran them through an XML-to-JSON converter, applied some magic, and dumped the pretty JSON in a gist: https://gist.github.com/pdehaan/0e781908fc64cca705b5

UPDATE: And the languages are dumped in a comment at the bottom: https://gist.github.com/pdehaan/0e781908fc64cca705b5#comment-1264897

ckarlof commented 10 years ago

@pdehaan, you passed the job interview, but we need the languages, not the countries.

ckarlof commented 10 years ago

The reason the countries are there is because I think they are constructing locales out of them, e.g., es-es. I'm not sure exactly the reasons for the way it is now with two select boxes, but we should use the available languages as a hint for what actual locales we should enable showing the subscription snippet. I confirmed this was fine with Jess a while back.

pdehaan commented 10 years ago

Well languages are even easier since it's a short list (not that it matters, it's just a script). I updated the gist above with a comment, but I'll inline it here too:

[
  {"id": "Bahasa Indonesia"},
  {"de": "Deutsch"},
  {"en": "English"},
  {"es": "Español"},
  {"fr": "Français"},
  {"pl": "Polski"},
  {"pt": "Português"},
  {"hu": "magyar"},
  {"ru": "Русский"}
]

ckarlof commented 10 years ago

Thanks @pdehaan!

shane-tomlinson commented 10 years ago

If we need a complete country->possible locale list, the CLDR from unicode.org has the info.

pmclanahan commented 10 years ago

There's also the JSON output of basket. We're not constructing locales. We simply store the users country and preferred language. The countries list is constructed with information in product-details which is in turn extracted from the Firefox codebase and is simply "all the countries". The list of languages represents the ones into which the newsletter is translated, and as you can see from my first link above varies per newsletter.

pmclanahan commented 10 years ago

Also product-details has that full country list translated into 85 languages

warner commented 10 years ago

Basket server API request filed in https://bugzilla.mozilla.org/show_bug.cgi?id=1041074 . This will enhance Basket's /news/subscribe API to accept an optional fxa_uid argument (but only when used with an API key that's known to the FxA server). It also adds a new /news/fxa-login API that takes an assertion and returns the already-existing Basket access token for the corresponding account.

@ckarlof and I came up with two steps for using this from the FxA side. In the first (temporary) approach, we do this:

add an fxa-content-server API which accepts an assertion, verifies it, extracts the fxa_uid and email, then turns around and invokes the Basket server's /news/subscribe API, with fxa_uid=, the API key that enables fxa_uid=, and an empty newsletters= argument. This endpoint will not respond to the caller until the Basket request has finished.
change the email-verification landing page to obtain a session token (https://github.com/mozilla/fxa-auth-server/issues/763), create a pubkey, get a signed certificate, create an assertion, then invoke the fxa-content-server API, and wait for it to finish.

This ensures that the user has a Basket server account set up before they see the "done" message. If we decide to automatically sign users up for a newsletter at this time, we can change the fxa-content-server implementation to submit a non-empty newsletters= argument. If we decide to present an opt-in subscription box, we can change the API to accept a flag (or list of newsletters) and pass them on to the Basket server.

Then in the longer-term approach, we:

remove that API, and the landing page code which calls it
arrange for fxa-auth-server to queue data dumps of new verified FxA users for delivery to a new "new FxA user" process somewhere. Maybe SQS or RabbitMQ or just logfiles that get delivered once an hour. For each new account, when (and only when) it finishes verification, we must record (fxa_uid, email).
this "new FxA user" process will hit the same Basket /news/subscribe API as above
the landing-page frontend won't be involved
eventually, new users will be subscribed to a single-shot welcome newsletter which gives them a link to a dashboard page from which they can subscribe to additional newsletters

The dashboard page, when we get around to building it, will:

require FxA login, so the page will get a valid session token
that token allows it to construct an assertion
the page submits the assertion to the Basket /news/fxa-login API and get back a Basket token
the page then uses the Basket APIs to find out what newsletters are currently subscribed, and to add/remove them

Seem reasonable? We want to minimize the coupling between fxa-auth-server and anything else, and we certainly don't want account creation to block on external servers, but it seems like things are a lot cleaner (and more reliable) if we can do this server-to-server instead of going through the browser-side client.

Also, we want to discourage folks from using the fxa-verifiedEmail field in the assertion, since we're trying to plan for accounts that have 0 or 2 such email addresses. By having our own "new FxA user" server extract the email address from the assertion, we'll have more flexibility to remove this field some day.

rfk commented 9 years ago

@ckarlof, has all the useful content and actions from this issue been moved into the bugzilla tracking bug? If so then we should close this one out to avoid splitting context.

ckarlof commented 9 years ago

I think this is indeed stale. Tracking bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1041074

mozilla / fxa-content-server

Pass User Emails on to our Marketing Team #993

I'll have to double check, but I'm pretty sure that you're right. Optin = true. Validated means you're sure the email is real. It bypases DNS record checks.

Should be in the next week or so.