Replace twitter lookup with medialist db mvp

[olizilla]

Pull out twitter api sync code in favour or a separate app and db for pulling and caching contact info from full contact api.

The app will be the starting point for the medialist db.

[olizilla]

FullContact API is restrictive, expensive, and not 100% reliable, so we're going to pause on integrating that any further and simply tidy up and port our existing twitter api code to https://github.com/Medialist/influence

[olizilla]

Currently, we have an observer to do a twitter lookup when a contact is added or changed and has a new twitter screenName. https://github.com/Medialist/medialist-app2/blob/ab238c331ab2a6eff897a9cff2049aa90ed9f862/imports/api/contacts/server/contacts.js#L5-L16

When we find a matching user on twitter, we grab the twitterId as this remains stable even if the user changes their screenName, but the code only ever uses the screenName. A simple improvement is to use the twitterId where available.

[olizilla]

Over in contact-task, we have a contact lookup queue, that can lookup 100 twitter users at a time, every 60s https://github.com/Medialist/medialist-app2/blob/ab238c331ab2a6eff897a9cff2049aa90ed9f862/imports/api/twitter-users/server/contacts-task.js#L16-L45

This uses getTwitterIdentifiers to pick either the twitterId or the screenName, which is good, but it'll only ever find the users first twitter handle if it they have more than one, which isn't a big problem, but worth noting. It's not clear how we'd represent multiple twitter bios in the current UI.

It exports a ContactTask object that lets us explicitly push contactIds onto the lookup queue, and another to lookup all of the contacts with a twitter, once per day: https://github.com/Medialist/medialist-app2/blob/ab238c331ab2a6eff897a9cff2049aa90ed9f862/imports/api/twitter-users/server/contacts-task.js#L100-L115

We used to explicitly schedule lookups during contact import, but now we only ever call ContactTask.periodicallyUpdate() over here https://github.com/Medialist/medialist-app2/blob/ab238c331ab2a6eff897a9cff2049aa90ed9f862/imports/startup/server/contacts-task.js#L4-L10

Which explains why there was a long delay on finding the twitter info for some users, though I'd still expect the observer to trigger a lookup when a contact was created or merged during import.

[olizilla]

We store all the twitter user docs in the TwitterUsers collection, and there is an observer that'll update a Contact that we have previously found a twitterId for with changes to the avatar or screenName

https://github.com/Medialist/medialist-app2/blob/ab238c331ab2a6eff897a9cff2049aa90ed9f862/imports/api/twitter-users/server/twitter-users.js#L8-L32

Where it finds an avatar image has changed, it edits the new url to point at the bigger version of the avatar as described in the twitter api docs here

Of note that image is 73x73px which we should take into account when designing the UI. Also of note, we're not checking for users that have a default avatar, their avatar url will be set but will point to the current twitter default avatar url

Some users may not have uploaded a profile image. Users who have not uploaded a profile image can be identified by the default_profile_image field of their user object having a true value. The profile_image_url and profile_image_url_https URLs provided for users in this case will indicate Twitter’s default profile photo, which is https://abs.twimg.com/sticky/default_profile_images/default_profile_normal.png.

[olizilla]

Requirements

• When creating a contact manually, lookup the user on twitter asap and add the users avatar, bio and twitterId to the contact object.

  • Ensure the bigger twitter avatar is used.
  • (Nice to have) upload the avatar to upload care and use that as the avatar instead.

• When updating a contact manually, if the twitter name is changed, look it up on twitter asap, and update the bio, avatar and twitterId as required.

• When importing contacts, schedule all the created contacts with twitter screeNames provided for looking up on the twitter api, in batches of 100.

• Quick https image lookups for the twitter scraper. It currently uses the old https://twitter.com/[screen_name]/profile_image?size=original twitter image lookup, which was convenient, but can fail (e.g cargotech as reported by BM) or return stale images or return html pages instead of images on mobile, and is no longer listed in the api docs.

Questions

What should we do when we find that a twitter user has changed their screenName, bio, or avatar?

Options are:

1. Do nothing for now.
2. Push change to all deployments with a contact with a matching twitterId
3. Notify all deployments that a contact has changed.

We're currently doing option 2 once every 24 hours.

What should we do when we don't find a twitter user for a given screenName?

Right now we just fail. The bio, avatar, and twitterId won't be set on the contact, and the stale / incorrect twitter name will be kept and linked to on twitter.

Might be better to inform the user. We could use the twitter search api with the contact name and offer suggestions.

[OllyGilbert]

@olizilla just digesting this. We should absolutely notify the user of bio changes. We could use a similar UI pattern to notify of avatar and screenName updates too. screenName updates will be infrequent but may be useful in the case of someone getting married.

[OllyGilbert]

As we've discussed, I think Twitter is going to be key in growth hacking our database. I think we can safely say the majority of influencers and journalists are on Twitter, and its occurred to me that the majority of media outlets have twitter "Lists" of all of their journalists. Check this one out: https://twitter.com/FinancialTimes/lists/ft-journalists/members. I think we can start creating our central database with the fields: name, job title, outlets, location sooner than later. Perhaps even hiring someone to curate lists of journalists. We can then let customer's subscribe to "smart" contact lists and autocomplete the above contact fields when user's create a new contact—speeding up data entry. I'm investigating how we can glean email addresses without a problem. I know email verification services like Hunter exist.