Migrate all the data from sepcontent platform to contentful

[query-string]

I'm going to create a new branch in the old sepcontent repo and add the new exporter class there. AFAIU – my local DB has all of the articles that we need to export already, so we even wouldn't need to merge this exporter library with master ¯_(ツ)_/¯

[query-string]

Hi guys! Could you please confirm that we have to import all of articles available on the content platform at the moment, not only articles related with Finanzas, Inmigración, Salud, Educación categories? Initially I thought only these four categories should be imported because only they appear on the top bar of content.saberespoder.com. But it looks like we have more categories among 135 active articles ;) It is no sweat at all, just let me know how to handle those extra categories @skatkov @FanaHOVA.

[FanaHOVA]

@query-string most of those articles aren't relevant anymore, but you should ask @etavenn/@stan2020 to make sure, they might want to migrate them over to keep links/search results alive. Does Contentful have any limits on amount of articles?

On Mon, 3 Apr 2017 at 09:35, Alex Timofeev notifications@github.com wrote:

Hi guys! Could you please confirm that we have to import all of articles available on the content platform at the moment, not only articles related with Finanzas, Inmigración, Salud, Educación categories? Initially I thought only these four categories should be imported because only they appear on the top bar of content.saberespoder.com. But it looks like we have more categories among 11799 active articles ;) It is no sweat at all, just let me know how to handle those extra categories @skatkov https://github.com/skatkov @FanaHOVA https://github.com/FanaHOVA.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/saberespoder/sepcontent-v2/issues/7#issuecomment-291069157, or mute the thread https://github.com/notifications/unsubscribe-auth/AGMJPnsO7ZwesbCQjqB20XjCsqQPDRlNks5rsKE6gaJpZM4MYhcC .

[query-string]

@FanaHOVA as far as I know – no, there is no limitation on entries number, only users and content types are limited.

[FanaHOVA]

@query-string sounds good, let's wait on the green light from product team then!

[query-string]

@stan2020 @etavenn, just FYI – there are eight categories associated with active articles ["Impuestos", "Finanzas", "Salud", "Inmigración", "Educación", "Preguntas", "Noticias", "Trabajo"]. I agree with Alessio – once current implementation of content platform website will be deactivated it would be nice to have copies of active articles accessible, even though some of them is not relevant anymore.

[query-string]

Articles data exporter is ready for review 🎉 Though, I would like to discuss some migration issues @skatkov @FanaHOVA:

1. It was not obvious from the beginning, but content platform articles in test database seems to be different from articles in the production database. Thereby, to have relevant articles on Contently export process must be initiated on the production instance.
2. I think this issue related with the test database and will disappear on production, but it looks like some articles don't have assigned images (or they don't accessible on S3).
3. Since I'm still not sure about relevant categories for export, exporter uses all 8 of them ("Impuestos", "Finanzas", "Salud", "Inmigración", "Educación", "Preguntas", "Noticias", "Trabajo"), but only "Finanzas", "Salud", "Inmigración", "Educación" marked as active. It means that articles from inactive categories will appear on the website, but links on these categories will be hidden from the navigation bar.
4. There are some images from S3 inside articles text and I wonder if we want to transfer them to Contentful as well.

[FanaHOVA]

@query-string wooho! I think sepcontent has its own database which isn't updated like we do with with inboundsms/research. Let's run the migration in production. About the images, I think we can leave them there. They might be linked somewhere else too and we might break something accidentally. About the categories I'm not entirely sure, but we will know more once we launch this and product peeps can take a look!

On Tue, 4 Apr 2017 at 16:20, Alex Timofeev notifications@github.com wrote:

Articles data exporter is ready for review 🎉 Though, I would like to discuss some migration issues @skatkov https://github.com/skatkov @FanaHOVA https://github.com/FanaHOVA:

1. It was not obvious from the beginning, but content platform articles in test database seems to be different from articles in the production database. Thereby, to have relevant articles on Contently export process must be initiated on the production instance.
2. I think this issue related with the test database and will disappear on production, but it looks like some articles don't have assigned images (or they don't accessible on S3).
3. Since I'm still not sure about relevant categories for export, exporter uses all 8 of them ("Impuestos", "Finanzas", "Salud", "Inmigración", "Educación", "Preguntas", "Noticias", "Trabajo"), but only "Finanzas", "Salud", "Inmigración", "Educación" marked as active. It means that articles from inactive categories will appear on the website, but links on these categories will be hidden from the navigation bar.
4. There are some images from S3 inside articles text and I wonder if we want to transfer them to Contentful as well.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/saberespoder/sepcontent-v2/issues/7#issuecomment-291514732, or mute the thread https://github.com/notifications/unsubscribe-auth/AGMJPkVCvwt2WeLrCOR8Ea7NNDWURj_oks5rslGKgaJpZM4MYhcC .

[stan2020]

@query-string @FanaHOVA sorry for the slow response! Yes, please do keep all of the articles. The ones that fall outside of our current focus categories still bring in a trickle of search traffic, and we may broaden categories in the future.

Let me know if you have any other questions.

Thanks.

[etavenn]

@query-string so if I understand correctly:

There are 135 articles, but not all of them are under the 4 categories on today's content site. There are 8 categories.

The question is: 1) Which articles should we import 2) Under which categories should those articles be imported

As @stan2020 said, let's import all of them. If the articles without a category HAVE to belong to a category then stick them in "finanzas" category.

[skatkov]

I should correct Alessio and clarify that:

• old sepcontent, inboundsms, research project share SAME database
• Problem with images is that, we don't migrate them to staging instance of S3 - so they are probably missing because of that

regarding which articles to migrate, those smart gentlemen up there ^^ nailed it down properly. it would be easy to delete them from contentful if some of them are wrong. So migrating all of them seems like a proper way.

I'm more concerned about some of HTML tags in articles though :P

[etavenn]

@query-string @skatkov what about existing comments on articles?

Had a few people point out that the new site doesn't support it. Were you thinking of importing them as well?

[skatkov]

@etavenn currently we don't support comments on new platform, so there is nowhere to import.

[etavenn]

@skatkov got it. Is there a plan to add comments section in future iterations?

Got a lot of feedback that the current comments might be worth archiving for import to the new platform.

Reasons listed:

1. user experience - they can see what questions they've asked
2. community validation. Platform seems more reliable/trustworthy if others have used it
3. Users feel more comfortable asking their questions when others have already done so.

[skatkov]

@etavenn no plans.

http://www.niemanlab.org/2015/09/what-happened-after-7-news-sites-got-rid-of-reader-comments/

[etavenn]

@skatkov interesting article.

Here are my thoughts:

1) Our commentors aren't pseudonumous avatars, are they?

2) The news organizations quoted in the article are news organizations. I would argue that we go beyond "news" and actually are more of a "how-to" resource. Our most popular "articles" are "how-to's". Our users aren't simply commenting on the article, but they are actually asking questions that we or the community then answers.

The question then is, what happens if a user has a question after reading the article? How does the user ask it? Will they go to SMS and ask? How do we effectively channel their intent into actual engagement?

[stan2020]

@skatkov @etavenn i've been on the fence about Comments.

one the one hand, as Stas notes, many publishers have dropped their Comments section. we dropped our disqus comment sections at Ranker and never looked back. they were getting less action the more mobile & social our audience went, and we couldn't attribute more than 10% of our search traffic to them (not worth the real estate vs other page elements like related content blocks and ads).

on the other hand, i agree w/Erik that SEP isn't just a publisher, we're also a solutions provider offering direct/personalized advice to individual users. articles are a natural stimulus for questions, and, if you look at the comments on the current site, 99% are very individualized (like dental clinics or english classes in that user's zip code).

instead of restoring a proprietary or disqus-type comment section, what do you think of doing a combination of: a) invitation to text 72237 to get a personalized answer (gets signups), and b) Facebook social plug-in, called Comments (article and facebook dev page). It's probably not as good for SEO (last I checked, Google's crawlers couldn't read inside the fb comments module), but it's very good for social and checks off some of the boxes Erik cited in his original 3 reasons -- showing that we do address article-inspired questions. the downside is that some people won't want to post more personal or embarrassing questions via fb. but i'm hoping that users w/personal or very specific questions (dental clinics near me) will SMS us, while those with more general questions will use fb.

lmk what you think!

[FanaHOVA]

@stan2020 I think the only reason why it'd make sense to add the Facebook Comments would be connecting user accounts to Facebook accounts. If that's not a possibility, we should just replace the comments section with the Ask a Question bar similar to the homepage one. Two pros that I see:

• User gets sent back to their profile, they find out about other stuff that we do
• Questions aren't public, which makes users less worried about posting personal info online.

And other than that, they get a consistent experience and they will always know that to ask something they can use Ask a Question, they won't have to wonder about whether they should comment on the article directly or go to their profile.

[etavenn]

As both @stan2020 and @FanaHOVA mentioned, we could add some sort of functionality that allows the user to ask a question. I lean towards "ask a question" bar @FanaHOVA mentions that takes the user back into the SMS experience. It allows the user to take action immediately without having to take further steps to text in the question.

I wonder if users are less likely to ask a question if they haven't seen others do it. The way to find out is by not adding a comments section and seeing if we get any questions via SMS. If we don't see any questions come our way, we could add a call to action at the end of the article encouraging users to ask a question via the question bar? If that fails, then we could consider adding a comments section like Facebook.

Thoughts?

[skatkov]

I've been reasoning with myself about comments and I think it's not worth it:

• Disqus has huge downsides (read more here -> http://donw.io/post/github-comments/ )
• Migrating comments from old platform to Disqus is close to impossible
• Building our own version of disqus is very time-consuming - spam protection, profanity protection, moderation and etc.
• We actually need to moderate those

With all of these efforts ^^ i think it's more reasonable to push our content to as much social networks as we can and focus on answering there. All the discussion doesn't happen in comments - it happens on reddit, facebook and twitter nowadays.

I was not really thinking about any replacement for comments.

BUT if i need to find a compromise, then i agree with Alessio here. If we would build 'ask a question widget in js, it could be re-used on multiple webistes (not just content platform), we can potentially offer it for webmasters and offer way to for them to monetise (by bringing us new users)

But unfortunately, i don't see immediate need for it right now.

[skatkov]

The best open-source Disqus-like solution i've found back then i was planning things is https://github.com/phusion/juvia

But project is not maintained, it doesn't have all the features we need.

[etavenn]

I think we're all in agreement that a traditional comments section is not needed. If we realize we need one later down the road then we can think about it then.

What I think we also agree on is that users will have questions when they read these articles. We want to make it as easy as possible for them to do so and get answers. @FanaHOVA 's suggestion is promising. I think we should consider that feature and implement in an upcoming release. My vote is to start with something simple. Add this bar at the end of every article. We can then play around with different placements should there be a need for it.

Thoughts?

[stan2020]

Sounds good for launch.

[skatkov]

I think we need to cross post on all social networks too and better track comments/hashtags in social media.

We can totally automate cross posting to different social networks.

[skatkov]

in slack we can start monitoring some hashtags if that is needed, takes 5-10 minutes to setup.

[skatkov]

A lot of companies offer support through twitter. I enjoy it. And I can read other people questions.

[FanaHOVA]

Let's see how things develop, our Twitter only has 700 followers which is really low; I'm wondering if it's just lack of interest or lack of users in our demographic on the platform (maybe @stan2020 has more data about this). Either way every comment should also be recorded as a question by the system, this way both Q&A and users have a way to access all questions easily, one through Staff Tasks and the other through their Preguntas page in the profile.

[stan2020]

Yeah, Twitter isn't very popular with our users, per our digital preferences survey. 85% said they never use it and only 2% use it multiple times per day. Whatsapp, facebook and fb messenger rule the school.

BTW I'm gearing up for another sampling of that survey in the coming weeks. If anyone has any new questions they want to get in there, pls let me know. We can't let the survey get too long, but there's definitely room for a few more and I may remove a couple as well.

[skatkov]

That report could hold more data then it shows ^^

BUT, it feels like your mixing up social networks and chat apps in a lot of cases. It would be grate to have better representation across those categories, like here is list of chat app i found.

I bet there are more social networks out there, besides facebook, twitter and instagram :-Р

[skatkov]

Which of these chat apps do you have on your phone?

• list

Which of these chat apps you use at least once a day?

[etavenn]

@skatkov to your point, the survey does not ask how users communicate with businesses. But the answers that we did get show FB, Whatsapp and SMS as the dominant players. Posting screenshots below that caught my attention in regards to Twitter.

[etavenn]

@skatkov We could ask users about the other chat apps you mentioned. Telegram, Line, WeChat, Weibo, Kik, Kakao are not popular this side of the world and I would assume less so with our users. But we only truly know if we ask them! Viber might be the only exception...

[FanaHOVA]

Yea I agree that Viber might have a little tick simply because it was heavily branded as "call abroad for free" a few years back (I downloaded it too while traveling); the introduction of WhatsApp calls definitely hit their user base though.

@stan2020 I think "If you need support regarding a product from a company, how would you contact them?" and as options: Twitter, Facebook, Email, Phone, Physical Store. It'd be interesting to have this as a rank order question, not as a select one. For example WhatsApp is definitely the #1 for me in terms of family/friends communication, but if I have business questions I always go on Twitter and/or Facebook.