search

MozillaFoundation / plan

What the MoFo production teams are working on
https://build.mozillafoundation.org
17 stars 4 forks source link

Evaluate Chat (Mattermost) to prep for 2017 #837

Closed xmatthewx closed 7 years ago

xmatthewx commented 7 years ago

Hi team. It’s time for us to make a choice: double-down on Chat (Mattermost) for 2017, or make a plan to move to a new platform.

We’d like each production team member to weigh in with arguments for Exit vs Double-down.

Let's establish our stance on Chat before we dive into a debate on the alternatives.

DARCI

D: you? A: @simonwex R: @flukeout @kristinashu @sabrinang @taisdesouzalessa @Pomax @gideonthomas @gvn @alanmoo @mmmavis @cadecairos @acabunoc C: @xmatthewx, @hannahkane I: @Lottarao

gvn commented 7 years ago

It's hard for me to make a case without discussing alternatives, but my vote is Exit.

Mattermost feels like a half measure to me. It doesn't have great UX. It doesn't have a mobile client. I think it's, unfortunately, always going to be playing catchup to more established chat clients. Slack is its "living" design document/roadmap, and has significantly more velocity and resources behind it.

Not having an IRC adapter available is also a huge limitation IMO.

If we want to "go where people are" then we should do that (cough, Slack, cough). If we want to be interoperable with MoCo then we should go back to IRC.

cadecairos commented 7 years ago

I'm 100% for sticking with it. In the beginning, things were pretty rough, but as Mattermost progressed through version 3.0 and onwards it's become a much more viable platform for communication. I expect it to just get even better moving forward. Take my opinion with a grain of salt though, I've invested a lot of my time in improving the ecosystem through installers and integrations.

Notable things about mattermost thus far:

cadecairos commented 7 years ago

It doesn't have a mobile client

Yes it does!

Not having an IRC adapter available is also a huge limitation IMO.

It has one of these too :)

I would not recommend migrating to Slack, for the comparable features we've got, we're looking at $4500/year (for ~200 users) versus $50/year (for up to 1000)

edit: We should take into account costs of self-hosting Mattermost too - ~$100/month IIRC

Pomax commented 7 years ago

I can't honestly vote on this without understanding what our current motivation for running and keeping it is, so these are the options as I've been thinking about them:

The initial idea was that it would lower the bar to entry for contributors. That has not materialized, but then we also never planned and executed on making the public aware of this option, so if that is still the idea (with the same methodology): we should stop investing in mattermost and go back to IRC, we had an idea that it did not pan out.

If the motivation for mattermost has changed, for instance to one where our staff has a richer communication medium available, then mattermost is absolutely better than IRC but I'm not sure that there is a reason to "double down"; the rate of improvement in the clients has been fine for that purpose. It's not as polished as Slack, but we have goodwill built up, and features that are missing seem to show up in a matter of months if not weeks.

Finally, if the motivation is to be a test pilot for the rest of Mozilla, to see if we can move the entire org off of IRC, then... I have no idea, that would be a lot of stakeholders who need to be poked with RFC.

The thing I still absolutely don't like about mattermost is that it moved our communication from "one Mozilla, without barrier to entry other than get an IRC client and then you can talk to us" to "well actually, Mozilla isn't just one thing, and you'll need a login before we'll let you even talk to the non-profit side of us", which I still think is a bad idea because it means we drive people away (best case to perhaps tweeting or facebook at us, worst case to just moving on). Best case, the places they do try to then engage us will only be seen by a fraction of our staff, as opposed to someone talking on mattermost, which can be seen by all staff who feel they should stay appraised of the goings-on around specific topics.

I would recommend either going back to IRC, because it frees up the hours we spend on MM to be spent on something else, or we stick with MM because its rate of improvement is high so I don't see a justification for doubling down there - the only place I feel we should double down, irrespective of which course we steer, is in being clear about our use of which medium to our users: explaining to them that "this" is where you engage with us, listed as primary option on all our "contact" pages, and written into all our mailing list and periodic email notices, as well as mentioning it in public discussions where we might want to elicit external participation. This would also mean communicating our availability internally, so that people on the MoCo side know to say "oh yes, you want to go to this URL, which is our mattermost server, and then go to the somethingsomething channel" instead of MoCo only knowing IRC and MoFo only knowing mattermost.

I would vote against any other course of action, like setting up Slack or Discord, mostly because they would swap one medium that is involved in fragmentation of communicating with "Mozilla" and requiring logins for another system with the same properties.

gvn commented 7 years ago

I question whether we should factor cost into this discussion.

And if we are, then shouldn't we factor in the cost of our paid staff contributing code to MM?

Ultimately, I think we should pick the best tool, not the cheapest one...

PS:

Slack for Nonprofits

The Slack for Nonprofits program offers eligible organizations a free upgrade to our Standard plan for teams of up to 250 members. We offer an 85% discount on the Standard plan to eligible teams above that size.

flukeout commented 7 years ago

Q: What does double-down entail? Continue using it, or put in some kind of extended effort to improve it? If we're improving it, what are the plans, features, etc?

I don't think mattermoz has been successful in terms of attracting and engaging outside contributors, so I think we should ditch it unless we have a magic bullet idea for improving that aspect.

Internally, we'll use whatever we decide to use, it's more about buy-in than a feature set. IRC via IRCcloud is basically free and does pretty much all the same stuff we use mattermoz for.

Does chat suck for engagement? I think so, it creates an expectation of an immediate response and I don't think we can provide that most of the time. We're busy and in different timezones.

I don't know if Slack, or another chat-based thing would solve that. I think something like a forum or github issues might be better.

xmatthewx commented 7 years ago

Thanks for all the thoughts. Let's continue to gather input and perspective from the entire team and keep this fairly open and exploratory.

abbycabs commented 7 years ago

I vote Exit with the caveat that I suspect my vote would change if we waited another 6-12months. Mattermost has gotten remarkably better since we started, but not enough for me to vote Double-down in its current state.

I'm only really able to compare this with Gitter and IRC since I haven't organized any programs/events using Slack.

IRC

IRC has a steep learning curve, and despite best efforts writing guides and bots I could never get good adoption from (science) community members or program staff. I think Mattermost has been better at this. Personally, I would rather stay with Mattermost than return to IRC.

Gitter

There was a huge shift in our community when we switched to Gitter. Firstly, program staff actually used it! We put entire mentorship cohorts in a room and watched the community bond and learn in a way we never saw in IRC. Some of the most positive feedback we got from our first round of mentorship was around the peer group community members were able to find in the Gitter chat.

Also, Gitter has a lot of GitHub integration (& uses your Github login). This integrated well with our programs and trainings.

Mattermost

It might be the people, but I haven't seen the same kind of uptake using Mattermost for the latest round of mentorship. I think there's just a little bit more of a barrier that's stopping the kind of community magic I saw in Gitter.

cadecairos commented 7 years ago

I think that if we plan on using something other than IRC to communicate with community, cost per user should be a factor.

I'm not sure if we've even tried passing Chat off as a means to engage outside contributors. How many repos have we linked it in? which sites do we advertise it on? Measuring it's success as a contributor engagement tool seems too soon based on how we've shared it out to people.

I also think we need to get more opinions from non-engineers and designers who've been collaborating with Mattermost this year to get their perspective, I think it'll be a lot different than all of ours, and I'm interested in hearing it.

I also don't believe that IRC+IRCCloud is better for connecting with community. No matter how you connect to an IRC server, it will always feel like a "space for hackers", and while Mozilla Corporation is decidedly a mostly engineering organization, Mozilla Foundation is much the opposite, as is the community of people that we interact with in our work. This being said, applications like Slack and Mattermost definitely feel less "hacker-like". Should we decide to continue with a non-irc communication tool, I think it will reflect better on us to back a community backed tool that we can contribute back to versus a closed silo.

xmatthewx commented 7 years ago

re: Cost

Let's figure out what we need and what we want before we worry too much about costs. (Two weeks of one production staff person's time with overhead is roughly equal to the year of slack as estimated by @cadecairos)

edit: As MoFo infrastructure, we can also budget for cost differently than other production projects.

cadecairos commented 7 years ago

Let's figure out what we need and what we want before we worry too much about costs. (Two weeks of one production staff person's time with overhead is roughly equal to the year of slack as estimated by @cadecairos)

indeed, but that's for a community that's not growing :D

flukeout commented 7 years ago

@cadecairos I didn't suggest IRC is better for external engagement. I think its useful and cheap for internal communications and MoCo is also using it. With that in mind, we can use it and it frees us up to explore something new that isn't necessarily a chat interface for engaging community.

ryanwarsaw commented 7 years ago

My vote is to stay with Mattermost. It is a solution that is scalable, cost effective and capable of growing with Mozilla Foundation. Additionally, the barrier of entry is much lower than that of IRC.

xmatthewx commented 7 years ago

motivation for running and keeping it

It would be good to share a link to original rationale, plan, evaluation if anyone has it.

My personal assumption on the value of any comms platform would be ranked: 1) More effective MoFo with less email. 2) More engagement with Fellows, MozFest Wranglers, and lead Network members. 3) Engagement with contributors.

I'm curious what @simonwex @Lottarao would say in terms of strategic priorities.

gideonthomas commented 7 years ago

tldr; I don't think Mattermost is the right fit, although we need adequate research from people other than the production team to make a proper decision

This was probably just a formality but I feel like the 'C' in the DARCI is missing a lot of people. IIRC, Mattermost was meant to be a solution not just for the Production team, but for MoFo in general as well as Mozilla's network. This includes not only MoFo staff, but also community members from different countries, Mozillians who attend MozFest, the Advocacy team, members of the Leadership network including Hive and Gigabit (for instance, Hive Toronto wont use Mattermost because we continuously change our communication mediums - so we should consider their input whether we decide to keep or leave Mattermost).

Also, I believe it was meant to serve as a scalable platform to facilitate real-time discourse that could be seen by anyone interested and would live beyond the scope of when those discussions took place. This could be between MoFo staff and the community, between members of a single community or between different communities. I believe that this calls for a user-centric solution where we are only a fraction of the users who want to use this.

If that is not the case, I think using Mattermost is fine (basically as an internal communication tool) but then I would suggest switching back to IRC so that "internal" doesn't just cover MoFo, but also includes MoCo who heavily use IRC, so that we keep that chain of communication between the two organizations seamless.

If however that is the case, imo, Mattermost fails to live up to those expectations. MozFest might be a good place to test this theory by marketing the use of Mattermost as a communication medium during the event (I don't know if it's too late for that). One way or the other it might help answer the question if Mattermost is right for us or not.

I think we should conduct a better research endeavor with the above mentioned people that's purely about data gathering and less about suggesting tools. This will also help us understand if our criteria for evaluating Mattermost (and other tools) makes sense or not based on the use cases we discover are important to those users.

xmatthewx commented 7 years ago

What does double-down entail? - @flukeout

Double down? Chexit? Personally, I could make the case for either (I'll post that soonish). I don’t, however, think there’s a good argument for Stay the course. Our attempts to engage and onboard all staff and some community (Hive, MozFest) have not succeeded due in part to platform shortcomings. This platform is mission critical – if we do it, we should commit resources and do it right.

I'd summarize the blockers to community engagement of Hive and MozFest wranglers (based on anecdotal evidence) as: poor interface UX, difficult desktop installation, poor mobile and low-connectivity experience. I'd also add that Mattermost's marketing is weak, making it unlikely that network members would know about it before we asked them to make the leap and join us.

Double down to me would mean that we make a commitment of staff time to improve the project with Mattermost.

xmatthewx commented 7 years ago

although we need adequate research from people other than the production team to make a proper decision - @gideonthomas

Yes. And, if the production team isn't eager to drive this as one way we have strategic impact on our mission, then I would focus that research on other options. edit: That's why I started this thread, rather than drafting a survey for all staff.

edit: by drive this, I mean drive a double-down on Mattermost. Not drive the research.

xmatthewx commented 7 years ago

Thanks everyone for the thoughtful comments. I can see that there is a lot of passion here. I appreciate the friendliness of the debate.

abbycabs commented 7 years ago

I would like to officially throw my vote for Gitter here (in case that wasn't obvious from my response). Not open, but a much nicer experience, and good amount can be done for free.

ryanwarsaw commented 7 years ago

If the comparison is Mattermost vs. Slack then that isn't a fair comparison. These companies offer similar communication solutions but under very different circumstances.

Slack benefits from having plentiful amounts of funding, more man power and most importantly having more time to mature their solution. Mattermost lacks in all of the above, and hasn't been around as long as Slack.

Just look at how much Mattermost has "evolved" if you will over the span we've been using it. This will continue to be the case, and it'll continue to get better and better as time continues.

gideonthomas commented 7 years ago

if the production team isn't eager to drive this as one way we have strategic impact on our mission

Sorry @xmatthewx, I totally phrased that incorrectly. I meant that the research done should include feedback from the production team, but primarily include feedback from Hive, Gigabit, etc. (whether the research is driven by the Production team or by someone else).

cadecairos commented 7 years ago

@ryanwarsaw I agree, and I like Mattermost and what it offers, but we need to focus on finding the right tool for Mozilla and our community. It may not be Mattermost, Slack, IRC, or Gitter.

ryanwarsaw commented 7 years ago

@cadecairos That's completely true, and if we can find a tool that is the right one for Mozilla and the community, I'm all for making the transition. My concern is mainly directional.

Edit: By directional, I mean finding a solution that works for everyone and lowers the barriers for things like contributor introduction. Making sure it's a change in the right direction rather than a step in the wrong direction is very important.

xmatthewx commented 7 years ago

No, I'm sorry @gideonthomas. I think I understood you correctly. I meant drive a double-down on Mattermost, not drive the research. I added an edit to my comment above.

To reiterate: If the production team isn't eager to drive a double-down on Mattermost, then I would focus research with staff, hive, etc on understanding their needs at-large, rather than on mattermost specifically.

taisdesouzalessa commented 7 years ago

Hey team, I am new to the company and probably I don't understand all the complexities involved in such a decision (and I am new to Mattermost as well ). So my perspective is more like an outsider I guess... I think we should think about the goals, as @flukeout mentioned in his post above. Is the main goal: 1. to attract and engage outside contributions? Or 2.to facilitate internal communications? If the primary goal is 1. I think we should do a research on the main contributors we want to attract and see which tools they are already using (lowering the barrier for contribution). Personally, I used Slack and it seems that many people are already using it and they make the onboarding very easy (but I am not sure this is the tool of choice of our target audience). If the primary goal is 2., Mattermost seems to be aligned to the open source principles and Mozilla's culture. Again, this is just my opinion and I may be not considering all the nuances :).

jgmac1106 commented 7 years ago

My two cents as a contributor on both the MoFo and MoCo side.

Mattermost was cool with many features above and beyond IRC. Yet I had to remember to open the link and I never did.

In IRC I opened the client and my channels were there.

This kind of tool shouldn't be about attracting new contributors. It's for the kool aid drinkers. This will not change for IRC or Mattermost. Casual users will always find social media channels first.

On the MoCo side of things its IRC (biggest among FOSS or die crowd) , but mainly a mix of Discourse (no one uses) and Telegram that many use.

So I think maybe you should consider what works best for staff and core contributors rather than growing the people who contribute.

So when deciding think, "Did this tool make my job easier or better?"

On Fri, Oct 7, 2016, 3:28 PM Taís de Souza Lessa notifications@github.com wrote:

Hey team, I am new to the company and probably I don't understand all the complexities involved in such a decision (and I am new to Mattermost as well ). So my perspective is more like an outsider I guess... I think we should think about the goals, as @flukeout https://github.com/flukeout mentioned in his post above. Is the main goal: 1. to attract and engage outside contributions? Or 2.to facilitate internal communications? If the primary goal is 1. I think we should do a research on the main contributors we want to attract and see which tools they are already using (lowering the barrier for contribution). Personally, I used Slack and it seems that many people are already using it and they make the onboarding very easy (but I am not sure this is the tool of choice of our target audience). If the primary goal is 2., Mattermost seems to be aligned to the open source principles and Mozilla's culture. Again, this is just my opinion and I may be not considering all the nuances :).

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/MozillaFoundation/plan/issues/837#issuecomment-252340403, or mute the thread https://github.com/notifications/unsubscribe-auth/AKC-pgSiEVeKTqnZ7vQHeKGFFgMiu4_Lks5qxp1RgaJpZM4KQgpz .

alanmoo commented 7 years ago

Let me start by saying I feel like I don't have enough information to cast an informed vote, though I'm leaning towards staying with MatterMost.

I saw this conversation going on Friday while I was on PTO, so I haven't had a chance to respond until now, but @cadecairos covers a lot of the thoughts I was having.

My instinct is that this effectively boils down to "IRC or Mattermost" as all the alternatives are going to have their own set of issues and there will be significant inertia involved in moving people around.

I think there are a few topics that we should think about when considering what to do:

Discover-ability (Of chat platform and channels within)

One of the reasons we haven't seen much contributor uptake is likely that we haven't been advertising Mozilla Chat. I did a quick find on the ReadMe of a bunch of our current repos, and none of them mention "chat" or "Mattermost". That being said, we've obviously got a lot of internal use as it has been mentioned in calls and during onboarding.

Once people are in, non-IRC solutions seem to have the upper hand on discoverability of rooms/channels, as they have a built in directory- something IRC is sorely lacking unless a separate wiki is maintained, which can be a lot of overhead and is generally non-obvious.

Internal communication/planning

My gut here is that Mattermost is a much more effective tool for internal use than IRC. Just look at how many steps are required to create a private channel in IRC. The ability to rope people into conversations and know that the history of them will always be visible is valuable.

Contributors (new vs existing, coding vs non-coding?)

IRC may be more standard for coding contributors, but I agree with @jgmac1106- I don't believe that whatever we use should be a tool for drawing people in; It's better thought of as a way to communicate with staff if desired. In my eyes, Mattermost has a bit of an advantage in that regard- it is easier to jump into (login requirements notwithstanding; I'd wager people generally understand those way more than what IRC is and how to log in to it)

MoCo

I don't believe this should be a test-pilot for the rest of Mozilla at all. They are primarily an engineering organization and have significantly different needs than MoFo. Sure, it's easy to jump between channels within the same platform, but what use is that if the people you're trying to communicate with aren't on that platform? Anecdotally, I feel like it's easier to reach non-engineers now than when we were on IRC. And I've seen plenty of MoCo folks using Slack for their own small teams when I'm in offices, so using a second platform definitely isn't unheard of.

Async

The dual sync/async nature of non-IRC platforms shouldn't be understated. If a contributor does find their way in at some odd hour, having the ability to get back to them via the same channels is important. Generally speaking, if an IRC user went offline, we wouldn't be able to reach them again hours later. The fact that Mattermost does have email notifications for when you're offline can be leveraged in ways that previously required someone to be using something like IRCCloud.

Doubling down

I see an opportunity to enhance integrations with whatever chat platform we're on. We've attempted to integrate GitHub events, but I think we need to spend the time to figure out what we really need from these integrations (not just from GitHub, but from other platforms too) without making the signal:noise ratio worse. The fact that Mattermost is a controlled ecosystem (vs a myriad of IRC clients) may mean we can have some input in what features are worked on. Though I wouldn't make a decision strictly based on that hypothetical.

(Sidenote: @acabunoc mentions Gitter's GitHub integration; I do like that specific feature, and sidebar activity integrations in general, but I don't think that alone makes Gitter better. We have to remember that we're looking at this from a repo-centric perspective which doesn't necessarily translate to the rest of the org).

That all being said, I do think we need to gather input from the rest of MoFo that's been using it and see what they think. I'd like to bring in the most active (excluding prod team) and least active staff users and ask them why they are/aren't using it.

ryanwarsaw commented 7 years ago

It's important that the final decision ultimately be one that can be agreed upon universally as a step in the right direction in terms of user intuitiveness, easy of use and its ability to foster the end users ability to collaborate and interact with others without technical hinderance.

In order to determine whether or not a solution fits this bill, I think it's extremely important that the communication needs from various MoFo initiatives (and projects) be included as consideration, as well as specific scenarios (such as a new contributor, or new non-technical employee).

I agree with @alanmoo on his point on the test-pilot situation, it's very likely that this isn't a one size fits all solution and that the communication platform Mozilla Foundation decides to use, might not be the right one for MoCo.

I disagree with the concept of choosing a platform for communication based on what the target group of people (who are or may become Mozilla contributors) may potentially be using in terms of communication in order to attract and engage with this group of people.

As long as the solution is one that is easily user accessible (regardless of technical ability) and efforts are made to continue to lower the barrier of entry, those who have a genuine interest in contributing (or what ever the case may be) should be able to oversee the minor inconvenience of potentially needing to install a new piece of software on their computer.

This was already the case with the previous usage of IRC, and is still the case with Mattermost.

TLDR; The solution needs to continue efforts to lower the barrier of entry (and improve on the users ability to carry out basic communication tasks) as well as be the solution all want to use.

hannahkane commented 7 years ago

Have moved to a combination of Mattermost and Slack