Analyze the Etherpads

[xmatthewx]

Here's some info we could try to extract from quick brainstorm with @gvn.

easy:

• total pads
• total word count

harder:

• word cloud
• word count per pad mean, median, deviation
• links to gdocs, flickr,

hardest:

• author count
• bold, italics?
• sentiment analysis
• jargon, buzzwords, acronyms,
• inclusive language, gender bias

[xmatthewx]

@auremoser – can you add some quick notes on the tool and process you identified?

[xmatthewx]

yay

[xmatthewx]

count of

• +1s
• Mozilla Issues

[auremoser]

Super dupe, requirements and ideas are in the readme along with the data now, in this github: https://github.com/auremoser/mofolondon-ether.

more counts:

• number of links/urls
• number of "+1"
• number of edits
• number of words
• search issues:
  • web literacy
  • decentralization
  • open innovation
  • digital inclusion
  • online privacy + security

Might be best to use this as a light pilot, only 20 etherpads have content. I'm not sure folks knew there were pads. It's a shame! We should push it next time.

[auremoser]

I'll add more soon!

[xmatthewx]

Thanks again for pushing this forward @auremoser. A couple thoughts:

• When we share some stats (and your techniques) with staff, we can also request people to send me missing etherpad links and add gdoc links to their unused pads. Even if we don't rerun the stats, others can find stuff later in the app.
• Ultimately, this serves as a demo of what could be done (and how). So, let's just find some interesting stats without too much fuss.
• I really like the filename duds.csv
• Also, noticed this while looking at the duds: https://help.github.com/articles/rendering-csv-and-tsv-data/

[auremoser]

Cool @xmatthewx ! I'm off today but going to get back on this tomorrow. Yes I should have made headers for the csvs! bad formatting. I would like to make it so all you have to do is run a few scripts, for mozfest etcet and you get some basic stats for your etherpads. :) I'll get back on it tomorrow! Thanks for the feedback!

[xmatthewx]

I did a quick look and found:

• "literacy" appears 30 times, "inclusion" 8
• there are 74 +1s

Then I dug into top words, filtering out common words (might have been a bit too aggressive using "custom 9300".)

• top words: mozilla 127, mozfest 126, web 72, (internet 6)
• twice as many links to https 115 as http 58

I removed names and uninteresting words, then tried to form a bit of a narrative:

We talk about our platforms

• google 47
• github (gh) 36
• chat 18
• etherpad 15
• discourse 12
• arduino 11
• slack 7
• youtube 7
• twitter 7
• makey 6

...and our issues & programs.

• literacy 30
• mln 25
• clubs 28
• wow 20
• iot 9
• inclusive 7

We focus on people

• members 22
• participants 15
• communities 13
• cohort 10
• users 8
• attendees 7
• students 6

...especially distributed leadership.

• wrangler(s) 24
• leaders 19
• captains 10
• fellows 10
• coordinators 9
• partners 9
• facilitators 8
• mentor 8
• educators 8
• researchers 6

We work together

• network(s) 31
• connect(-ed, -ing) 31
• feedback 18
• convening(s) 16
• collaborate(-ive) 11
• coordinate 9
• inviting 8
• interviews 8
• pulse 8
• mentorship 6

...and do it smartly.

• proposals 29
• takeaways 15
• onboarding 13
• practices 11
• roles 11
• consulted 10
• darci 9
• lessons 9
• metrics 8
• goals 8
• norming 8
• decider 7
• template 7
• accountable 5

Communication is central to our work

• shared (-ing) 20
• email 13
• blog 14
• comms 12
• engaging 9
• conversations 6
• tweet 5
• emails 5
• documentation 5
• demystify 4

...and we broadcast confidence & excitement.

• +1 74
• guaranteed 19
• awesome 16
• gratitude 11
• excited 11
• opportunities 11
• yay 6

The work continues...

• challenges 5
• tbd 5

[auremoser]

Yay!!!

[xmatthewx]

a huge thanks to you @auremoser for making this possible. 👏 🚀 🌚 🎸

[xmatthewx]

We should do an unfiltered search for issue names and synonyms. Maybe also for words that express confidence, excitement, concern. Here's a site for finding text analysis tools, such as examining sentiment. No idea if these are the best options.

@samanthaburton – it'd be useful to have keywords for the issues. Can you point us to the summary language you had on your nest posters? Thanks!

Again – this is just a demo of what we could do, if we get better at documenting things with tools like Pulse and the Event App.

cc @edrushka @kristinashu @mmmavis

[mmmavis]

@auremoser this is so COOL! 👍 I especially love the idea of appending /export/txt to each pad's URL! What an awesome hack - I was trying to figure out how Etherpad API works & how I can acquire a API key... and now the /export/txt totally solves the problem!

[samanthaburton]

Summary language is in the updated 2020 strategy!

We don't have 'official' keywords yet, but @KevZawacki is going to be working on defining some for each issue to help our monitoring & evaluation efforts - so, depending on the timeline for this analysis, that work could be helpful.

[auremoser]

Thanks @xmatthewx for the idea and the narrative breakdown :), and yah @mmmavis !! one of our fellows, richard, suggested that hack to me :)

[xmatthewx]

Thanks @samanthaburton. It's going to be challening to count keywords with high confidence that they're related to the issues. Insights are tricky!!

We could (1 )find keywords, (2) grab surrounding text, then (3) analyze for relevance.

Keywords directly from issue statements:

• Online Privacy & Security : online data, safe, protect, companies, governments, ownership, digital identity(ies)
• Innovation Open : open source, open standards, products, policies, practices, entrepreneurs, innovate, compete, permission
• Decentralization : technologies, platforms, interoperable, standards, systems, transfer of information, content
• Web Literacy : skills, read, write, participate, digital citizens, consuming content, creating
• Digital Inclusion : access, opportunity, participate, diversity

[mmmavis]

@xmatthewx

Got something up just for us to play around: http://allhands-etherpads-analyzer.herokuapp.com/ We'll need to spend effort on data mining to make raw Etherpad data useful though 👀

Word Cloud: http://allhands-etherpads-analyzer.herokuapp.com/wordcloud ☁️

[auremoser]

really cool @mmmavis !