hypothesis / report

The internal global reporting product for Hypothesis
BSD 2-Clause "Simplified" License
0 stars 0 forks source link

Launch reports as indicator of social engagement #149

Open jon-betts opened 1 year ago

jon-betts commented 1 year ago

This is very WIP at the moment, but there is a desire to measure some forms of the level social engagement in an organization / course / assignment.

The initial suggestion was:

We might want to modify this:

Some alternative ideas

Launches / users - a quick approximation

If we take the time / unique user elements out of the above, then it boils down to average launches per user (per object). We already have the data to calculate this for various things and so we could present this number very easily.

Showing reply graphs

Much more ambitious, but might showing a graph (as in nodes and edges) of annotations and replies might be informative, or might not. It might inform which other types of graph metric would be useful. For example are replies very deep, or shallow? Is the pattern of replies spikey (some deep, some not) or relatively flat?

Brain storming other ideas for engagement

Should we think about what might indicate engagement more broadly, or go with this single idea?

Is there something there about engagement per user? Are some users highly "engaged" and some not at all?

darylhedley commented 1 year ago

Ok - so after reading this write up and having a few other catch ups, I think a good first approach to getting the data we need for the project is:

  1. Have a global stat that is based upon all users and should live on the business dashboard (or organisations overview). This stat should be launches / user / assignment. So, how many launches per user within an assignment. This will help us know if we are making improvements to social engagement across the board. I wouldn't mind this being over a course rather than an assignment.

  2. Have a metric based upon individual assignments. We currently don't have a good place to put this. But I would like to have a view of an assignment and the launches per user in a graph like described here: This was refined to: As a graph, show the number of launches per user, ordered by first launch date This is better, because it shows all users This will help us understand which assignments are doing better for us to dig into the data further if we need to.

I'm going to put it out there that since we discussed the idea of a reply graph - I cannot get it out of my head. I think this is a great idea Jon. I could imagine us using this in Product to find out where conversation is happening, how long threads are and who is keeping the conversation going - is it returning teachers who are proactively pushing the conversation or is it all student led?