Using Hypothesis annotation to improve Programming Historian tutorials

[amandavisconti]

It could be useful to build on Hypothesis web annotation as a path for anyone working through a PH tutorial to leave inline suggestions for improvement, questions, "this is where I got stuck" flags, and other comments. Already, anyone can do this using a Hypothesis browser extension or just by prepending the tutorial's URL with https://via.hypothes.is/ (e.g. https://via.hypothes.is/http://programminghistorian.org/lessons/intro-to-the-zotero-api). But we can do more to make annotation benefit the PH community, such as:

1. Using social media with the hastag #DHannotates (and possibly a note on the PH tutorial site?) to make potential tutorial users aware of this as a way of both supporting their work and giving back to PH through suggestions. @ShawnGraham and I were thinking about encouraging this actively during the week of February 8-12 (I believe his students will be using PH that week), with the idea that annotations added during just that week would be looked at by us (and we'd address any questions, create an issue here if we couldn't answer the questions, and/or locate people via Twitter/the DH Slack who might be able to help).
2. Automating the creation of issues in this GitHub repo from these annotations, so that when someone makes a suggestion or asks a question this gets added to the issue queue.

Automating issues from annotations There isn't currently an IFTTT or Zapier Hypothesis channel, but Hypothesis does create RSS feeds we could use to automate creating issues from annotations:

1. An RSS stream of all annotations with a given tag (https://hypothes.is/stream.rss?tags=yourtag), OR
2. An RSS stream per PH tutorial (e.g. https://hypothes.is/stream.rss?uri=http://programminghistorian.org/lessons/intro-to-the-zotero-api)

The second option sounds better to me: we would need to create an IFTTT recipe for each new tutorial, but I think it'd be easier to add that to the current list of steps for publishing a tutorial, then to make sure every time someone adds an annotation they also think to tag it a specific way.

On the other hand, maybe there are some types of annotations on the PH tutorials that shouldn't be turned into an issue—conversations in the annotation sidebar, for example, might spawn a bunch of separate and confusing GitHub issues? One way to address this is creating an issues queue label that will only be applied to new issues coming in through automation; these could be relabeled as time/volunteers allow.

Any thoughts on using Hypothesis/inline annotation in general to improve PH, getting the word out about annotating PH via social media, and/or automating issue creation from annotations would be awesome. (Huge thanks to the PH editors, writers, and users for this great resource!

[shawngraham]

I like the idea of RSS per tutorial. My gang are working through several tutorials as part of our digital history seminar (http://grad.craftingdigitalhistory.ca/weekly.html) so you'll start seeing some annotations happening already. I came across waffle.io that turns issues into kanban-board style cards for managing all this kind of thing. Perhaps a bridge too far for the current conversation, but perhaps useful in the future.

[wcaleb]

I love this idea! Thanks for organizing it!

I'm a little concerned about creating an issue for every annotation, for the reasons @amandavisconti describes. That could potentially clutter our issues inbox. Some annotations might just be questions that a person needs assistance with, which isn't necessarily an issue ticket sort of thing.

Is there a happy medium where annotations that we want to become issues have some sort of hashtag or trigger word in the annotation itself? That's probably not a whole lot different than tagging the annotation, though, and I see the point that this could be difficult for people to remember to do. I'm not yet very familiar with the hypothes.is interface, but is it possible for a volunteer to reply to an annotation that has forgotten to add the hashtag and add it there, so that the whole thread would still be caught in any automated dragnet (IFTTT recipe or other script)?

[amandavisconti]

It looks like IFTTT can be set to only fire if the RSS item (annotation) contains a certain word or phrase (so, it wouldn't need to be added as a tag).

I tested an IFTTT recipe looking for the keyword "help" in the annotation, and tried replying to an existing annotation—and replies to annotations aren't being pulled into that RSS stream, so it didn't work. (Same thing but with an annotation, rather than a reply to an annotation, worked.) I didn't see a way to include replies in the feed URL in Hypothesis' issues queue, so I asked about it on their Google Forum. Will update if it's possible!

Is there a way to turn off inbox notifications for a specific issue label? If that were the case, we could drop all incoming annotation issues to that label, and they could be sorted into other labels or deleted as volunteers had time.

[amandavisconti]

Hypothesis says there currently isn't a way to include replies in the RSS, though they're working on it. They noted that the permalink to an annotation includes its associated replies, but there isn't currently a way to trigger an issue being created from a reply.

[wcaleb]

Thanks for looking into this! Maybe for the #DHAnnotates week we should just do manual issue creation. After we see what kinds of annotations people leave and how many of them there are, we could then make a decision about how to automate?

Sent from my iPhone

W. Caleb McDaniel Associate Professor of History Duncan College Master Rice University

http://wcm1.web.rice.edu/ http://www.duncancollege.net/

On Feb 2, 2016, at 12:45 PM, Amanda Visconti notifications@github.com wrote:

Hypothesis says there currently isn't a way to include replies in the RSS, though they're working on it. They noted that the permalink to an annotation includes its associated replies, but there isn't currently a way to trigger an issue being created from a reply.

— Reply to this email directly or view it on GitHub.

[shawngraham]

@amandavisconti what if we fed the hypothesis RSS into a twitter account for the purpose (ifttt)? Then we could just monitor that twitter account and anything that needs serious work could be made into a ticket/issue, where the issue references the original tweet....?

[shawngraham]

Or maybe IFTTT hypothesis rss -> google spreadsheet?

[amandavisconti]

@shawngraham Both good ideas! You also mentioned (on the DH Slack) streaming annotations there (either to a dedicated channel for PH, or to something like the #DHanswers channel?). I'd like both having a record of all annotations on PH tutorials (in case that's ever useful, to see them all at once?) via a spreadsheet, and also having new annotations go through a place where we wouldn't need to go out of our ways to see them (Twitter and/or Slack).

[shawngraham]

@amandavisconti isn't it possible to expose all messages in slack to a webpage? Didn't Lincoln code that up?

[amandavisconti]

@shawngraham Yes, Matt Lincoln created a nice way: https://github.com/mdlincoln/jekyll-slack (example page for the #conference channel: http://matthewlincoln.net/jekyll-slack/conferences/index.html).

[amandavisconti]

I created a gist listing for all the current PH lesson URLs:

1. each lesson URL with Via prefix (direct link to annotate with Hypothesis)
2. each URL for the stream of all annotations per specific lesson Any annotations (anywhere on the web...) tagged with #DHannotates will appear in this stream: https://hypothes.is/stream.rss?tags=DHannotates

I also created a #ProgrammingHistorian channel on the DH Slack (tinyUrl.com/DHSlack) and pointed a couple of the feeds of lesson annotations there, but stopped to investigate ways of combining all those feeds into a single one. Any suggestions appreciated! (Needs to let me paste in a list of URLs instead of entering one by one, and needs to handle 50-odd links: the top Google results failed at this.)

[shawngraham]

Can a google spreadsheet ingest rss?

[amandavisconti]

@shawngraham Looks like yes (multiple RSS to one Google Spreadsheet) with IFTTT... and then we could have the Google Spreadsheet's new items all go into one RSS feed? (Zapier supports that if IFTTT doesn't.)

[amandavisconti]

@wcaleb @acrymble @shawngraham I'm going through annotations, and would like suggestions on addressing these in the way best for PH (e.g. not cluttering the issue queue). I have three possible ideas (and would appreciate others!):

A. Adding as repo issues (perhaps with new issue label[s] like "questions", "suggestions"):

1. Questions particular to a specific user (not likely to be helpful to many others/so specific that inclusion in the lesson would clutter it)
2. Questions on something that the lesson should address and doesn't (should be addressed in the lesson)
3. Other suggestions for lesson improvement (examples, links, detail)

B. Add a FAQ section at the bottom of the lesson with these questions and details (pull requests on lessons)

C. An editable wiki page for listing outstanding questions/improvements with links to the annotations needing addressing (can be addressed as pull requests, but unlike option B these would be suggested changes to the main lesson rather than a new FAQ section of the lesson)

If we could make these questions and suggestions findable on GitHub, it might increase who is willing to submit pull requests addressing these issues.

[acrymble]

There's a bit of an added dimension here we need to consider.

Firstly, these are people's publications and they've gone through a review. I think we can make minor changes without getting in touch with an author, but we'll have to go slower with something one might reasonably consider 'major' changes to a lesson.

Related, people use these tutorials in a classroom setting, and we all know how stressful it is to rely on a resource you don't control, only to find out it has changed the day before you need it. Any changes will therefore also need to improve without breaking anyone's use of it.

[amandavisconti]

Great points I hadn't considered. Thank you!

[wcaleb]

Of the options @amandavisconti lists, the only one I'm somewhat wary of is C: an editable wiki page. That would create another thing to curate and keep up to date, whereas issues and pull requests keep everything within the same basic queue. It also seems to me like it could largely reproduce the functionality already provided by the hypothesis platform and RSS feed. I'm not unalterably opposed, but those are the potential issues I see.

Out of curiosity, do we have a sense of how many issues would be created just by converting the annotations from the #DHAnnotates event to issues?

[wcaleb]

Another thought I meant to include. Some open-source software programs have both discussion lists and GitHub issue trackers. I'm thinking of Pandoc, for example, which has a Google Groups list called pandoc-discuss for general user questions, FAQs and problems, and reserves the issues on GitHub for actionable edits and bugs. The listserv might not be that different than a wiki, but for some reason (largely impressionistic) it strikes me as easier to manage. Will have to think on this further.

Thanks for all the great work and ideas, @amandavisconti and @shawngraham!

[amandavisconti]

There are currently 9 lessons with any annotations (one only has annotations reviewing the lesson's arguments about using certain software, so no issue creation needed). I've linked below all the annotations that have potential for changes to lessons. (Note that none of these are show-stopping or time-critical, just suggestions for improving and broadening the audience for some already fantastic lessons. Yay Team Programming Historian!)

Issues that seem more critical to address

1. Windows error and specific solution to that error here
2. Short tips to avoid problems later in lesson (here and here and here and here)
3. Clarification of what a command does and how to customize it here
4. Error message verified by two users here
5. Suggestion for clearer language here
6. Lesson missing indents are critical (change meaning in Python) here

Nice-to-have changes (doesn't impede use of lesson for most users)

7. An example of how to work backward from something to figure out further use of the lesson's skill link
8. Request to define two abbreviated terms in lesson link
9. More specifics for Windows users and for Mac/Linux users link
10. 2 places where brief alternative instructions for Windows (suggested in the annotations) could be stuck in. link
11. Instructions for solving a common error

@wcaleb Your thoughts on C (editable wiki page) make sense to me—the best outcome would be not adding to what PH editors are already doing, just making user thoughts easy to find and address (when/

[shawngraham]

@amandavisconti thank you for being so awesome!

[wcaleb]

I second @shawngraham!

[acrymble]

@amandavisconti are all these comments visible somewhere all in one place? Sorry there's been a lot of conversation in many different places, so I wasnt' sure.

[amandavisconti]

@acrymble Not all of them currently, but many were tagged with "DHannotates" which lets them all appear here: https://hypothes.is/stream.rss?tags=dhannotates. We could get them all by somehow combining the annotation feeds from every lesson URL; I'd be willing to set up one Google Spreadsheet that does pulls in all PH annotations via IFTTT, if it would be useful to have!

[acrymble]

Thanks. I think our general feeling in the editorial board is that we don't want to lose these, but we are all feeling a bit stretched And can't keep track of more than our own tickets on github. If someone was willing to make tickets if these, perhaps by lesson, we would certainly appreciate it. We know a lot of work went into this so we want that to translate into change, but we may need a little help.

On Wednesday, February 24, 2016, Amanda Visconti notifications@github.com wrote:

@acrymble https://github.com/acrymble Not all of them currently, but many were tagged with "DHannotates" which lets them all appear here: https://hypothes.is/stream.rss?tags=dhannotates. We could get them all by somehow combining the annotation feeds from every lesson URL; I'd be willing to set up one Google Spreadsheet that does pulls in all PH annotations via IFTTT, if it would be useful to have!

— Reply to this email directly or view it on GitHub https://github.com/programminghistorian/jekyll/issues/182#issuecomment-188356507 .

[amandavisconti]

@acrymble I'm happy to make tickets for these by lesson, as suggested—and then anyone can claim them as interest/time allows. Thanks to you and the rest of the editorial board for all your work on this awesome resource!

[amandavisconti]

I've added these annotations as issues using the form "User notes on [lesson name]". Closing this ticket, but I'd be happy to help brainstorm in the future about ways to include lesson user feedback (like annotations) in the review model that don't add to the project volunteers' workload. Thanks for the conversation and for experimenting with this!