Planning for Community Discussion on Programming Historian Goals

[walshbr]

During the August call - #553 - we discussed #552 and decided that the conversation would be a good opportunity to engage the community again in the same mode as #152. The conversation internally about the goals and mission of PH is one thing, but would be useful to hear from the larger community about how they see the project, what they get out of it, what would be useful, etc. We agreed that a ticket for public consumption could be a good approach, so this ticket is meant to serve as planning for that community conversation:

What kinds of perspectives are we interested in? What questions and conversations do we want to cultivate? What is the goal of the conversation? Is GitHub ticket the best format?

After some brainstorming we can move to engaging the community more directly.

[acrymble]

I think survey monkey will be a good idea

[acrymble]

Sorry I missed the meeting. Who is the target of this consultation? As I've said a number of times, what authors got or thought they'd get out of it might be starkly different from what readers do.

Can we focus this? And what do we intend to do with it? If our readers all suggest that we take a completely different approach than we want to, do we all step down?

[walshbr]

Would you mind weighing in and reframing your thoughts on this @alsalin? I believe the idea was yours, and would want people to respond to it as you articulate it rather than as it gets inflected through my memory of the conversation.

[alsalin]

hi all, actually the initial survey idea came from @mariajoafana, I believe. the sense of it that I recall was that it might be time to check in with our readers/users to get a sense of how they use the materials they find through this publication. the survey might end up being part content survey and UX survey - how do you find our content? what do you do with it? do you use it as a reference? do you read through it in its entirety (which I'd be shocked to hear, but cool if so)? etc. I think a survey monkey or google form would be fine and we can use this space to cook up some appropriate questions. @mariajoafana - did I get the genera overview right? what did I miss?

[drjwbaker]

A general survey with elements that help us reframe our goals would be sensible. I'd be keen to ensure that we know who those completing the survey are (crudely, historian or not historian) and where they are from (geographically). I'd also be keen to ensure that we see this as a substantial piece of work, involving things like:

• keeping the survey open for a long period (6 months?)
• clear articulation of why we asking readers for their time
• in person/conference promotion of the survey
• an international strategy so we don't just get our friends completing the survey
• banners on all the lessons (start and finish?) asking readers to complete the survey (and why)
• a short co-authored report/paper based on the survey

[drjwbaker]

(I should add, sorry if a) any of this was covered in the meeting I couldn't make b) this is totally off piste and not what you want to do :) )

[acrymble]

We'd need a separate initiative for Spanish, I would think.

Should we start drafting some questions somewhere?

[walshbr]

Like all of those points a lot @drjwbaker. @acrymble - I had intended this ticket to serve as a space for drafting of questions and the conversations around it, so go ahead. But if you think that deserves its own particular ticket feel free to move the conversation to another issue. Don't really see a problem with having that conversation happen in the open, but we could move to the google group if people desire.

[acrymble]

Of course, @walshbr.

Our earlier survey on gender had these questions, which might be repurposable:

1. Which of the following ways have you engaged with the Programming Historian? (reader, reviewer, editor, author, educator, none of the above).
2. On a scale of 1 (novice) to 6 (advanced), how would you rate your confidence with technology?

[walshbr]

I like those. Maybe something like these too -

1. In what ways does the Programming Historian enable your work? What works well for you?
2. What would help the Programming Historian be more useful for you? What could we do better?

But those might be a bit too pointed in the direction of utility, and imagine that we might want more specific questions as well. Those are a bit vague and general, but could be general grab bag, write what you will opportunities for capturing general public feeling. Those also might point a little too much towards the pedagogical aims of the project than the research publication side.

[mariajoafana]

Yes, @alsalin that sounds right. I agree with @acrymble on the need for a separate survey for Spanish. For the survey we did in the Spanish team last year these questions might be useful here:

1. Choose your disciplinary field(s) (information science, history, art history, linguistics, literature, archeology, etc...)
2. Which digital methodologies, tools and/or workflows do you use often in your research and teaching?

[drjwbaker]

Quick holding note. I've been reading a great pedagogy book and it has an excellent section on structuring feedback. Takehomes: don't ask about things you don't want to change; structure questions in a particular less-open ended way that means you can easily analyse the results. Anyway, I don't have the book in front of me right now (hence the holding note). More tomorrow!

[acrymble]

Good points @drjwbaker. What do we want to open for change here? Not everything obviously. Presumably we don't want to consult on the lessons page having just spent a year building it. What do we want to know?

We can learn a lot about user behaviour from our Google Analytics, so it should be something we can't already find out. Knowing what proportion of our readers are humanities scholars might be helpful (though we won't be able to capture that easily). And finding out about research made possible from the lessons would also be great. But neither would be about changing the project.

I'd be more interested in consulting with our authors, because I see those as the group with whom we have the closest relationship and greatest obligations towards. The readers seem to come no matter what, but we have to be good to our authors.

[drjwbaker]

Okay. The module feedback/evaluation thing I found uses the following format.

--

Part 1 is split into two columns. Column 1 is headed 'I like the way the XXX:' and Column 2 'I would like the XXX to:'. It then has 15-20 statements in each columns that the respondent can check if they agree with it.

So, we could do something like:

Column A

I like the way the Programming Historian: _is written by historians. _has enabled me to do better research. _has supported my teaching. _is peer reviewed. _is open access. _has a Spanish version.

Column B

I would like the Programming Historian to: _summarize the main points more often. _drop peer review. _enable me to edit lessons more easily. _seek people willing to translate its lessons in my native language.

Then in Part 2 of the survey it has one or two more open-ended questions which people can write free text responses to.

--

The up side of using this is that it give us lots of quantifiable and easy to manage feedback, helps us steer the survey towards things we are willing to change, and encourages respondents to only write in free text things the things they really care about or which they think are not on the survey. My hunch - if we go with this model - is that we use Part A to mostly find out if what we think PH is is shared by our users (I'm not too bothered if we use leading questions, this isn't contributing to proper social research after all) + then sprinkle in a few more controversial topics that we are willing to shift on (eg, wikis, user editing, depreciation).

Oh and the book this is from is Teaching What You Don't Know. Which is super awesome and one of the many things that @gvwilson has put me onto over the years.

[drjwbaker]

If we go ahead with a survey, note idea to tie to mission statement https://github.com/programminghistorian/jekyll/issues/552#issuecomment-333105923

[walshbr]

Is someone willing to take on this initiative and/or #579? My sense (that I articulated on #644) is that, as conversation has died down, we're either ready to table these for now or take action. My suspicion is that this might be an initiative that we're interested in but lacking the bandwidth for at the moment given the current people on hiatus (was thinking especially that this might be something to save for when @alsalin has more time to devote to this).

[drjwbaker]

@walshbr Revisit in the new year? PH has changed a lot this year. It would be useful to see if we are on track with our users.

[walshbr]

Discussion at #678 was that it makes sense to think in the direction of a survey. Assigning @alsalin to continue the discussion working off of what has been shared here so far.

[alsalin]

Hi folks!

To summarize, here are the suggested questions so far, though of course the way we end up phrasing them may be different (e.g. 'add specifics to guide reader as suggested by @drjwbaker and always have an 'other' field):

• Which of the following ways have you engaged with the Programming Historian? (reader, reviewer, editor, author, educator, none of the above).
• On a scale of 1 (novice) to 6 (advanced), how would you rate your confidence with technology?
• In what ways does the Programming Historian enable your work? What works well for you? (add specifics to guide reader)
• What would help the Programming Historian be more useful for you? What could we do better? (add specifics to guide reader)
• Choose your disciplinary field(s) (information science, history, art history, linguistics, literature, archeology, etc...)
• Which digital methodologies, tools and/or workflows do you use often in your research and teaching? (add specifics to guide reader)

Some more ideas?

• Where do you primarily do your DH work? (some kind of geography question)
• How do you primarily find PH lessons? (social media, google search, peer suggestion? - though we have analytics for this, this question is tied directly to the other given answers here)
• As a user of PH, would it be helpful to interact with other users of PH to discuss methods and issues with implementation of lessons? (e.g. I'm learning python and getting stuck with lesson x, who else is too?) If so, what would you like to see?
• On the PH website, are lessons easy to find?
• Do you prefer to read through an entire lesson or skim to find relevant information? (alternatively, we could ask a question about learning styles).

Other ideas? For editors, readers, authors? In UX and engagement? In content?

[acrymble]

What would we do with the information we gained from this survey? What process does this feed into?

[drjwbaker]

On which, a good way of writing a question is to think about what we might do with the answers.

For example - as @alsalin notes - it seems clear we can say we would use responses to 'How do you primarily find PH lessons?' in two ways: 1) compare/contrast with data we have on how people find PH; 2) revise comms strategy.

On 'What would help the Programming Historian be more useful for you? What could we do better?' - however - we'd have to ensure we only include things we are willing to action.

Great starting point @alsalin. I vote for keeping the survey simple and not adding more (it should take no more than 2-3 minutes to complete), and focusing - as @acrymble suggests - on what we want to get out of responses to each question.

[alsalin]

@acrymble excellent question. As I was summarizing the questions we had already, I found that topics fit into certain categories: knowing our audience (and thus reaching out to them more effectively, choosing content to publish that matches this audience and/or expanding audience), website experience (a good thing from a QA perspective after a redesign anyway), content audit of sorts (is what we have what our audience needs/expects). Not that we have to go with this set of questions at all. We could choose categories to investigate and work from there. Either way, I think we need to ensure that we are willing to engage with the survey results to change things at some level, e.g. better audience engagement or different audience engagement, more lessons on x, etc. Now what process does this feed into? That I'm not sure :) Not sure if there are existing pathways for this kind of thing, though I know surveys have been done before at PH - how were those interpreted and what came out of them?

@drjwbaker I agree - let's keep it short, preferably to a page of questions - two, my goodness, at tops.

[acrymble]

@alsalin in the past we started with a problem (too much male participation in the project) and used the survey to find out what barriers we had inadvertently introduced into the project. We then applied some of the suggestions to reduce or remove those barriers. I also wrote up a reflection on that process and published it in a pedagogical journal.

[drjwbaker]

Potential problems:

• we don't know who our users are (aside from stats)
• we don't know what our lessons are being used for
• we don't know if people know what we do (eg, ES work, infrastructure/sustainability work)
• we don't know if people have thoughts on what we can do better
• we don't know if some of things we are thinking of doing are wanted
• we don't know what barriers to learning our lessons present (personal eg, I've grappled HARD to get Python 3 working..)

[acrymble]

Thanks @drjwbaker. I think knowing what people are using the lessons for (and connected to that, what we've enabled with this project that feeds into #686 our desire to understand our impact, would be a good use of this process.

[alsalin]

Hi folks - just checking in here. Does anyone have any other question suggestions? If not, at this point, given the issues that @drjwbaker has summarized, I can reword the survey questions we've come up with into some thematic groups and repost that here for review.

[drjwbaker]

Go for it @alsalin. Thanks so much for taking a lead on this!

[alsalin]

Hi all!

I've narrowed down the questions to 3 main areas - who are our users, how do they use the site, what could we do better. I also cut out some questions in favor of making other questions more robust and trying to keep the survey short. There are a few areas where I'd love some help coming up with options for users to choose from - see below!

Who are our users?

• Which of the following ways have you engaged with the Programming Historian? (reader, reviewer, editor, author, educator, none of the above).
• On a scale of 1 (novice) to 6 (advanced), how would you rate your confidence with technology? Choose your disciplinary field(s) (information science, history, art history, linguistics, literature, archeology, etc...)
• Which digital methodologies, tools and/or workflows do you use often in your research and teaching? (text analysis, network analysis, data visualization, web development, virtual/augmented reality, data cleaning)
• Where do you primarily do your DH work? (some kind of geography question)
• How do you primarily find PH lessons? (social media, google search, peer suggestion? - though we have analytics for this, this question is tied directly to the other given answers here)

How do they use PH?

• In what ways does the Programming Historian enable your work? What works well for you? (reference, learning tool for the first time, use for students/in-class - other options)
• What would help the Programming Historian be more useful for you? What could we do better? (better search for lessons, lessons in other languages (which ones?), other issues )
• Are PH lessons that you have used easy to understand and get up and running? If not, why?
• For reviewers and authors: Are our instructions for getting involved with PH clear? If not, why?

What can PH do better? We seek to continually improve how we present our lessons to our readers and work closely with reviewers and authors to create sustainable and reliable tutorials. Let us know if you have suggestions, questions, or comments on lesson presentation, content, website design and experience, or anything else. [open question]

[drjwbaker]

@alsalin Awesome job. I would remove "On a scale of 1 (novice) to 6 (advanced), how would you rate your confidence with technology?" simply because it is super subjective and is unlikely to change how to read their responses to other questions.

If it would help, perhaps open a separate issue for each of the two 'other issues' questions so we can thrash out options we want in there outside the main discussion?

[arojascastro]

Should we add some questions concerning Spanish translations? It would be a good way to assess the quality of our work. Otherwise, we may create our own survey. What do you think @mariajoafana @vgayolrs @jenniferisasi @jamotilla ?

[acrymble]

I'd do that separately since it makes most sense to ask in Spanish.

On Fri, Feb 23, 2018 at 10:26 AM, Antonio Rojas Castro < notifications@github.com> wrote:

Should we add some questions concerning Spanish translations? It would be a good way to assess the quality of our work. Otherwise, we may create our own survey. What do you think @mariajoafana https://github.com/mariajoafana @vgayolrs https://github.com/vgayolrs @jenniferisasi https://github.com/jenniferisasi @jamotilla https://github.com/jamotilla ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/programminghistorian/jekyll/issues/576#issuecomment-367970564, or mute the thread https://github.com/notifications/unsubscribe-auth/AAnd94jzLw_JITykEi4LDO93_QgpwJrUks5tXpJHgaJpZM4PBre1 .

[jenniferisasi]

I am inclined to say that we do a separate survey with the same questions in Spanish, however, I feel like our audience can/should also take part in the survey in English mainly for one reason: we have fewer tutorials so people might be using the PH in both languages. What do you think @arojascastro? Of course we could also add that question on the survey in Spanish.

Also, on the question about geography: "Where do you primarily do your DH work?" would it be worth to ask something along the lines of "What is your primary language of research and/or publication?" I think it could be interesting to see if users are learning from our tutorials (both in English and Spanish) but then still adapting them to their own languages for research outreach.

[alsalin]

@jenniferisasi I like both of the ideas to add spanish questions to the english survey and to add questions about primary language for research/publishing.

[alsalin]

Hi all! If there are no other suggestions, I'd like to get this rolling in an actual survey platform. What do folks recommend? SurveyMonkey or Google Forms? Do we have preferred accounts to serve out the surveys from? (e.g. like a PH account for SurveyMonkey?) Also will we just be tweeting out the survey or will there be a writeup about it?

[drjwbaker]

Also will we just be tweeting out the survey or will there be a writeup about it?

@alsalin You raise an important point here as the scope and outputs of the survey should be clear to the participants. Some ideas (which you might have thought of already..):

• state the period it will be open (3 months?)
• state that we won't keep data (those on the European side of the project need to be careful we don't breach data protection laws) and will meaningfully anonymize all results (so any published data removes outliers)
• state the output. I'm for a short blog report. Happy to help write this.
• state the impact. You've designed the survey so that it only includes questions the answers of which we are willing to consider in our future work, so there should be line on how we get from participant responses to change. Say: sub-group convene to consider results, propose actions to editorial board, we publish changes that come of our of survey (within ~1 year of survey publication).

No preference on survey platform.

[alsalin]

Hi folks! A rough draft of the survey is up in my google drive: https://goo.gl/forms/2w0PnVtnTjlJa3ss2

We can always move platforms, but it might help to see the thing in action to get a sense for what we might need to edit.

[acrymble]

@alsalin thanks for the link. The survey looks good, but it implies all of our users are or should be academics. That's probably not the case given our large numbers of readers.

[arojascastro]

I couldn't respond yet but I will. DariahTeach launched a survey as well. Here is the link: https://www.surveymonkey.com/r/dariahTeach

[drjwbaker]

@acrymble Good catch. This is easily fixed, by making some questions non-compulsory and then beginning with something like "If you research or teach, .."

[alsalin]

@acrymble @drjwbaker ah yes - our internal biases apparent. do we have any suggestions for engaging our non-academic readers? I can easily add, "If you research or teach," to the first part, but might we want to get info on other occupations?

[drjwbaker]

Prompt people to offer their non-acdemic roles rather than assume a limited set of categories?

[alsalin]

New version of professional role question up - took inspiration from the dariahTeach survey (thanks @arojascastro !)

[arojascastro]

I like it because it is short and clear and I hate long surveys. Just one comment: None of the aboe --> None of the above

[alsalin]

Assuming folks are ok with the current state of the survey - how do we proceed in rolling it out?

[mdlincoln]

@alsalin A short blog post and several tweets would be good, of course - but I wonder if we might also send out targeted emails as well to our many reviewers & authors?

[drjwbaker]

Great job @alsalin. On the blog, key points to tease out are why we are doing this, how it will feed into our work, and how long the survey will be open. Agree with @mdlincoln: we should authors and reviewers, plus target digital history centres/groups/seminars (where to host a list of these?)

[alsalin]

@drjwbaker could you help me get started with writing up the blog piece? I haven't worked in that area yet!

[drjwbaker]

@alsalin Sure. To clarify, do you mean that you need help with the technical side of hosting a blog on PH or with writing? (I'm guessing more the former?)

[alsalin]

@drjwbaker yep! also, would it be best to post a draft for comment in this forum or elsewhere?

[drjwbaker]

@alsalin Not sure what the protocol is here. I've seen some blogs posted here first for comment. @mdlincoln: can we stage a blog for preview via Jekyll like we do lessons?