What could replace the pull request part of checkout?

[gvwilson]

Lesson maintainers are still finding the PR portion of checkout onerous:

1. Lots of work for maintainers.
2. We don't really need material (esp. low-quality material) for mature lessons.

However, it's important for instructors to have hands-on experience of contributing to our lessons: telling people they can update things doesn't foster contribution nearly as effectively as having them do it at least once. Possibilities include:

1. Scrap it and hope that people will contribute anyway. (I'm skeptical.)
2. Create throw-away lessons for people to submit exercises against. (Not nearly as compelling, doesn't help them learn their way around our actual lessons, and who will create an endless stream of throwaway lessons?)
3. Have them proof-read existing lessons and file detailed issues rather than submit exercises. (This would level the playing field between SWC and DC by removing the Git/no-Git distinction, but someone needs to turn those issues into PRs...)
4. Something else (suggestions welcome).

Please add suggestions as comments on this issue.

[lexnederbragt]

A reusable (!) variant of 2) would be:

• create a copy of one of the lessons
• cripple it at a bunch of places and add FIXMEs there
• import this lesson each time a TT workshop is given
• have students submit PRs to fix a FIXME (file issue claiming a FIXME, file PR to fix, referencing the issue).

Advantages:

• we can make this crippled-lesson once and reuse it (and improve on it)
• mimics real lesson development without disturbing existing lessons and flooding maintainers
• more uniform experience between TT workshops
• PR acceptance can become requirement for finishing training
• others than the TT workshop instructors can help processing PRs (e.g. instructor-trainers-in-training)

Disadvantages:

• smart learners can find previous instances of PRs and reuse them (solution could be private repo, but this may invoke costs)
• not sure which lesson to pick
• lots forks start floating around, but that is already the case I guess

[bkatiemills]

There has been a number of discussions lately about an open-ended proliferation of lesson material; point them at that? Creates an authentic experience and marshals a bunch of effort for something we are currently elevating.

[ethanwhite]

I'm very happy that we're having a discussion about how to improve this issue. Thanks @gvwilson!

Feedback in response to suggestions of the same number:

1. Whether or not this is ideal I think it's a definite improvement over the current system. I think the downsides of lesson creep, maintainer time and energy, and disappointment for new contributors are substantial enough that this would be an improvement. If it's difficult to figure out a better alternative I think we should do this until such time as an alternative is figured out.
2. I don't like this for the reasons that @gvwilson and @lexnederbragt mention. In addition, I think that people who are smart and busy don't tend to respond positively to things that look like busy work, which this does. If this is going to be done then I think it should be as part of the workshop, where practice time will be more acceptable to most folks, but I suspect we don't have time to add it.
3. I like this as the requirement. Students could be encouraged to submit PRs to fix issues they find, but not required to do so.
4. Alternative: Have lesson repos maintain a set of issues for thinks they want fixed. When I've maintained SWC/DC lessons in the past there have always been small things I wanted to fix that I didn't have time for. Make the requirement a PR that addresses one of these issues. This requires some up front work on the part of the maintainers, but accomplishes the original goal without the lesson creep and wasted time and energy. It also gets meaningful positive work done.
5. Alternative 2: Since generating the list of things to do in (4) is part of the work the requirement could be something like: "Option A) Submit a pull request fixing an existing issue. Option B) Review one of the lessons and open one or more issues clearly describing improvements to be made to those lessons." These are both great ways to contribute to the lesson material, this approach reduces the amount of time that maintainers need to spend generating issues, and removes the git/PR requirement thus making the requirement the same for both SWC & DC.

[wking]

On Tue, May 10, 2016 at 08:34:10AM -0700, Ethan White wrote: “Option A) Submit a pull request fixing an existing issue. Option B) Review one of the lessons and open one or more issues clearly describing improvements to be made to those lessons.”

This sounds good to me. And lots of projects suggest leader issues to avoid wasting effort working up a PR that the maintainers will never land (e.g. because it's out of scope), so some practice with leader issues can translate into trainee's other open-source contributions as well.

[uiuc-cse]

How about contributing to bonus lessons, like shell-extras? That's not a long-term solution, but it does kick the can down the road a year or so, plus those do see some use.

[karinlag]

I quite like @lexnederbragt ´s suggestion. To enhance the learning potential of that I would deliberately "break" the lesson in some places, and tell the students that something is wrong with it. My reasoning for this is that I have run some discussion sessions this spring, and I have experienced that students seemed to have quite a passive relationship to the material. Breaking it and telling them to find out where, and then asking them to submit PRs to fix it would go a long way to get them to interact with the material.

[ErinBecker]

I also like the suggestion from @lexnederbragt. A possible modification is that instructors could be given a choice between fixing one of the "broken" demo lessons or making a substantive contribution to a real lesson. This way those who might be turned off by the idea of "busy work" could still have an opportunity to participate in a way that feels more legitimate to them, while still reducing workload for maintainers.

[rgaiacs]

Scrap it and hope that people will contribute anyway. (I'm skeptical.)

I'm skeptical too.

Create throw-away lessons for people to submit exercises against. (Not nearly as compelling, doesn't help them learn their way around our actual lessons, and who will create an endless stream of throwaway lessons?)

I think this is a huge waste of instructors and learners' time.

@lexnederbragt's variant is OK. But instead of break the lesson I would suggest to get the version from 6 or 12 months ago and add the "FIXME"s based on the pull requests that we accepted.

Have them proof-read existing lessons and file detailed issues rather than submit exercises. (This would level the playing field between SWC and DC by removing the Git/no-Git distinction, but someone needs to turn those issues into PRs...)

+1 for that. To avoid work for maintainers I would suggest that instructor trainer mark those issues with "IT-XX" tag where XX is the round of instructor training. For SWC we could also require that learners send a pull request for one issue tagged with IT-XX where XX is any of the previous round of instructor training since doing it this will be almost mechanical.

[raynamharris]

I really like @ethanwhite point #4 (aka suggestion 1) up there. This creates a very controlled environment that promotes mutually beneficial collaboration. I like how this gives the maintainers some control and it helps the newcomers feel more confident that what they suggest will be and should be incorporated.

[karinlag]

+1 to @ethanwhite suggestion from me too.

[tracykteal]

Thanks for all this discussion. I like @ethanwhite point #5

"Since generating the list of things to do in (4) is part of the work the requirement could be something like: "Option A) Submit a pull request fixing an existing issue. Option B) Review one of the lessons and open one or more issues clearly describing improvements to be made to those lessons." These are both great ways to contribute to the lesson material, this approach reduces the amount of time that maintainers need to spend generating issues, and removes the git/PR requirement thus making the requirement the same for both SWC & DC."

People could do a PR to fix an issue, or create an issue. When we're in the discussion section meetings, I find that when people are reporting back after their workshop we're encouraging them to either file a PR with a change they've already made, or to submit an issue on something they saw that didn't work. Both are very valuable, helpful to this checkout process and to the lessons. It also gets them interacting with github in a meaningful and substantive way, but filing issues doesn't require much github background, so as Ethan says, the checkout requirements could be the same for SWC and DC.

Collaborative lesson development is one of the core and foundational principles of Software and Data Carpentry, so I think contributing to the lessons should still be a part of instructor training and of check-out. I think we already spend some time in instructor training talking about why collaborative lesson development is important and why we do it, so this checkout just builds off of that.

[ChristinaLK]

I like @ethanwhite 's suggestion 5.

I'll also mention that I had a dream last week where I (?) suggested that trainees should get ready to teach a lesson by preparing their own teaching notes and then submit those for "review" as one of the checkout exercises. Ha!
Obviously, this is not feasible as is, but I wonder if contributing to the instructor's guide would be a feasible option as well. If we had a more set structure for the instructor's guide, with notes and lesson specific "FAQs", then we could have trainees read through that and open an issue if they have an additional question not addressed by the guide (which they can also take to their discussion session).

[raynamharris]

Christina. This is a unique option. It seems like a nice way to get feedback on the utility of the instructor's guides. It also nicely ties in some thoughts Erin had about making an FAQ and some thoughts that Kate and Maneesha have discusses about resources for instructors.

On Wed, May 18, 2016 at 9:49 AM, Christina Koch notifications@github.com wrote:

I like @ethanwhite https://github.com/ethanwhite 's suggestion 5.

I'll also mention that I had a dream last week where I (?) suggested that trainees should get ready to teach a lesson by preparing their own teaching notes and then submit those for "review" as one of the checkout exercises. Ha!

Obviously, this is not feasible as is, but I wonder if contributing to the instructor's guide would be a feasible option as well. If we had a more set structure for the instructor's guide, with notes and lesson specific "FAQs", then we could have trainees read through that and open an issue if they have an additional question not addressed by the guide (which they can also take to their discussion session).

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/swcarpentry/instructor-training/issues/178#issuecomment-220050469

Rayna M. Harris Graduate Student, Hans Hofmann Lab http://cichlid.biosci.utexas.edu/ ​Training & Outreach Coordinator, Center for Computational Biology and Bioinformatics http://ccbb.biosci.utexas.edu/ http://ccbb.biosci.utexas.edu/ The University of Texas at Austin raynamharris.github.io ​@raynamharris​ https://twitter.com/raynamharris

[tracykteal]

@ChristinaLK I really like the idea of having people contribute to the instructors guide (and your dream!). New instructors would be good people for this because they would see the gaps as they were working through how to teach the lesson. It could even be tied to the lesson they're planning to do their teaching check out on.

The instructor notes are in github too, so they would get a chance to interact with GitHub. Here too, I could see that they could either put in a PR or file an issue.

[anelda]

If I can chime in... Just participating on a discussion in GitHub is already breaking boundaries for many people. It's really scary to say stuff here. Maybe contributing to a discussion could also count? We don't want random comments just to get qualified, so I don't know how to measure that, but it's probably not worse than random PRs?

[gvwilson]

Summarizing discussion so far:

• Lesson maintainers find the PR portion of checkout onerous.
• But it's important to require instructors to get hands-on experience contributing to our lessons.
• Greg's initial options were:
  1. Scrap it and hope people will contribute anyway.
  2. Create throw-away lessons for trainees to submit exercises against.
  3. Have them proof-read existing lessons and file detailed issues rather than submit exercises.
• @lexnederbragt suggested copying lessons and introducing issues to be fixed.
  • @rgaiacs suggested that instead of introducing issues, we could have trainees work with a deliberately out-of-date fork.
  • But fixing deliberate errors, and having those fixes thrown away, isn't particularly motivating, and doesn't help improve our lessons.
• @uiuc-cse suggested that we steer people at the bonus lessons.
  • We're currently steering people toward less mature lessons, like the new Python lesson and some of the DC lessons.
• @ethanwhite suggested that:
  1. Scrapping the requirement entirely would be less bad than continuing to do what we're doing.
  2. Having trainees proof-read and file PRs would be useful.
  3. Have trainees fix outstanding issues that have been filed against lessons.
  4. Allow trainees to either proof-read and file an issue or fix an outstanding issue.
• @ChristinaLK suggested that we ask trainees to submit their own teaching notes and submit those for review.
  • Which works well for trainees who have seen that lesson taught, but less well for those who haven't.
• @anelda pointed out that GitHub is intimidating for people who are new to it, so we ought to accept participation in a discussion in place of PRs.

My biggest concern in all of this is resources: whatever we do cannot increase the load on maintainers or mentors, or assume that new people will magically appear. (I hope they do, but I'd like to be conservative in planning.) I would therefore like to propose a variant of @ethanwhite and @ChristinaLK's ideas:

1. Trainees must do one of the following:
   • Proof-read one of the lessons that we think currently needs work and file an issue outlining something that needs fixed.
   • Submit a pull request that addresses an issue someone else has identified (they can't file an issue and then address it themselves).
2. Trainees must also provide feedback on an issue that someone else filed or a PR that someone else created.
3. If a trainee has attended a workshop, her issue or pull request may address something in the instructor's guide for one of the lessons from that workshop.
4. We switch to these rules once there is support for tracking all of this in AMY (because we won't be able to do it by hand with the staff we have).

Please up-vote or down-vote this proposal, and then add comments below if you want to elaborate on your thoughts.

Thanks, Greg

[ethanwhite]

I like the general idea, but feel like this increases the required number of things to be done which is already substantial and there's a lot of moving parts which makes it feel... complicated. I'd suggest that 2 become a third option for 1 and 3 just be open to everyone. I.e.,

Make a contribution to a lesson (either to the lesson itself or the instructor's guide) by: 1) submitting a PR to fix an existing issue; 2) proof reading a lesson and adding a new issue describing something to be improved; 3) providing substantive feedback on an existing issue or PR.

This also simplifies evaluation because it just requires that the trainee provide a single link to their contribution.

[ChristinaLK]

Per discussion on #192, can I confirm that the rationale for this checkout piece is indeed that new instructors "know how to contribute to lessons"?

[gvwilson]

Yup.

[tracykteal]

I +1'd @ethanwhite proposal.

@anelda raises a good point too about the general fear/concern of participating in a public forum. I think this is important to address. When people are doing their checkout contribution, can we have them add CHECKOUT or NEW INSTRUCTOR to the PR or issue, so that we know as a community to be particularly friendly to a first-time contributor? That already tends to happen and people are often labeling PRs, but we could add that to the instructions for checkout, and also let the current community know best practices for interacting with new instructors.

[tracykteal]

I still think that we need to make sure new contributors are welcomed and comfortable with the process, but thinking more about it, here is not the place to address that. That's a separate discussion. I would move that we accept @ethanwhite proposal as stated. And I do see that it has 5 up votes as of now. @gvwilson, what do you think of Ethan's proposal?

[anelda]

Hi @tracykteal. I've also been thinking about this a lot - for people who aren't already comfortable with being part of an online community and participating in online discussions, how do we introduce them to it so that a larger percentage of the new instructor community can really become active contributors. I was reading [1] from the instructor training lesson [2] again recently and think we are maybe not providing opportunities that have low enough risk for new-comers to engage in over a longer period to build courage.

newcomers become members of a community initially by participating in simple and low-risk tasks that are nonetheless productive and necessary and further the goals of the community[1]

Risk is experienced differently by everyone. For someone who have very little online engagement, have never participated in a forum such as Stackexchange etc, who are not really encouraged to disagree with "authorities" (the experts in this community might be seen as authorities), the risk of "disagreeing" with (in our terms improving) what is currently published as a lesson and taught by hundreds of instructors around the world, might be seen as very high.

I don't have a suggestion about activities that fit the description quoted above but on another note..

I've suggested to our African Guides [3] that we use 13 June's Bug Barbeque [4] as an opportunity to help everyone in their group to get their PRs in on that day. The suggestion was that people discuss their PR with their guide before the 13th and on the 13th we can help them to get their PRs in. I've set the day aside to help with lessons in any case, so it won't be a problem for me to help our 23 new instructors with their PRs. They could either pick up issues from the milestones [5](with the help of their mentors) or just add new PRs that could be considered for integration after the 5.4 release. I'm hoping that this will encourage our new instructors to go for this goal date to get their PRs in and also give them a better sense of the events and opportunities that exist to help them become more part of the community.

Maybe having a quarterly dedicated day where a few people are available to mentor new instructors through their first PR would help to give them even more of a feeling of community?

Apologies for the long posts :-)

[1] https://en.wikipedia.org/wiki/Legitimate_peripheral_participation [2] http://swcarpentry.github.io/instructor-training/01-introduction/ [3] https://github.com/swcarpentry/board/issues/118 [4] http://software-carpentry.org/blog/2016/05/bug-bbq-blog-post.html [5] http://swcarpentry.github.io/SWC-bug-bbq/

[lexnederbragt]

@ethanwhite I assume trainees pick one of your 1) 2) or 3)? Slackers could get away with a quick one for 2), but I hope we'll be able to pick that up.

@tracykteal AFAICT submitters cannot add an 'instructor training' tag to their own PRs, issues or such, only admins can. Se we'd have to make sure trainees send us the link to their contribution (or add it to AMY).

[JasonJWilliamsNY]

But it's important to require instructors to get hands-on experience contributing to our lessons.

How important?

I'm not claiming it's unimportant, but just glancing over the conversation, it seems like the overall process might be obscuring purpose.

One set of goals seem to be instructor focused:

• Make instructors better teachers
• Familiarize instructors with how to contribute to lessons

Another set of goals seem to be process focused (particularly the lesson development process):

• Improve existing lessons (by fixing and or identifying issues)
• Improve existing pull requests (by review, comment)

Is there value in asking if this is sustainable in the long-term? Having everyone contribute to lessons or proof things is self-limiting (which is why suggestions in this thread lead to realization that mature lessons don't need help, or that we should propose an indefinite task like making throwaway lessons). I think there is also an implicit angle that instructor effort can be marshaled to areas where we need help with a process (this isn't necessarily bad, but it makes it easier to loose something from instructor-focused goals).

Can we think of a solution that would improve teaching ability and work if every instructor actually did it? I don't there is one single activity that fits the bill for everyone, and so one activity that contains variation on a theme (PR) may be inadequate.

If instructors need to know how to do a PR, then they should. But with more clarity on what the goal of this exercise originally was, I bet new solutions could emerge.

[ethanwhite]

@ethanwhite I assume trainees pick one of your 1) 2) or 3)? Slackers could get away with a quick one for 2), but I hope we'll be able to pick that up.

Yes, that's what I'm proposing, but I disagree with this statement: "Slackers could get away with a quick one for 2), but I hope we'll be able to pick that up."

The idea is to get people to engage with the lesson development process. Reading through a lesson and providing suggestions for improving it, even if minor, is a valuable contribution. It is also a doorway for folks who might not want to create content in the long run to see that they can contribute through issues like "I just taught this and noticed this thing that didn't work well" or "there was a typo that made the code break", etc. Our lessons would be substantially improved if we got this light level of feedback every time there was an issue in a workshop, but only a small fraction of instructors do this. I think we should encourage all types of contributions and my general reasoning throughout this thread is that we do that by defining contribution broadly and accepting that different folks will want to contribute in different ways and at different levels.

[gvwilson]

Revised proposal: we replace Part 1 of http://swcarpentry.github.io/instructor-training/checkout/ with:

Trainees must make a contribution to a lesson's content, exercises, or instructor's guide by doing one of the following:

1. Submitting a change request to fix an existing issue.
2. Proof-reading a lesson and adding a new issue describing something to be improved.
3. Providing substantive feedback on an existing issue or PR.

Trainees must do their work on one of the lessons for which we are currently seeking improvements. We encourage trainees to submit their changes through GitHub, but will accept submissions for #1 and #2 via email from those who are more comfortable working out of public view.

I believe this includes all of @ethanwhite's suggestions while addressing @anelda's concerns. It will require a bit more work on our part, since comments (for #3) can't be tagged separately as "instructor-training" (the label we're using for #1 and #2), but we'll figure that out.

Please vote on this item as shown below:

[ethanwhite]

Looks great to me. A big thanks to @gvwilson for doing the all the hard work to find a nice consensus outcome.

[raynamharris]

With 6 thumbs up, are we good to go with this new plan, @gvwilson? This modification has not yet been merged into the Day 2 Wrap Up episode or the Instructor Training: Checkout Procedure. Can I try it out on Thursday? Or should I stick with the current checkout procedure as is?

[gvwilson]

:+1: - @raynamharris can you please fold the wording into _extras/checkout.md and I'll merge the PR when you send it? We can then do a double blog post for DC and SWC. Thanks everyone, Greg

[raynamharris]

@gvwilson sounds good.

[gvwilson]

Fixed in 4df18cd21b11