Breaking up this repo

[gvwilson]

This issue is for discussion of our plans to break this repo into lesson-sized pieces. Please use it for comments on lesson templates, mechanics of importing material via Git, etc.

[wking]

On Sun, Sep 28, 2014 at 12:43:02PM -0700, Greg Wilson wrote:

This issue is for discussion of our plans to break this repo into lesson-sized pieces.

I'm in favor of this breakup, whatever it looks like (obviously: #102 ;). However, I don't really see a need to simultaneously adjust the default procedure for consuming swcarpentry/bc. In your plan (swcarpentry/site#610):

Almost all workshops will be able to set themselves up by cloning just that repository, which will (probably) contain only a couple of dozen files, then point at rendered lessons on the Software Carpentry website instead of having their own (redundant) copies.

I think redundant copies are a good thing, because they avoid link rot (or a maintenance burden on centrally hosted content, see #488 and the resultant fallout in #496 and #503). If you branch off the central workshop repository, I'd rather you had as few external dependencies as possible. For example, it would not be fun to show up to teach your lesson and learn that someone had adjusted your particular lesson the previous night.

Since we'll have all the lesson content in Git, there's no reason we can't use pulls (or submodules, or Bower, or whatever, see #102) to include a particular set of lesson content in the per-workshop repository.

The way I see it, we can make maintenance easier by breaking out the lessons and whatnot into external repositories, and then the central workshop seed (currently swcarpentry/bc) can pull in specific versions of those external repositories. We already do this for the Windows installer (maintained here 1, see 2) and the installation test scripts (maintained here 3, see [2,4]) and nobody who's using swcarpentry/bc needs to care that “upstream” for those bits is actually a separate repository. I'd postpone changes to the instructor workflow until after the maintainers had settled into the per-lesson repository workflow.q

[swaldman3]

If a group of people want to fork one of those repositories and tweak the examples to be more relevant to economics, they can do so;

This concerns me a little, because it sounds like we end up with duplicate versions of the lesson content (i.e. the stuff being taught) that happens to use different example data, which then risks either things getting out of sync, or extra management overhead in preventing that from happening.

Speculation on how to solve this, which is probably wrong because of my limited git knowledge: I think either the example data needs to be in a seperate repo to the other lesson content (probably over-complicated and awkward), or the various sets of example data need to be in the same repo as the lesson content. (maybe different "release" branches, where changes to lesson content get merged into all of them?).

we need volunteers who aren't experts in Git or page layout to act as a reality check on those who are. If you'd like to help in either capacity, please give us a shout: this will be one of the biggest changes we ever make, and we'd like everyone to be part of it.

Maybe state how to give a shout? (i.e. preferred means of contact, and to whom). Incidentally I'm not an expert in either of those things, and am happy to help so far as limited time permits. :-)

[jeixav]

Source code duplication is one obstacle I have faced when becoming acquainted with the bc repository. For example, intro.md and book/introduction.tex contain almost identical text, likewise for novice/r/00-first-timers.md and novice/r/00-first-timers.Rmd. The template in the proposal does not seem to fit the examples above. Could the proposal be revised to address examples like the ones above?

[wking]

On Mon, Sep 29, 2014 at 06:54:49AM -0700, Kristjan Onu wrote:

for novice/r/00-first-timers.md and novice/r/00-first-timers.Rmd

For the R lessons, the .md form is built automatically from the .Rmd source. In my view, that means only the *.Rmd source needs to be in the development repository, and folks can supply a built branch (e.g. gh-pages) for folks that don't want to bother building locally.

[gvwilson]

We decided a while back to include the .md's corresponding to .ipynb's so that non-Python people wouldn't have to install IPython to rebuild a web site. Now that we're splitting lessons into chunks, I think it's safer to require people to have the tools needed to build X from its raw material. Thoughts?

[wking]

On Mon, Sep 29, 2014 at 10:50:54AM -0700, Greg Wilson wrote:

We decided a while back to include the .md's corresponding to .ipynb's so that non-Python people wouldn't have to install IPython to rebuild a web site. Now that we're splitting lessons into chunks, I think it's safer to require people to have the tools needed to build X from its raw material. Thoughts?

I don't think instructors should be responsible for building the lessons at all. Of course, it should be easy for them to rebuild the lessons if they want to make changes, but my preferred workflow is:

• Source (e.g. *.Rmd) in swcarpentry/some-lesson's master branch
• Compiled version ready for Jekyll (e.g. *.md) in swcarpentry/some-lesson's gh-pages branch
• Compiled version merging swcarpentry/some-lesson's gh-pages branch in swcarpentry/bc.

The development workflow would be:

Integrate a pull request:

some-lesson-maintainer$ git checkout master some-lesson-maintainer$ git fetch origin some-lesson-maintainer$ git merge origin/pr/123 some-lesson-maintainer$ git push origin master

When you're ready to cut a release (assuming you build things with Make, otherwise substitute your build script):

some-lesson-maintainer$ git tag v1.2.3 master some-lesson-maintainer$ git checkout gh-pages some-lesson-maintainer$ git pull master some-lesson-maintainer$ make some-lesson-maintainer$ git add . some-lesson-maintainer$ git commit -v 'Compile for Jekyll' some-lesson-maintainer$ git tag v1.2.3-compiled some-lesson-maintainer$ git push --tags origin gh-pages

Then assembly maintainers like bc's merge the compiled sources:

bc-maintainer$ git checkout gh-pages bc-maintainer$ git pull --log git://github.com/swcarpentry/some-lesson.git v1.2.3-compiled

And instructors carry on using bc as they always have before.

[tbekolay]

I think it's safer to require people to have the tools needed to build X from its raw material.

I agree, and this would be preferred IMO. It's too easy to forget to keep two things in sync!

As for how exactly this looks in the end, I would be more comfortable with the type of workflow that @wking suggests if we could have significant portions of it automated through TravisCI. If it's as simple as someone running make and pushing that to the gh-pages branch, then it should be simple for TravisCI to do the same.

[wking]

On Mon, Sep 29, 2014 at 11:53:43AM -0700, Trevor Bekolay wrote:

If it's as simple as someone running make and pushing that to the gh-pages branch, then it should be simple for TravisCI to do the same.

I don't cut releases so often that it needs much automation, but if individual maintainers feel that offloading to Travis makes sense for their lesson, more power to 'em ;).

[gdevenyi]

Question re: repo breakup, should an instructor's guide for each topic live with the lessons in the repo?

[wking]

On Mon, Sep 29, 2014 at 02:30:56PM -0700, Gabriel A. Devenyi wrote:

Question re: repo breakup, should an instructor's guide for each topic live with the lessons in the repo?

Yes: https://github.com/swcarpentry/site/pull/610/files#diff-ef320ce94fa57eb02be8c4a00cd63388R238

[wking]

On Sun, Sep 28, 2014 at 02:12:15PM -0700, W. Trevor King wrote:

I'm in favor of this breakup, whatever it looks like (obviously: #102 ;). However, I don't really see a need to simultaneously adjust the default procedure for consuming swcarpentry/bc.

I'm not sure on the planned timeline, but if we want to wrap up the splitting before the new year, we'll probably have to start on the history extraction in the next week or two. Do we want to wait on a finalized lesson template before starting the histroy extraction (I think not)? If any lesson maintainer teams are interested in the pull-based integration (transparent to bc users) that I'm currently using for the installation-test scripts [1,2,3], let me know, and we can start in on the history extraction for your lessons.

[rgaiacs]

The second draft of our new lesson template depends on jgm/pandoc#1711.

[rbeagrie]

I have an operational question. How are we planning to handle code review for new lessons that will be their own repository? Suppose someone want to write three lessons on a new python library that they think will be useful, and they create them in a new repository. Under the old system, they could issue a pull request to get some feedback on their work. Has anyone thought about how this should work with the new repository organization?

[gvwilson]

Yup: if you're creating a brand new lesson and want some attention, you mail 'discuss' and add it to the list on the Software Carpentry site of "lessons you may be interested in". Think CPAN, CRAN, PyPI, etc.: you come to SWC to find packages, you go to their sites to contribute.

[gvwilson]

Decisions have been made.