roadmap to 1.0

[mr-c]

The big feature for the version 1.0 release is a commitment to following Semantic Versioning ( http://semver.org/ ) where the API is the command line interface exposed by the scripts.

Therefore the only must-haves for a 1.0 release is documentation of this API and any remaining changes needed to make it more consistent.

As the 1.0 release is also a time-based release the remaining bug fixes and features can be incorporated if there is enough time. Otherwise they can wait for version 1.0.1 or version 1.1. We've done a great job at 'releas[ing] early'; I'd like to improve on 'releas[ing] often.' http://en.wikipedia.org/wiki/Release_early,_release_often

Here are the required issues to close for 1.0:

Document and lock down the 1.0 API #16 scan through scripts/ and remove unused/unsupported scripts #118 Add documentation on how to tie into a particular tagged version #29 Update README to point to publications & citation info #49 Provide an updated list of papers and maybe blog posts, somewhere. #162 add documentation for sample-reads-randomly #192 doc/examples.txt needs updating #201 doc/guide.txt needs editing #202 doc/blog-posts.txt hardlinks to ctb's github #203 document what happened to old scripts #206 all scripts need to output their version #236 Add "when you shouldn't use diginorm" to our documentation #275 ~~environmental variables need documenting #303 git clone of the khmer repo reqs > 0.5 GiB #245 Add Torsten's rant into our docs re command line stuff #209 http://khmer.readthedocs.org/en/latest/ tracks the master branch, not releases #230 C++ code should be consistently formatted #261 Inconsistent and incomplete output filename handling #81

Bonus: Write a khmer citation handle paper => arXiv #72

Bugs that could be fixed in 1.0 but if not then added to doc/known-issues.txt:

https://github.com/ged-lab/khmer/issues?direction=asc&labels=bug&milestone=1&page=1&sort=created&state=open

[mr-c]

Pursuant to documentation related issues on this list I printed out the PDF generated by Sphinx and did a two pass edit: for global consistency and flow and then for details.

Here's is a digital version of many of the changes I'd like to make:

• [x] spell check the entirety
• [ ] add a subtitle
• [x] clarify document authorship: only CTB & Jason Pell are listed on the front cover
• [ ] document our documentation conventions
• [x] use the sphinxcontrib.autoprogram extension to pull the argparse parsers from each script so that the documentation auto-syncs with changes in each script's option
• [x] move much of the text in the docs back into the scripts' argparse --help output
• [ ] building on the above: use the Sphinx unix manual page builder to build and ship manual pages for each script
• [x] organize the bulk of the documentation into two labeled parts: user documentation and technical (dev) documentation.
• [ ] for each part: list intended audiences and expectations
• [ ] merge the following chapters elsewhere or drop them completely: "6. Blog posts and additional documentation", "8. Partitioning large data sets", "9. ktable, simple k-mer counting", "11. Miscellaneous implementation details", "16. Sequence access patterns in khmer"
• [ ] remove the incomplete '22. Indices and tables" and "Python Module Index" (for now)
• [ ] In "1.1 Introduction" explain what hashing & hash tables for the non-C.S. folk.
• [ ] in "1.2 Using khmer" make sure we mention the appropriate script name for every use case listed. They should be auto linked using sphinx magic
• [ ] split the install instructions up for user & dev use
• [ ] document how to install a specific version and do a system-wide install
• [ ] add a section under "Chapter 3. A few examples" describing and linking to the khmer protocols
• [ ] review the usefulness of examples/
• [ ] determine which scripts can make use of stdin or stdout as input or outputs. Document this and file issues if it doesn't work and it would be useful if it did.
• [ ] ensure that every script includes example usage (about half do already)
• [ ] document exactly how filter-stoptags trims
• [ ] elevate the following sandbox scripts to scripts/ or remove references to them: fasta-to-abundance-hist.py, abundance-hist-by-position.py, hi-lo-abundance-by-position.py, velvet-assemble.sh
• [x] filter-exact.py was deleted last year (ba3df5def2e8d990f7fad1c358adec55c9c81a11). Investigate, remove reference in Section 6.1 if need be (yes, do remove it as per https://github.com/ged-lab/khmer/issues/321#issuecomment-38999545)
• [ ] work with @qingpeng to update "Chapter 7: Choosing hash sizes for khmer"
• [ ] Possibly remove the "Architecture and Design" chapter as it is out of date. If we keep it then many sections need removing and updating. All TODOs within should be removed and become GitHub issues if they aren't already.

more to come in a bit...

[RamRS]

Awesome. May I work on the spellcheck, for starters?

[mr-c]

@RamRS I'd like to see you tackle something bigger next. Spell checking is an easy thing for me to do while I work on my checklist.

https://github.com/ged-lab/khmer/issues/86 needs a test case written + fix https://github.com/ged-lab/khmer/issues/247 needs a test case (tricky!) + c++ fix https://github.com/ged-lab/khmer/issues/152 the screencast still needs to be made regardless of the side conversation https://github.com/ged-lab/khmer/issues/253 another c++ project

[RamRS]

The spell-check was just the salsa dip. I'd of course have something more interesting for the entree!

I'm going to try and tackle the python test case first. Also, the C++ ones look quite scary to me, might need quite a bit of hand-holding - would that be OK?

[RamRS]

Also, I have a branch addressing #328 . If you think the issue is relevant, I'll fire up a PR.

[mr-c]

More notes from my documentation review:

• [x] In "Chapter 12: Development miscellany": talk about pylint
• [x] build a Makefile target to do code style and correctness checks on all files modified in commits that are a part of the current branch which haven't been merged into master yet. Add it to the docs
• [ ] Acknowledge the in-flux nature of the codebase; if a contributor modifies a file then they are responsible for making the entire file well formatted (using autopep8 / astyle / their hands)
• [ ] link to https://github.com/pjotrp/bioinformatics & file issue for outstanding items
• [ ] write up motivation for the coding standards, use Software Sustainability Institute articles as they are useful
• [ ] move the git/github strategies before the code review section
• [ ] talk about the two types of pull requests & workflow including requesting code review and who does the merge
• [ ] mention specific git techniques including rebasing to clean up commits
• [x] doc how to calculate code coverage on ones own
• [x] in "13.5 Command line scripts": argparse, examples in documentation
• [ ] configure pylint to respect our script naming convention + doc that if needed
• [ ] specific exit codes can be 2-255 but their meaning needs to be documented. I don't know how much demand for fine grained exit codes there are, though
• [x] add versioneer to "13.7 Upstream sources"
• [ ] "Chapter 14: Deploying the khmer project tools on Galaxy": we also support filter-abund.py
• [ ] fix numbered list in 14.2
• [ ] "Chapter 17: How to make a khmer release": add step to scan for TODOs BUGs and FIXMEs that need to be moved to Github
• [x] automake checking for updates to ez_setup.py & versioneer
• [x] versioneer installs via pip now
• [x] document batlab
• [ ] add version check to each stage of the release check; replace interrogative comments with statements
• [x] drop the redundant section 17.1 "Upstream sources"
• [ ] document that the setuptools version requirement is enforced by setup.py

[mr-c]

• [x] document ChangeLog format and expectations http://en.wikipedia.org/wiki/Changelog#Format
• [ ] document .gitignore usage http://stackoverflow.com/questions/1753070/git-ignore-files-only-locally

[mr-c]

@ctb or others: is there a suggested replacement for filter-exact.py? (deleted in ba3df5def2e8d990f7fad1c358adec55c9c81a11 as part of the ReadMask removal last February)

[ctb]

On Sat, Mar 29, 2014 at 08:12:23AM -0700, Michael R. Crusoe wrote:

@ctb or others: is there a suggested replacement for filter-exact.py? (deleted in ba3df5def2e8d990f7fad1c358adec55c9c81a11 as part of the ReadMask removal last February)

filter-abund, I believe.

C. Titus Brown, ctb@msu.edu

[mr-c]

Okay. Since that is used in the khmer-protocols I'll drop it from https://github.com/ged-lab/khmer/blob/master/doc/blog-posts.txt#L18

[ctb]

Remaining components of this issue should be moved to a new issue :)

[mr-c]

Remaining components moved to #1091.