Auto-generated, published and aggregated user and API documentation

[aschrijver]

(NOTE This showcase is part 1e of Positioning, vision and future direction of the Dat Project)

This proposal was created to follow up on @joehand comment in PR for new awesome page where I had removed the ugly folder icons, not knowing they were technical markers:

This will break our docs builds immediately. There are probably better solutions for aggregating docs but we have something implemented that is working for us.

Current documentation procedure

• README's of documented dat project modules contain the official user + API documentation
• after changing API signatures in js code the developer updates the README accordingly
• documented projects are marked with a special folder icon in dat-awesome list
• at build time Docs project reads marked entries from dat-awesome to aggregate README's

Weaknesses of this approach:

• updating README after code changes is easily forgotten and error-prone (scan through code)
• reviewing README requires time-consuming lookups in code
• awesome page presents non-descriptive folder icon (a technical marker) to visitors
• should not forget to rebuild Docs after a change to one of the documented modules
• README's of documented projects are 'hijacked' by Docs project:
  • cannot add any project information that is not directly relevant to docs.datproject.org

Proposed approach

(its the idea that matters, not the exact details)

• every documented dat project module is fully self-contained
  • (optional) has a docs folder for user manual markdown
  • (optional) has an examples folder with working code, including snippets for use in docs
• code has JSDocs (or similar) inline comments to document the public API
  • If not having docs + examples folder then user manual is defined in top comment in index.js
• TravisCI build of a documented project:
  • auto-generates markdown documents
  • publishes markdown to the project's gh-pages
  • copies the markdown to the Docs project gh-pages at the right location
• README of a documented project:
  • has a link to its own user and API documentation
  • provides text and links that explain how the project / module fits into the larger ecosystem
  • has project-specific information not relevant to docs.datproject.org
  • e.g. contribution guidelines, code of conduct, future plans, thanking contributors, etc.

Advantages:

• documentation goes in one easy flow together with coding, no context-switching
• review is similarly easy, just review the code
• awesome page is no longer polluted with technical markers
• individual README's can be used for non-official documentation again
• much less chance to forget documentation -creation, -generation, -publishing
• docs.datproject.org is never out of date

The only downside being a couple of extra seconds added to the TravisCI build of a module project

---

For similar reasons I have already tried auto-generating of protocol buffer specification docs from hypercore-protocol/schema.proto.

This because I found 3 of the same specification docs, 2 of which where outdated (off-topic: if you version the specification, they would just be old versions)

Currently the generator creates nice markdown output, but still WIP. See:

• Auto-generate protocol documentation from inline comments in schema.proto
• Example: https://github.com/aschrijver/hypercore-protocol/tree/feature/document-generator-new

---

Next part: A message-based system on top of raw streams

[aschrijver]

There is much hesitancy to using inline comments, especially JSDocs:

[19:06] <barnie> if you could look at the docs proposal and give your opinion. I still have time to help with the changes
[19:07] <jhand> In general I think we are hesitant for auto generated docs stuff. I tried it in dat-node and wasn't happy with JSdoc, need to remove those comments.
[19:07] <barnie> do the boring stuff, so you can code ;)
[19:07] <barnie> the separate readme as the doc is also not working very well
[19:08] <jhand> I've not seen a good example that is easy to read
[19:08] <barnie> you mean of an auto-generated JSDoc?
[19:08] <jhand> And easy to maintain for people who don't do JSDoc all the time
[19:08] <jhand> Yes
[19:08] <barnie> there are alternatives besides jsdoc.
[19:08] <barnie> the gist of the proposal is having the docs inline with the code they refer to
[19:09] <barnie> jsdoc is just the most used one currently, i think
[19:16] <jondashkyle> barnie:  yea i personally find inline docs unhelpful, and if anything a distraction when becoming familiar and working with a codebase.
[19:17] <jondashkyle> much prefer docs authored for readability and have a nice cadence and flow from intro to extensive. i think choo is a super solid example of that: https://github.com/choojs/choo
[19:17] <barnie> i personally find reading jsdoc comments helpful when working in the code
[19:18] <jondashkyle> ¯\_(ツ)_/¯
[19:19] <barnie> problem is that api docs now are often outdated and require system-icons in dat-awesome for the aggregation
[19:20] <barnie> jhand: instead of parsing the documented modules from awesome-dat README it could also parse from a separate file
[19:23] <pfrazee> barnie: yeah but docs are docs. If they fall behind in the readme, arent they just as likely to fall behind in the code?
[19:24] <barnie> well, its an concession, if we don't want inline jsdoc or similar, then keep the procedure as-is but do not derive the list of project README's to parse from the awesome-dat but from some other location
[19:24] <barnie> parse --> i mean aggregate
[19:25] <barnie> the docs project does this in the package.json
[19:25] <barnie> https://github.com/datproject/docs/blob/master/package.json#L9
[19:26] <pfrazee> ogd: just reviewed the multiwriter diff on the docs, so far looking good
[19:30] <barnie> pfrazee: in https://github.com/datproject/discussions/issues/73 i described a number of pro's/con's. inline comments go in one flow with both coding and reviewing and the text is right above the code
[19:31] <barnie> instead of updating a README after the fact of coding and separate file checking + review process (plus lookups)
[19:36] <jhand> barnie: not really understanding why the parsing of awesome-dat for modules to bring into docs is an issue. We can add it as a file to docs too, doesn't matter to me. Feel free to PR that.
[19:37] <barnie> the only issue is that there should be strange icons that awesome readers do not understand. they are system-markers for the docs project, and they are ugly-looking :)
[19:37] <jhand> I'd agree with pfrazee. Its not an issue of *how* docs are being written. More than they are falling behind. I'd think we're less likely to keep inline docs that require special syntax updated.
[19:38] <barnie> i can understand. i thought jsdocs where quite common. i am used to javadocs myself.
[19:39] <barnie> maybe there is an inline markdown commenter..
[19:40] <jhand> Also jondashkyle has a point that inline docs make working with the code harder.
[19:40] <jhand> barnie: I agree we need to improve our docs situation. I doubt moving everything to inline docs will be effective in doing that, knowing how things our modularized and how we work.
[19:41] <barnie> yea, no problem
[19:41] <jhand> re: how we pull in the readme from awesome-dat, I am happy with anything that works =).
[20:20] <@karissa> Jsdocs are great, I use them on my projects. They're industry standard as far as I can tell
[20:21] <@karissa> it's nice because you're reminded to update the docs when you update the code cause they are next to each other
[20:24] <@mafintosh> whatever works for the individual hacker
[20:24] <@mafintosh> thats my mantra

[aschrijver]

@karissa I agree with you, being used to Javadoc myself. In general I consider Dat Project modules as a toolkit with 'official' API's. For an API good QA is important! Users of the modules should not look in the code, and should be able to implement their apps with help from good, concise api documentation. The current separated coding + docs process makes that difficult to maintain.