Standard documentation format for Node.js

[Qix-]

cc @sindresorhus and @dthree, from dthree/wat@419654cfa180e321f2ac1a0d031ff195ec26d2f9.

---

The javascript community needs a standard, authoritative way to document libraries and modules. Markdown documentation is great, and it works, but actually using it other than reading is nearly impossible - nobody does it the same.

Preferably something that can be parsed, used, transcribed (to Markdown, JSON, HTML, etc.) simply because it's a structured documentation format.

It should also be human-readable by itself (similar to Markdown).

---

Discuss!

[Qix-]

JSON will be awful for this. If we write a DSL, I'd be happy to write the PEG.js parser for it (I love writing PEG parsers).

[sindresorhus]

Can you include something about what the other programming communities do? Go, Python, Ruby, etc.

[Qix-]

@sindresorhus I don't know what any of those specific communities do to be quite honest. I know that Java people use Javadocs (a real, parsable, built-in thing) and usually convert them to documentation using some stock tools or a tool like Doxygen.

Native people usually use manpages, or Doxygen as well. Same with PHP - usually doxygen with PHPDocs.

[iamstarkov]

we as community have a JSDoc for a while, if we are talking about having documentation in the code.

[Qix-]

@iamstarkov Sure, viable enough, and I know we've had JSDoc. My personal feelings towards it have always been a bit "meh" though. @sindresorhus why don't more modules use JSDoc?

By the way, WebStorm and other IDEs that work with javascript support JSDoc by default, offering helpful hints and such.

[RangerMauve]

I personally don't like JSDoc because some of the modules I make are too dynamic to really work well with it. I prefer just documenting apis with small examples in markdown where I can show how things should be used.

[dthree]

Perhaps a smart markdown parser that knows the general patterns people use in writing up their apps. It won't be perfect, what I was looking for was not a complete docset generator. JSDoc is that, so if it's not adopted, creating another one isn't going to get them more adopted. I think the thing inherently wrong with JSDoc is that it's too impersonal IMHO.

I thinking parsing markdown is important as what you see in the README are the salient points that the author wanted you to see. It has the human touch that JSDoc will never have.

READMEs are also probably the most maintained documentation a library has.

[RangerMauve]

In terms of markdown, there's DaSpec which might be similar to what you're looking for.

[dthree]

Thanks. Checking it out.

[RangerMauve]

Also, for code examples there's evalmd

[dthree]

evalmd is interesting - not sure how I'd use it, but definitely an interesting concept.

[RangerMauve]

Yeah, I was thinking it'd be cool for unit tests where you can have a description of what the test is doing and why it's there, and then having it get evaluated.

[dthree]

Yeah I can see that. I'm going to take a stab at a smart markdown parser. DaSpec looks pretty promising, but I have to get my feet wet.

[sindresorhus]

Might want to look into @wooorm's mdast parser.

[dthree]

Thanks

[wooorm]

Thanks @sindresorhus for the ping.

A few weeks back there was some movement on a related topic, originating in the feross/standard repo by maxogden. It’s called standard-readme. The project is a bit stale currently. But there is useful stuff in the issues (especially, related to API syntax).

Also: mdast reads markdown very well, and it’s relatively easy to convert to something else due to its syntax tree!

[wooorm]

P.S. I’m not sure if we’re talking about documentation in markdown in comments in JS, or documentation in markdown?

[dthree]

@wooorm we're taking about the latter - parsing readme / api / wiki markdown for general documentation.

mdast looks really good!

[ruyadorno]

relevant to the discussion, back from 2011: http://blog.millermedeiros.com/inline-docs/

unfortunately his mdoc documentation generator never really took off, but it has a really nice interface once converted to html, you can see an example of a very simple generated doc here

[AdrienFromToulouse]

Hello all, sorry for the may be stupid question but why woudl you like to use markdown?

From what I know npm takes the README from github as the documentation of the pkg. That is why the documentations could sound unconsistent.

By forcing to comment the code itself with a jsdoc style syntax and then we use the tool dgeni done by the Angular team to generate the templates, we might "solve" the issue.

We can tweak it to suit the specific needs of nodejs if needed.

So after each publish to npm the doc is regenerated with a gulp task. The module could not be pushed if any errors is returned from the doc build.

Am I on the good track? Does it sounds relevant to you?

[dthree]

@wooorm @Qix- @sindresorhus

I'm actively working on a smart readme / markdown parser that builds API docs using mdast. wooorm was talking about the standard-readme project above, which looks promising but is unfortunately stagnating.

Since I am doing a ton of work parsing readmes, do you guys think it would be a good idea for me to create a sort of mdlint project that sets a standard for a readme in terms of information, grammar, code linting as well as API doc standards? For example,

1. It could use Alex for proper word usage,
2. A general grammar language lib for general mis-spellings,
3. An existing project such as this for linting code,
4. The work I am doing to standardize API docs,
5. Validate test badges, etc.
6. General agreed upon standards, based on best practices learned over time; such as an introduction, table of contents and license section.
7. (ideas accepted)...

I think this would be cool, maybe we could team up if it interests any of you. Providing an actual lint library could perhaps push community agreement over the edge to provide standardized readmes for the first time. which could then be a stable and maintainable source of documentation.

With this, a readme could get a badge or recognition as being linted or standard md compatible, and wat could automatically generate high quality API docs in addition to local access to the readme with no human intervention.

What do you guys think?

[sindresorhus]

create a sort of mdlint project that sets a standard for a readme

Have you seen mdast-lint?

What do you guys think?

Sounds awesome, but it's going to be a lot of work. I think we should first focus on standardizing the readme API docs. That is what matters for parsing. The rest of the readme is just for consistency and quality, and will be a lot harder to standardize.

Obligatory xkcd:

[dthree]

Haha - amazing! For sure. Yeah, I was thinking (eventually, not now) just combining all of the work that's been done in to a single standard, it doesn't have to be a new one. I.E. Alex, mdast-lint, the most popular api standard, etc. mdast-lint looks like the perfect starting point. And I agree with you: standardizing the layout of a readme isn't necessarily necessary. I was more referring to the types of things mdast-lint handles, things like ## Header instead of ##Header because some md interpreters like npm don't recognize the latter, etc. etc.

---

But yes, less talking for me and more coding. I'll get back with you all when I have something workable to show.

[Qix-]

I was waiting for that XKCD to show up.

[stevemao]

https://github.com/verbose/verb https://github.com/zeke/package-json-to-readme

[sindresorhus]

@stevemao Instead of just posting random links, can you elaborate on what they are and why you posted them?

[stevemao]

They both generate a readme for you from package.json. With verb you can even define a template. According to the title of this thread you wanna standardise docs so I guess the opinionated, uncustomisable package-json-to-readme might be ideal to achieve this.

[dthree]

Mm. package-json-to-readme really doesn't do much other than boilerplate, and it's not totally real either. I've never seen someone include dependencies and dev-dependencies on their readme. Ironically, it doesn't even fully document what it does with its own auto-generated readme, e.g. where the usage data came from.

Verb basically seems to aggregate data from other markdown files, with a few fancy template options. Doesn't really solve the question at hand.

[dthree]

@sindresorhus @Qix- I think we need to discuss:

1. Why jsdoc doesn't fly, and why can't it become the standard doc format for Node,
2. What would need to be different about it in order to be acceptable.

In other words, we need an ideal scene of what we're trying to accomplish.

[sindresorhus]

1. I prefer writing docs in the readme, not as verbose and awkward code comments.
2. Standard format for writing API docs in markdown.
3. I don't want to have to have a build system for generating docs. A web service that could just read the readme would be preferable.

[Qix-]

I think the culture of JavaScript developers (web, in general) is quite a bit different than other areas of software engineering - and some of those differences are seen in documentation.

JSDoc was meant to mirror other forms of documentation. JavaDocs, PHPDocs, etc. Even low-level languages such as C and C++ have documentation inline with the source. Most of those languages that require external documentation do use some sort of generator.

In the JavaScript world, though, this isn't the case. NPM doesn't show you the code, it shows you the readme. Github shows you the readme first. Markdown took off and makes the readmes look pretty - and includes the ability to show code samples, with proper formatting and highlighting.

The whole idea behind Node and this whole JavaScript movement is to have low latency development (for better or for worse) and in that comes this weird disregard for what the code actually does (for better or for worse). Rarely do people actually look at the code to see how it functions - they rely on docs.

JSDocs also have a disadvantage - they don't really matter. For me personally, writing JSDocs back when I thought they were the :bee: :poultry_leg:, writing types was ambiguous at best. Javascript is a dynamic language, so types don't really work well on JSDocs (argument lists aren't enforced, duck typing is a thing, etc) - but types were required by most things that actually used JSDocs.

As much as I'm a young "old schooler" and would much rather look at the source for documentation, I don't think the spaghetti-file style of the Node-iverse is a great place for in-code comments. Searching through the code would be impossible. The language just isn't built for it.

I'd almost go so far as to say this is a side-effect of using a scripting language for software development. But I'm not deep enough into the philosophy of things to really say that with confidence.

What would need to be different about it in order to be acceptable.

It would probably need to not be javascript. :dancer:

---

Writing readme docs is easier thanks to Markdown. It's the smalltalk of documentation, and with only a few added symbols you have richly formatted documentation. It's a no brainer for web development.

The unfortunate thing is that there isn't a great way to do meta stuff in Markdown - god forbid we manually specify anchors, even.

What if we used comments? They don't show up, they don't go away, and they're even used for meta stuff on StackOverflow.

I can't imagine exactly what it'd look like, but maybe something like

<!-- api: exports.fooBar() -->
### `.fooBar()`
Do the foo

<!-- return: undefined -->
##### Return Value
Does not return anything

It doesn't really matter where the comments go, so long as the relevant information about an API call come after an associated <!-- api: ... --> comment, for example.

Just a thought.

[dthree]

Thanks for your answers, guys. I love the idea of using readmes / markdown as the doc source.

I don't know on the metadata: is there really a need for metadata? Headers themselves can really tell you a lot, so long as they are on a standard.

All of this may be as simple as agreeing on a standard, comparing it to a bunch of libraries to ensure it works across the boards, making a linter / mdast-type formatter for it to enforce that standard (just the API, mind you), and then getting the community to adopt it.

[Qix-]

How do you know which readme is following the standard and which isn't? A lot of readmes will look like they are. I, for one, would rather hear Hey, you have to just add a few comments to your readme to be compatible rather than Hey, you need to restructure your docs a bit in order to be compatible.

[dthree]

Well, first, we need a standard! I don't think people are going to need big refactors.

if we agree on something, maybe a good way to go is to create an organization and a repo based on the standard, and have it give multiple examples of (fake) libraries that implement the standard in an example folder or something.

The readme could then simply list all developers and projects that implement the standard, i.e. it can advertise them so people want to get their project adopting it. You, myself and Sindre, and anyone else interested in this thread for example, could do this with all our repos and add it to the list, and then we just start spreading the word.

---

In terms of the content of the standard, I don't think it has to be very complicated. Literally, just a uniform way of displaying method / property signatures, then perhaps a quick description, code example and then following details. Or whatever.

---

That wouldn't take a lot of work to get others to adopt that.

[Qix-]

That wouldn't take a lot of work to get others to adopt that.

I think you'd be surprised ;) Many people are stingy on how their docs look and feel. What if we wanted to, in the future, support something other than MarkDown? What if the format of markdown is governed by another standard? Comments are a pretty non-intrusive way to convey the exact same, parsable information without asking to change the appearance of your documentation.

[dthree]

So you're saying double up the API with md comments? LIke:

<!-- api: lib.foobar(my [, preference[, for[, optional[, args]]]]) -->
## lib.foobar(my, preference?, for?, optional?, args?)

[Qix-]

<!-- api: lib.foobar() -->
<!-- arg my: Some argument -->
<!-- arg [preference]: Some argument description -->
<!-- arg [for]: Another argument description -->
<!-- return: Returns some value -->

[dthree]

Ah ok. It's an idea. It's almost yaml. Do you expect someone to declare the whole API at once, or throughout their markdown?

[Qix-]

Eh kinda, just because of the :. The mechanics wouldn't be YAML at all.

Probably throughout, especially if we wanted to declare code snippets as <!-- example --> but that's up to whatever team would make the "standard".

[RangerMauve]

So how is it going to be that much better than having comments in the code, then?

[Qix-]

@RangerMauve My earlier post explains why. Sifting through code with Node projects in particular is rough.

---

Keep in mind this is just an idea. My word definitely isn't law ;)

[dthree]

@sindresorhus what do you think about this?

[RangerMauve]

@Qix- So it'd just be easier for a tool to find? Why not just have a file format for declaring the API? Maybe JSON, and then you could have something like api.json in the project root as well as the package. You could have all the structured fun of json to have the format you want for the api, have it be easily located, and not clutter up your markdown with a bunch of comments that nobody will bother to read.

[dthree]

"foo": {
  "bar": {
    "type": "method"
    "args": [
      "my": {
        "optional": false,
        description: "Some argument."
      },
      "preference": {
        "optional": true,
        description: "Some arg description."
      },
      "for": {
        "optional": true,
        description: "Some arg description."
      },
      "optional": {
        "optional": true,
        description: "Some arg description."
      },
    ],
    examples: [
      {
        header: "Using this to do so and so:",
        code: "err... i dont know how this would work in JSON"
      }
    ]
  }
} 

A little bloated...

[dthree]

@RangerMauve of course no one is going to read the .md comments - the point is to use a tool to then build the docs.

[Qix-]

@RangerMauve agreed with @dthree - that's a trainwreck. Further, the comments aren't for people to read, they're for machines to read. :)

[dthree]

@Qix- I'm not really into a million <!-- comments --> either. Hmmm....

[RangerMauve]

Well, maybe JSON isn't the best. But I still think that having it in its own file would be better than cluttering up the source of the markdown. If it's not for people to read, and it's made for this specific tool, then why not just have a file that the tool looks for? Is there any benefit to anyone for having it inline over a separate file?

[dthree]

Maybe YAML?

[Qix-]

@RangerMauve @sindresorhus mentioned, and I agree, yet another "dotfile" type of thing in a repository is just making an already bad dotfile problem worse.