A new permanent fork of ESDoc is forthcoming Q1 2017

[typhonrt]

This current effort has been a proof of concept fork of ESDoc. At the time, June 2016, ESDoc was stagnating (and will likely do so again!). ESDoc is the sole effort of one developer / maintainer who is not exactly interested in collaborating with others though this is mainly due to his lack of being able to communicate in English (he's from Japan). I made this fork to prove how little time it would take to upgrade ESDoc. The maintainer the next day opened a progress thread duplicating the major features in this proof of concept. It took him ~6 months to complete the work.

Recently, discussion has revealed that a permanent fork of ESDoc is pertinent. I will be starting with the latest official ESDoc version 0.5.x shortly and will replace this repo with the updated permanent fork Q1 2017. There will be several major differences from technical underpinnings completed before this occurs such as splitting up ESDoc into multiple repos and other structural changes to improve the architecture in addition to rolling previous TyphonJS ESDoc plugins to the core where applicable including proof of concept work which has been completed in the branches of this current repo.

A major distinction of the upcoming fork is that there will be an open governance model applied such that collaborators will be invited to participate directly once significant PRs are submitted demonstrating an understanding of the architecture. All applicable issues and improvements will also be handled in an expedient manner.

Please stay tuned as the future is bright. I wish the maintainer of ESDoc well and hope he continues to work on his version of the project. The new permanent fork will just move faster and further in addition to accepting best of breed efforts from the original ESDoc.

[ratson]

Hi @typhonrt, thanks for stepping up for a more community-friendly fork, really appreciated. I would like to share my 2 cents before you started,

• It is better to create a repo without forked from link, as Github does not count contributions for forks until commits merge to upstream.

• Maybe a new name / org, which helps understanding it is a another documentation tool.

• Since the plugins are likely constrained by core, would it be better to have a monorepo? Babel uses lerna to manage that.

• Permissive license, I see this repo is released in MPLv2, it maybe better to follow the same MIT license as ESDoc.

[typhonrt]

Hi @ratson. Glad you found the current proof of concept fork! Indeed I plan to start a fresh set of repos that aren't forked directly from ESDoc and thanks for the heads up regarding how Github handles contributions in unmerged forks.

About a new name... From a traction angle it makes 100% sense to come up with a new name. I've spent some time thinking about it and all of the obvious esdoc / jsdoc / documentationjs combinations are out there already. Like io.js perhaps it's best to just come up with a unique even non-documentation oriented name to help with visibility. I'm certainly open to suggestions in this regard.

I'm familiar with lerna and have at least read about that particular solution. Regarding mono-repos I'm actually not a fan in general and have been developing tooling to support multi-organization / repo efforts which is how the larger TyphonJS effort is organized. An example of this is typhonjs-npm-build-test which pulls in a complete dev toolchain from one dev module import in package.json which is standardized across all repos. Another example is typhonjs-node-scm. Particularly typhonjs-github-inspect-orgs which provides a utility to perform compound Github queries across orgs / repos collating all of the data into an actionable form. I have an unreleased management tool presently which uses typhonjs-github-inspect-orgs to bulk checkout or update one or more orgs and associated repos in addition to setting up WebStorm projects for each automatically. This alleviates the burden of potential contributors from setting up individual local repos / IDE projects separately. One final tool which is yet to be publicly launched before things kick off is also additional Gulp tooling which can compile dependent sub-modules and show all relevant tasks for these modules automatically in any given project. It's easier to pull this off with Gulp tasks than NPM scripts as things go for auto-discovery with separate tasks in WebStorm. This makes it easier to work in a core module with dependent repos / modules. I've been refining all of the multi-repo tooling with the typhonjs-node-escomplex effort, so there is at least some precedent / existing approach.

A final note is that what kept me away from working on open source efforts since August this year is over and I'm lucky to be able to tackle the permanent ESDoc fork full-time+. So, progress should be swift in the initial effort though I am going to spend about a month finishing typhonjs-escomplex first. ETA for all of this is roughly February, but no later than March.

[typhonrt]

Ah yes @ratson re: your final point of a permissive license. Regarding building communities and successful long term projects I take a lot of my reasoning for the MPLv2 license from the late Pieter Hintjens. While a long post here is an interesting treatise on community development from him. I'm adopting a slightly modified C4 agreement for all TyphonJS governance. These modifications are specific process details to style and source code conventions for the most part.

As a self-funded small player / entity I'm going to commit significant full-time+ effort to get things going to a fully professional level and with continued maintenance with or without a growing community initially. I have a lot of aspirations to really take the ESDoc architecture to the next level including supporting Flow and even Typescript in addition to new CI tooling for automatically generating up to date websites for documented orgs / repos and that's going to be a lot of time / hard work.

An interesting architecture direction that I already have working in typhonjs-escomplex is that the core implementation is actually a collection of internal plugins each of which can be replaced in addition to any user facing plugins. I'm going to evaluate this strategy as it may apply to rearchitecting ESDoc such that there isn't a monolithic core of hard coded functionality. There will be internal plugins which don't need to be modified for normal usage, but can be if so desired along with user facing plugins similar to the current way ESDoc functions. By taking this modular approach the core just becomes one big dispatch mechanism. Any part could be rewritten with any license desired. MPLv2 just protects the investment and encourages others to upstream efforts to the authoritative version.

The MIT license fits how the original ESDoc author is running his project with sporadic updates and as he stated for fun. I have the utmost respect for him and the initial architecture that he released into the wild.

I'd like to build a definitive and well maintained documentation tool which encourages experimentation at all levels without potentially leaking all that effort in the long-term. For me it's the difference between an adhoc open source effort or an intentional one. I think the highly modular internal / external plugin approach minimizes any perceived downside of a weak copyleft license like the MPLv2. I hope that this makes sense!

[raveclassic]

Great news, I'm really looking forward to seeing this happen! By the way, just interesting, how are you going to support typescript? Isn't it a completely different parser?

[typhonrt]

Very carefully! 😵

It is a different parser and a separate AST. I wished there was better documentation (I haven't found any at least) regarding the AST found in types.js. This compared to at least a spec with Babylon. Examining where Babylon and TypeScript AST nodes overlap will be useful at some point. As far as AST traversal goes sticking to the generic typhonjs-ast-walker versus the Babel specific babel-traverse used by the current mainline ESDoc release allows either tree to be traversed generically. This repo / proof of concept fork uses the former.

Roughly speaking though from an architectural perspective right now things are hardwired with static methods and such all over and between potential internal logical boundaries. Moving to an event passing mechanism that has the ability to return results allows each internal plugin / module to register & respond to events. If you want to parse a file instead of importing ESParser or heaven forbid doing something wonky like dependency injection (yesh!) and subsequently invoking the static method ast = ESParser.parse(filePath); an event based approach would be ast = mainEventbus.triggerSync('parse:file', filePath); This is preceded by when the parser module is loaded during an initialization hook as it could be passed mainEventbus and subsequently register as such, mainEventbus.on('parse:file', this.parse); Want to support Babylon load the Babylon parser module or if Typescript is desired then load Typescript version. There are all sorts of divisions that can be extrapolated from the current hard wired approach.

From a documentation generation perspective an intermediate step is to support Flow via Babylon first and add the Flow typing info and all other aspects to the output documentation result. There is a lot of overlap between the kinds of things Flow and TypeScript representatively add. This at least will provide an incremental step towards TypeScript parity as far as output results are concerned.

It's a lot of work and a lot of tests to add! Ugh especially sans formal TypeScript AST spec document separate from the code.

[raveclassic]

Yeah, agree, ts doc is terrible. Even their official examples of using compiler api are not quite working.

Regarding what you are saying about possibility of combining docs/typings from jsdoc, flowtype and typescript in one tool - well, this is going to be an outstanding feature! It is a real problem for now which can't be solved completely because of different languages mixed in one codebase. Although ts supports --allowjs, it is not enough to cover all code with documentation.

There is a lot of overlap between the kinds of things Flow and TypeScript representatively add.

No doubt it is but there are also new features of TS>=2.0 like discriminated unions, type narrowing, mapped/lookup types and others which can't be reproduced in jsdoc/esdoc/flow environment. Indeed this is going to be a huge amount of work to adopt all technologies in one tool.

Some words about naming. If you are going to implement a very generic parser or different drop-in parsers which will recognize both js, js+flow, ts, maybe a name for this tool should also be generic? Something like just documentation with an ability to plug in different language parsers (in future) but still rely on jsdoc-style comments where necessary?

[typhonrt]

Well, @raveclassic, your involvement will definitely be handy when things get rolling for TypeScript support and testing for all the latest features of TS!

Can't use 'documentation' as documentationjs already has taken it. I recall reading somewhere how they bemoaned the name as search engine rankings / discoverability are hard to break through with documentation being the primary name.

There is the fallback of typhonjs-doc for the NPM package name and tjsdoc perhaps for the CLI. typhonjs-doc also gets no hits for the exact phrase in Google and will be picked up as top link and any variant of typhonjs docs will also score highly and bring up a bunch of well TyphonJS stuff. After all Typhon is the father of all monsters w/ 100 heads (re: modular support for various language support). Of course other name suggestions are welcome as there is some time still (2-3 months) for initial launch.

---

Regarding Flow support it does provide a nice stepping stone to implement all of it's supported functionality and generate the static HTML docs with the new functionality output. Sure TypeScript will have more to deal with but when it comes to hooking things up there will be that initial cross-pollination of Flow examples which should match TypeScript generated output and this will be very handy for testing / compatibility concerns. The last step then would add the TypeScript specific functionality / output.

---

Another interesting tangent is whether it makes sense to make JSDoc annotations / Google Closure Compiler the eventual goal. I never really liked the bespoke compatibility of the ESDoc effort. First things first though is get what currently exists working via the architecture refactor then think about these larger compatibility concerns. It does look like TypeScript has support for quite a few JSDoc tags in the AST itself versus needing regexp the comment blocks.

[typhonrt]

So I have the new version of the Backbone eventbus up here: backbone-esnext-events

I've been doing a bit of updating Backbone for ES6 and now ES6+ design via typhonjs-backbone-esnext, but just the events functionality is available presently. I'll eventually be updating typhonjs-backbone separating things into modules that can be independently used. A main goal too is to remove the Underscore dependency with backbone-esnext in favor of using ES5.1 / ES6 / and ES6+ functionality.

For instance there are no dependencies with backbone-esnext-events / Underscore has been removed. This allows it to be used as a standalone module providing just eventbus functionality without pulling in extraneous modules. This is what I'll be using to re-architect ESDoc for modular construction as mentioned above where all internal plugins communicate over a main eventbus rather than direct method invocation. There is added functionality to the Backbone version adding methods triggerDefer (defers triggering to next clock tick), triggerSync (synchronously returns a value from invoked code), and triggerAsync (asynchronously resolves returned values via Promise.resolve() or Promise.all()).

Things are available via NPM here.

[typhonrt]

In one solid day finally hacking on ESDoc 0.5.2 I've made significant progress in moving things forward. I've only committed the auxiliary supporting eventbus related repos and NPM module so far. It's easy to now share an app level global eventbus between ES6 modules at the file level within the same repo or beyond that doesn't need a callback passing in the instance. Taking advantage of the ES6 module system I've been able to drastically re-architect the core functionality of ESDoc, still in a single repo, to provide the same hard-coded API in addition to adding a module level event system such that the primary API used now is purely implicit message passing oriented versus explicit direct method invocation. Thus detangling most of the core hard-coded infrastructure which was full of singletons at the class and ES6 module / file level and such. This pattern will serve just as well when fully modularizing everything into separate NPM modules / repos.

I'd like to get this effort committed into Github in the coming week or so and have created a new typhonjs-doc org and private repo. This before the whole public open sourcing and application of the C4 process as I'll be doing rapid direct commits until things go public. I'd be glad to pay the monthly cost per user (~$9) to get any early interested parties in on the game. So far both @ratson and @raveclassic are hip to this from commenting on this issue, so no problems facilitating early access to either of you. I'm not going to open this up until I get another few more solid days on it separating the common code from the Babylon specific code then hopefully adding Typescript counterparts to existing functionality.

There are several other little cool things to report. For instance I've overhauled the built in @external & @typedef loading functionality such that a plugin can now generate programmatic in memory code additions (instead of all file based before / re: MemoryDoc is now a thing!) which are parsed for external and other typedef doc tags. This allows users to now access new plugin callbacks onHandleExternal and onHandleTypedef to programatically add @external and @typedef references allowing them to be standardized in discrete NPM modules (and thus reused across repos!) versus file based additions directly in any given repo. IE the previous file based built in external loading is now plugin oriented. This solves https://github.com/esdoc/esdoc/issues/287 for instance.

So yeah... I've sent out invites to join the private org / repo to both of you. Anyone else is welcome too ask for access as well. I can justify up to a $90 a month burn rate until it's public to get up to ~10+ other eyes on the effort. I suppose it's worth letting @dandv and @connor4312 in on it all too (invites pending). Do let me know any interest and I'll start adding more folks to the private repo in a week or so.

[ratson]

@typhonrt Thanks for the invitation, I will see what I could help.

About a new name... From a traction angle it makes 100% sense to come up with a new name. I've spent some time thinking about it and all of the obvious esdoc / jsdoc / documentationjs combinations are out there already. Like io.js perhaps it's best to just come up with a unique even non-documentation oriented name to help with visibility. I'm certainly open to suggestions in this regard.

I propose runes-doc, since you always want to a doc for reading Runes.

Regarding mono-repos I'm actually not a fan in general and have been developing tooling to support multi-organization / repo efforts which is how the larger TyphonJS effort is organized.

I suggested monorepo for the following advantages,

Pros:

• Single lint, build, test and release process.
  • Easy to coordinate changes across modules.
  • Single place to report issues.
  • Easier to setup a development environment.
  • Tests across modules are ran together which finds bugs that touch multiple modules easier.

But if they have been considered, then separate repos is fine.

Ah yes @ratson re: your final point of a permissive license. Regarding building communities and successful long term projects I take a lot of my reasoning for the MPLv2 license from the late Pieter Hintjens.

Personally, I am fine with MPLv2, but some people may not, which could reduce adoption. Honestly, I do not see any widely-adopted MPLv2 project in the JavasScript community. If it has been weighted less important, it is ok.

[connor4312]

Hey typhonrt, thanks for the tag, glad to hear you're still pushing on this! I've got my hands full this month with node-sylvester (which I will also be esdocing initally 😉 ) but would be more than happy to contribute moving forwarding into February.

[typhonrt]

@connor4312 sounds great. It will be fantastic having you aboard especially for the Typescript angle!

@ratson Yeah the multi-repo direction does have concerns, but it will definitely be my job to make sure there is a clear and easy process with additional tooling to support the whole range of tasks involved that otherwise would be manually oriented. It may be until Q2 that all of this is worked through entirely, but it will happen. A big benefit for the multi-repo direction is that it breaks things down cleanly for separation of concerns w/ a single responsibility clearly showing which modules are stable. A lot of folks think a repo not getting tons of commits means it's not advancing, but in the multi-module approach not getting commits means it's stable and performing a single responsibility as designed and thus demarcated as such. If things are moving smoothly there aren't changes across modules in regard to API exposed.

From the design I'm angling towards there are two locations where changes in one module (DocFactory / extracting doc tags from AST) affects another (Publisher / creating the HTML result). In this case though additions to AST parsing should only expose new publishing capability versus breaking it. The ParamParser is another area which might be upgraded, but just bug fixes to enhance ability (API shouldn't change).

Regarding issue support it's possible with Github to turn off issues for a given repo, so issues will only be enabled in the main repo. All of the other modules will have their NPM issues entry in package.json pointing to the main repo issues along with a note in the README. An issue template in the main repo will describe any process involved. Any issues coming into the main repo will be tagged with the module they correspond to.

On naming outside of tjsdoc having the association with TyphonJS it also connotes Typescript support w/ JSDoc feature parity (annotations / tags supported) which is where I'd like things to move. typhonjs-doc will just be the organization on Github and it won't be referred to as typhonjs-doc elsewhere; just TJSDoc. Of course there is time to change all of this, but for now I'm changing all esdoc references to tjsdoc for the time being.

The MPL issue on affecting adoption is interesting to consider. Since everything is a plugin, internal & external, any part can be replaced + I'm angling to add hooks and extension points to make external plugins have more power in attaching new features to generated documentation without having to bulk rewrite HTML output (this is how my plugin esdoc-plugin-enhanced-navigation works). Also as a standalone documentation tool it's not expected that this effort will be directly incorporated in other software as a direct functional component. My assumption is that creating a documentation tool that works exceptionally well with timely upgrades and maintenance will assuage license concerns of the larger JS & Typescript communities.

[skurger]

This looks very promising. I am particularly interested in being able to declare some sort of @package tag or something similar. I know that the original project maintainer had a problem with the @module tag because of conflicting meanings with the new import export syntax. I also know he is a big fan of using classes in js but the reality is not everything belongs in a class. I really want to use Esdoc to document some Redux code that I have (mostly files filled with exported functions) but the limitation on using classes for the best user experience in the docs makes it impossible for me to use. The functions in the sidebar menu quickly pollute the sidebar and becomes unusable.

The only workaround I know in Esdoc is to create a folder that then contains the actual js file. This is a hacky workaround and it just makes sense to have a @module tag (note that by doing this all the functions still show up, you have to scroll down to see the results grouped by folder name). Has there been any thought about supporting this in this fork of Esdoc?

[typhonrt]

@skurger Check out my plugin esdoc-plugin-enhanced-navigation as it really is superior to the default navigation. This plugin will group multiple doc tags / exports, etc. by file and it's collapsible. Also adds Github / NPM links w/ a context menu, if things are setup correctly. This is one of the first things I'll be rolling into the default experience for this new splinter effort.

[typhonrt]

As far as updates on "tjsdoc" goes I just finished a 99% functionally complete version of a standalone plugin manager and will push it to stable once I get 90%+ test coverage; typhonjs-plugin-manager It has many more features than the very basic ESDoc implementation and also plays well with backbone-esnext-events / eventbus. IE all event registration for plugins is through a proxy so when the plugin is removed or the plugin manager is destroyed all events are unregistered and the plugin author doesn't have to worry about it. It's pretty nifty and allows plugins to self-wire into the eventbus system.

So far I have just a few remaining areas that I need to modularize, but things are coming along. I'm removing generic utility code and punting them out to separate NPM modules; plugin manager was the first, then logging, etc.

One note though is that I'm simply not interested in remaining compatible with ESDoc plugins. There are all sorts of little language (Japanese speaker / original author) here and there. For instance using "option" instead of "options" in the plugin config parameter. Since I am releasing independently the plugin manager potentially used in any project including tjsdoc I don't feel it necessary to adopt awkward (however small) aspects of ESDoc. So long story short.. There will be no compatibility at a deeper level nor do I think it is best to "wait patiently" for anything the original author might do as his reasoning has been suspect; he was a great starter / initial architect though! I did a search on NPM and all of the existing ESDoc plugins are really simple and MIT licensed. My plugins such as esdoc-plugin-enhanced-navigation, esdoc-plugin-jspm, esdoc-plugin-dependency-graphs are by far the most complex out there, so anything useful that isn't this can be rewritten in an afternoon.

[skurger]

@typhonrt I tried the enhanced sidebar plugin and I think this will solve my problem. I'm going to start documenting more and see how it works out. I tried the plugin before but it was not working with the current version of esdoc (not this fork) so I gave up for a while. I switched over to your fork and it is working great.

Thanks for taking charge on this. I am going to try to look more into integration with React when I have some time. I talked with you briefly on gitter about this last year.

[typhonrt]

Right on.. The enhanced sidebar plugin is working with the latest ESDoc source I've started the fork with; I haven't been aware of it not working with the mainline release of ESDoc per se. Nonetheless it will be great to have some folks who work with React to be involved in TJSDoc!

[CoenWarmer]

Wow, great work @typhonrt! I'm following with interest!

[jdylanstewart]

I am excited to see this project flesh out. I'm very glad you are taking this on.

I'll be adopting your documentation as soon as possible for all of my team's work and hope to provide feedback/help when possible.

[typhonrt]

@jdylanstewart right on; thanks for the support! Progress is moving forward daily. Definitely follow https://github.com/typhonjs-node-tjsdoc/tjsdoc/issues/2 for sparse, but ongoing updates. I'll have a new update posted in a few days about the latest progress. It's pretty exciting though ( well, as much as doc tools can be! ;) as I'll have finished a couple of plugins for very quick incremental doc regeneration w/ built-in watch functionality suitable for live regeneration and am pairing things with optional launching of live-server for automatic browser refresh. To pull all of that off though I had to really streamline the runtime, so things are considerably more efficient compared to ESDoc especially memory usage. I'd like to get things out publicly on NPM by early mid-April or before and I'll likely be working on TJSDoc and related plugins full-time+ through June easily and many things are to come.