Node.js port.

[ghost]

Can't this be ported to javascript on node.js instead.

Why use another language to generate documentation for one language.

If you are using the same we can all contribute much more easier and don't have to install ruby.

[nene]

It's not the first time this question is asked. Of course it can be ported, but it's a huge amount of work. Is it all really worth it? Are the gains really so big to justify this huge effort? Is installing NodeJS really so much easier than installing Ruby? Is Ruby so very different from JavaScript that it's a huge burden for contributing? I don't really think so.

And what are the downsides of NodeJS? There surely are some. Not all features of Ruby will match up with features of NodeJS. For example I'm pretty confident that JavaScript is more verbose than Ruby. For comparison:

cubes = [1,2,3].map {|x| x**3 }

var cubes = [1,2,3].forEach(function(x){ return Math.pow(x, 3); });

Ruby has so far justified itself as a development language for this project. Porting it is a huge work with questionable gains. You are free to do it, if you really want to.

[coreybutler]

@nene - This issue is over a year old & I'm purely curious if your thoughts around porting to Node.js have changed since then. I am not asking for a port... just observing that node.js seems to be sprinkled into JSDuck now in a couple of places.

[nene]

I'm still on pretty much the same position.

It's true that JSDuck now embeds a V8 engine which it uses to run esprima.js (a JavaScipt parser written in JavaScript). So this might look kind of crazy - running JavaScript inside Ruby to parse JavaScript. That's really all quite coincidental - I was looking for a JavaScript parser to use in JSDuck and the only one that happened to satisfy my requirements was esprima.js. Of course, if I would have started the project in Node.js, this would have now been a really nice fit. But there was no way to know back then as esprima.js didn't even exist when I started.

But I still think that porting JSDuck to JavaScript is too much work with questionable benefits. There are other parts of JSDuck that I'm planning to rewrite, with each of those I have concrete reasons for doing so and direct benefits that I'm planning to derive. I don't see that with Node.js port.

Also, I should note that replacing the old parser with esprima.js was a huge amount of work and it took way-way longer than I expected (I started experimenting with it almost half a year ago, and only a week ago did I release the final JSDuck 4.0). Porting the whole JSDuck to NodeJS would be a task at least as big.

[gjtorikian]

I've always been really impressed by the styling of the JSDuck documentation. I think it got a lot of things right in terms of usability, so much so that the two API doc projects I've been involved in recently (NodeManual and the Ace API) have been heavily inspired in terms of design.

Recently I was tasked with generating API documentation coming from JSDoc style comments. I had worked on a parser that understood pdoc-style comments. Given that Rene was not interested in doing a conversion to Node, I took the task of porting the entire parsing segment of JSDuck into Node.js. You can find the relevant content here.

All in all it only took about a week to do. With the use of UnderscoreJS and a StringScanner library I was pretty much able to reproduce all the Ruby specific functionality.

My port is for the work I am currently doing, so it's got a lot of stuff in there that only we would use, and it does not conform to any need for parsing Ext.js documentation--only regular JSDoc. As well, I don't do comments, examples, videos, .e.t.c.. And, since I'm piggy-backing on a parser I already worked on, the JSON information I generate is vastly different than the stuff JSDuck produces and expects. I use Jade to render the HTML. There's too many other differences to list here! I am using Esprima and various expander/merge classes in the exact same way as JSDuck, however.

I hope that this at least helps people using JSDuck or JSDoc in advancing a proper JavaScript port, if one is still required.

[nene]

Thanks for input. You've sure put quite an effort into this.

I have to warn you though that all of the parsing code in JSDuck is a subject to change at any time. I'm not really happy with the way the parsing code is currently structured, so I'm planning to refactor the whole thing considerably. But as you said, you're really not attempting a direct port.

Anyway, good luck with your project!

[gjtorikian]

Yes, exactly, I am not out to recreate either JSDuck or JSDoc. But I do think it is silly to not have a doc generation tool in the language being documented.

I do know Esprima is planning a release where comments are inline the AST, instead of grouped at the end. I imagine that that will be the only future planned work.

[micmath]

Re: "Esprima is planning a release where comments are inline the AST, instead of grouped at the end," this would be the bare minimum necessary before I could consider porting JSDoc over to NodeJS. Do you know more about when and how this will happen?

[coreybutler]

I've been using JSDuck for my (very alpha level) node project, NGN. It helps that the project follows a similar class inheritance approach as Ext, so JSDuck fits in quite nicely. I have aspirations to make NGN completely self contained, meaning no dependencies on anything other than node.js.... and I'd love to distribute JSDuck with it (NGN is designed to be extended by other developers), but the Ruby dependency makes this challenging. It's totally outside the scope of JSDuck, so no complaints/requests.... but if there were a node version, I'd definitely choose it over a Ruby version since my workflow is so JS-oriented.

That said, I've come across several things I really wish I could add to JSDuck. I love the UI and layout... it makes sense to a lot of people. I just wish I could use it for more documentation. For example, I have what I feel is a good vision for documenting node-based REST services/API's as another tab in JSDuck. There are other tools, but it's a pain to merge them into a single documentation portal, and I'm not keen about the layout compared to JSDuck. If only I had more time to pickup Ruby to extend JSDuck.

I realize how much effort it would be to port JSDuck to node... so I'm not going to ask for it when I know I can't really contribute to it. I guess I'm just throwing in my 2 cents to say I'd be more apt to contribute if the code base used a language I already use every day.

[gjtorikian]

Re: "Esprima is planning a release where comments are inline the AST, instead of grouped at the end," this would be the bare minimum necessary before I could consider porting JSDoc over to NodeJS. Do you know more about when and how this will happen?

This is irrelevant to JSDuck, not sure why the comment is here.

I'm also not sure when it's going to happen, but there is an open issue about it on Esprima's issue tracker.

[vmadman]

I had to weigh in on this, being a long-time ExtJS evangelist... I was just taken back by Nene's comments. Every comment is filled with supportive information, then followed by a contrary statement. Sorta like this: "Gold is rare, relatively durable, and its value is universally agreed upon, which is why it would make a bad currency."

I think the Sencha gang should not only port JSDuck to Node, but also port the Sencha CLI tool to Node and the SCSS to LESS while they're at it. So, please allow me to answer the questions he put forth.

It's not the first time this question is asked.

Right, because its an obvious question..

Of course it can be ported, but it's a huge amount of work. Is it all really worth it? Are the gains really so big to justify this huge effort?

Yes...

gjtorikian:
All in all it only took about a week to do. With the use of UnderscoreJS and a StringScanner library I was pretty much able to reproduce all the Ruby specific functionality.

Is installing NodeJS really so much easier than installing Ruby?

Yes, but that's not the point. The point is that, for a lot of us, the only reason Ruby exists in our lives is because of JSDuck, the Sencha CLI tool, and Sass (all brought to us by Sencha). The rest of our lives are filled with JavaScript and whatever backend platform we use (which may or may not be Ruby).

Is Ruby so very different from JavaScript that it's a huge burden for contributing? I don't really think so.

Ok, wow, yes... let me see if I can drum up an example..

cubes = [1,2,3].map {|x| x**3 }
var cubes = [1,2,3].forEach(function(x){ return Math.pow(x, 3); });

Even the few similarities within those two examples only exist because the JavaScript is poorly written (intentionally brief and abbreviated). It would be difficult, I think, to design a language that is less like JavaScript than Ruby is.

And what are the downsides of NodeJS? There surely are some. Not all features of Ruby will match up with features of NodeJS. For example I'm pretty confident that JavaScript is more verbose than Ruby.

No, there aren't any downsides to node, even if that was relevant. Again, the point is that you're preaching Ruby to a JavaScript choir here. What if I were arguing for Pascal right now? Same thing..

Ruby has so far justified itself as a development language for this project. Porting it is a huge work with questionable gains.

Some may have questions about those gains, but I personally do not. Its so fundamental and obvious that its really hard to argue, but let me reiterate the points anyway:

• ExtJS is a JavaScript library
• The only known commonality amongst its user base is that all users are JavaScript programmers
• JavaScript has been proven to be very capable of documentation generation (10+ active doc generators, easily)
• JSDuck is a Ruby application that outputs html, css, and javascript by parsing javascript

It's true that JSDuck now embeds a V8 engine which it uses to run esprima.js (a JavaScipt parser written in JavaScript). So this might look kind of crazy - running JavaScript inside Ruby to parse JavaScript.

.. and Gold would make for horrible currency. (C'mon, you have to see what I'm talking about, right?)

See also:

// https://www.npmjs.org/package/esprima
npm install esprima
var esprima = require("esprima");

.. That's really all quite coincidental

Coincidental or not, its still crazy.

I have to warn you though that all of the parsing code in JSDuck is a subject to change at any time.

.. all the more reason ..

I'm not really happy with the way the parsing code is currently structured, so I'm planning to refactor the whole thing considerably.

.. so, while you're at it ...

---

With all of that said... I totally recognize that what you do with your time is your prerogative, and since JSDuck is free software, none of us have any right to tell you how to drive from the back seat. Obviously you can do whatever you'd like and I will remain grateful for JSDuck and your efforts either way.

I'd love to see JSDuck ported to Node, but that's not really what I'm arguing here. I only wanted to point out the blatant weakness of your opposing points. Much better would be: "Meh.." , "Naaah..." or ".. if I do it, will you pay me?" ;)

[micmath]

@vmadman, that sounds very much like the internal debate I had when deciding to steer JSDoc away from it's original Perl codebase and towards (firstly Rhino, and then) NodeJS instead. Then, as now, I fell on to all your sides the argument.

Yes, it was a lot of work, but in every case the tool was significantly improved from that work. In retrospect it was absolutely the right way to go.

[nene]

@vmadman I have to point out, that you're responding to one really old thread, a lot has changed since then - e.g. the parsing code has actually been rewritten several times and it's now using a Ruby-based JS-parser instead of embedding V8 to run Esprima.

I perfectly understand a desire to have tools for language X written in language X. It's a nice thing to have.

I still fail to see the actual problems that using JavaScript over Ruby would solve, outweighing the huge cost of porting. I have actually considered porting to Go, because of the performance benefits it would bring to the table, but JavaScript?

Currently the situation is, that about half a year the project has been in stand-by and I've only barely managed to apply some fixes, let alone do any major development.

[vmadman]

@nene I apologize for not noticing the dates, this issue is very highly ranked by a Google search for "jsduck node".

Please understand that while my methods of communicating are direct, I mean no disrespect in any of this, and am thankful for having JSDuck, regardless of its source language.

I perfectly understand a desire to have tools for language X written in language X. It's a nice thing to have.

I think that sentence is at the core of this discussion as I think every proponent of porting JSDuck to Node would call that an incredible understatement.

If I could find a suitable Photoshop replacement built in Node.js, that would be nice. Having my JavaScript API documentation compiled by a JavaScript lexer is far more than nice. Depending on the use it could easily become a deal-breaker.

When we talk about source documentation we're talking about programming at its most fundamental level. In many ways API source documentation is more about programming than the source code itself, because it represents a transcendence of the source code.. an abstraction layer even deeper than our lowest level code. (i.e. Ext.base).

(We're now at the point where one of my co-workers usually springs out from behind a corner with a new and clever way to combine the word "Inception" with a word from this discussion, like "codeception")

The only thing closer to the source code (perhaps) than the documentor would be the IDE, and we test drive Cloud9 and Nitrous.io about once per quarter to see how much they've progressed...

If you talk to a Node.js guy, for example, he is very likely to use NPM exclusively for his package management, LESS (or stylus) for his CSS, Forever (or equiv) for process management, Grunt as his task runner, and so on and so on. We're talking about an audience of people who want to tamper, that's why they started programming, and then you put a tool in front of them that assists with their tampering and......... its in a language that they do not know.

The most extraordinary point in all of this is the contributor angle. The one thing you do know about your entire user base is that they all know JavaScript. Surely you can imagine that the number of people who could and would contribute to the project is likely to increase, substantially, if you ported to Node.

[vmadman]

Also, to me personally, my absolute favorite thing about JSDuck is how it organizes and outputs the documentation, not about how it parses. One project that I've found recently, Dox (https://github.com/visionmedia/dox), has went as far as removing its graphical output all together. By outputting to JSON they allow a separation of the computation and visual aspects, and its pretty clever.

@micmath Thank you, also, for JSDoc. JSDoc paved the way and continues to set the standard for JS documentation. Good stuff.

[nene]

Well, the most I can do at this point is to open this issue back up again, as there really is a larger desire for this.

[micmath]

@vmadman, while I admit that this is going off on a tangent, having a standard defined for how to represent API documentation as JSON, in the form of a JSON Schema for example, would be very helpful to maintainers of documentation generators -- JSDoc also has the option to output JSON, but it is not in a format compatible with Dox, and presumably JSDuck wouldn't want to invent a third way of representing this information on top of those. Probably a discussion for a different thread though...

Regarding the "contributor angle" you mention, that was probably the greatest benefit that JSDoc received from converging the implementation language with the source-input language. As you point out, it's an undeniable fact that you know to be true about every user of JSDuck: they like JSDuck and they write JavaScript. What better army of helpers to invite into your project repository? It's a valid judgement to make however: are those ease-of-contributing and ease-of installing angles more or less valuable than say, speed-of-execution or ease-of-not-rewriting issues. In my experience speed-of-execution is a relatively low priority for a documentation generator, assuming we're talking a difference of a second or so, not many minutes. And assuming the main goal of JSDuck isn't to be known primarily as the fastest generator in town.

Regarding ease-of-not-rewriting, this is a convincing argument. It is very hard to see a lot of wisdom in just changing the language of an application's source code, when that entails a great deal of work and results in a tool that is functionally the same, unless there was some value to be gained in other ways [see my points above]. In the case of JSDoc we were intending a rewrite anyway, so that consideration had no weight in the decision. In JSDuck's case however, I suppose it is a valid debate to have: does an increase in ease-of-contributing justify the work to rewrite, particularly if that comes with an potential performance degradation (I'm not convinced it would, but for the sake of the argument let's say it did). I know which way I'd go--and did go--but the priorities of this project could correctly and rightfully lie elsewhere.

[vmadman]

@micmath I agree with everything you said there, and I think you've more or less hit upon the practical answer to the overall question. Your answer is the same as @nene's, but from a more precise angle that, I think, is easier to manage and discuss.

I am sure that I did a poor job of it, but I tried to convey that while I am saying a Node.js port would be extremely good for the project, I cannot account for the practical concerns surrounding it, such as Nene's time constraints and motivations. As it often goes, even in commercial development, the best solution from a technical standpoint is not the most pragmatic.

I only wanted to comment on the reasoning behind not making the jump because it seemed like it was being blocked for the wrong reasons which could cause it to never happen even if an opportunity presented itself.

Hopefully, JSDuck will recognize the idea of porting to Node.js as a noble (albeit long-term) objective and will, whenever possible, take small strides in that direction. Of course this is all derived from my selfish desire to stop installing Ruby in our development environments and, by extension, my long nightmare that is Ruby could eventually come to an end ;)

[nene]

JSDoc also has the option to output JSON, but it is not in a format compatible with Dox, and presumably JSDuck wouldn't want to invent a third way of representing this information on top of those.

Well... JSDuck already has its own incompatible JSON export format. So yeah, a standard for docs export format would be great.

[vmadman]

If you guys do find an appropriate external medium and discuss it, I'd love to join in. One of the nice features of JSDuck is the @config tag, and the ability to document object properties, which other systems seem to miss, or at least deemphasize. You also have the often overlooked @inherits tags and things like that. Plus, as I learned as I migrated to Node.js, which emphasizes RESTful APIs, I've had to implement two separate parsers, one for the traditional API docs, and one for the REST API endpoints. (See "Swagger" and "apiDoc"). Hopefully any talks of standardizing a format would leave room for such things.

[wrumsby]

But I still think that porting JSDuck to JavaScript is too much work with questionable benefits.

Have you read the installation instructions for Windows?

Theoretically a Node-based implementation of JSDuck would work a lot better cross platform than the current implementation, because the current implementation is not ideal in Windows environments.

[vmadman]

Not huge on using Windows for "real life" very much. I basically use windows as host for my linux VM dev empire. So, I'm sure you know more about this than me... but just-in-case: Couldn't you leverage Chocolatey and/or Vagrant to help lessen the Windows burden?

[coreybutler]

I'd be interested in discussions around a standardized format.

What's wrong with the Windows binary? I've been using it for quite a long time, quite happily.

Node doesn't offer any additional cross-platform benefit that Ruby doesn't. Just think about native module compilation, privileged directories, etc. The problem is with the OS differences, not the language. What Node really offers is a wider audience of potential contributors (as well as hacks).

[wrumsby]

What's wrong with the Windows binary?

I don't want to check in an .exe into my git repository - it's only relevant to a subset of my users. And it's a "devDependency", not a runtime dependency.

A globally installed .exe isn't a great solution because I may want to use different versions of JSDuck for different projects.

Really I just want to npm install jsduck --save-dev (and not have to have installed gem jsduck beforehand).

Node doesn't offer any additional cross-platform benefit that Ruby doesn't.

In theory that's true, in practice that's not true. There are lots of gems that seem to have a problem running under Windows, especially if the devkit is required.

Asking folks who only have a tiny amount of experience to install the right version of Ruby for their version of Windows + the right version of the devkit + the JSDuck gem is a lot more complicated than "install Node, then run npm install in your project directory".

Couldn't you leverage Chocolatey and/or Vagrant to help lessen the Windows burden?

Maybe, I don't know a lot about Ruby or about effective ways of managing gems (I'm playing around with Bundler ATM), but again that's a much more complicated story than "install Node, then run npm install in your project directory".

JSDuck seems to support documentation features that similar JS documentation tools don't (more tags, live examples, etc.) so it's appealing to people using JavaScript, while anything Ruby based is unappealing for the environment I work in where there are a mix of folks, some of whom are only really comfortable using Visual Studio. We can convince them to install Node (and Visual Studio provides some assistance here I believe), we have a much more difficult time convincing them to do anything more complicated than that.

[wrumsby]

(Removing the requirement on Ruby DevKit would actually go some way to making the gem easier to work with cross platform.)

[nene]

Would love to do that, but the last time that I looked, all pure-ruby Markdown parsers were unacceptably slow :(

[vmadman]

No doubt, and to be fair, Node suffers from the same sort of problems that Ruby DevKit creates, since some of the fundamental modules require source builds. I had to acquire Visual Studio express just to get socket.io (I think it was?) to install and sometimes have to specify --msvs_version=2013 when running npm install.

I do not know what the solution to that is, but its not unique to Ruby.

Also, to be fair, there are certain occasions where node modules are also locked in by a version of node or npm. It seems to happen a lot less often when compared to Ruby, but then again Ruby is much older (allowing more time for divisions and such to form) and the authors of NPM no doubt had the benefit of their experience with Ruby Gems.

I will tell you though, there is at least one potential game changer: https://github.com/atom/atom-shell

Every documentation system out there is mildly incompatible with my ideals (granted that my ideals mean little in the grand scope, but its still rather annoying to me). I wish I could have the syntax of Natural Docs, the watcher of YUIDoc, the simplicity of JSDoc, the REST API features of Swagger but styled more closely to apiDoc, and the output and extended features of JSDuck. Would be awesome. One day when I have time I'll hopefully be able to contribute to that dream somehow.

[hashar]

I believe this task should be closed. On https://www.sencha.com/forum/showthread.php?342460-Are-the-docs-still-generated-via-JSDuck Greg Barry (Sencha documentation lead) wrote:

No, we no longer use JSDuck

We are currently using a new node-based solution that we have developed internally. JSDuck is still a fine solution though. We would like to share our solution for the public, but I don't know what the timeline is on that. We still have some work ahead before we'll feel comfortable sharing it with the community. We really want it to be successful.

You can read more about the initial release here: https://www.sencha.com/blog/new-sencha-docs-are-coming-soon/

https://www.sencha.com/forum/showthread.php?311491-Will-the-new-documentor-be-available-for-our-own-apps&p=1137925&viewfull=1#post1137925 seems to hint at:

sencha doxi build