Custom options for jsParser service in jsdoc

[lmores]

During document generation I need to parse javascript files the uses the async/await keywords. According to this issue from espree repository the value of the option ecmaVersion passed to the parser must be 8 or greater to support those keywords, however I don't know how to pass this option to the parser used here inside jsdoc package.

(Sorry if it's a silly issue, I'm using dgeni for the first time)

[petebacondarwin]

Instead of configuring the parser you can simply provide your own jsParser service that does what you want it to do. Create the service in your own dgeni package and depend upon that after depending upon the jsdoc package. Then your version will override the version provided by the jsdoc package.

[lmores]

Hi, thank you for your super fast reply, really appreciated! Unfortunately I'm still not able to make my code work (I think I'm breaking something while overriding the default jsParser() service). Here is my code (I'm using dgeni from typescript, hope it's not a problem):

// myPackage.ts
import * as jsParserImpl from 'espree';
import * as base from 'dgeni-packages/base';
import * as jsdoc from 'dgeni-packages/jsdoc';
import * as nunjucks from 'dgeni-packages/nunjucks';

import { Package } from 'dgeni';

export const myPackage = new Package('myPackage', [base, jsdoc, nunjucks])
  .factory(
    // override the service from jsdoc package to set custom parsing options
    function jsParser() {
      return function(code) {
        return jsParserImpl.parse(code, {
          ecmaVersion: 9,  // minimum version required to parse `async/await` keywords and '...' syntax
          sourceType: "module",
          // ...all the other options from jsdoc.jsParser() implementation...
        });
      };
    }
  );

In the main file I set the config options and then I run dgeni:

import * as path from 'path';
import * as base from 'dgeni-packages/base';
import * as jsdoc from 'dgeni-packages/jsdoc';
import * as nunjucks from 'dgeni-packages/nunjucks';

import { Dgeni, Package } from 'dgeni';
import { myPackage } from './myPackage';

const config = new Package('config', [base, jsdoc, nunjucks, myPackage])
.config(function(log, readFilesProcessor, templateFinder, writeFilesProcessor) {
  log.level = 'debug';
  readFilesProcessor.basePath = path.resolve(__dirname, '..');
  readFilesProcessor.sourceFiles = [{ include: 'scripting/JsTest.js', basePath: '' }];
  templateFinder.templateFolders.unshift(path.resolve(__dirname, 'templates'));
  templateFinder.templatePatterns.unshift('common.template.html');
  writeFilesProcessor.outputFolder = 'build';
});

const dgeni = new Dgeni([config]);

dgeni.generate().then(
  (docs) => { console.log(docs); }
)

Executing the above code gives me:

info:    running processor: readFilesProcessor
debug:   Source Info:
 include=[...OMISSIS.../scripting/JsTest.js], exclude=[], basePath=...OMISSIS...
debug:   Found 1 files:
 0=...OMISSIS.../scripting/JsTest.js
debug:   Reading File Content
File Path: ...OMISSIS.../scripting/JsTest.js 
File Reader: jsdocFileReader
info:    running processor: extractJSDocCommentsProcessor
info:    running processor: parseTagsProcessor
info:    running processor: extractTagsProcessor
info:    running processor: codeNameProcessor
info:    running processor: computeIdsProcessor
info:    running processor: computePathsProcessor
info:    running processor: renderDocsProcessor
info:    running processor: unescapeCommentsProcessor
info:    running processor: inlineTagProcessor
info:    running processor: writeFilesProcessor
info:    running processor: checkAnchorLinksProcessor
[]  // <-- content of 'docs' variable, empty!

and no files are written in the output folder.

However, If I remove myPackage from the dependencies of config, i.e.

const config = new Package('config', [base, jsdoc, nunjucks])

I get

info:    running processor: readFilesProcessor
debug:   Source Info:
 include=[...OMISSIS.../scripting/JsTest.js], basePath=...OMISSIS...
debug:   Found 1 files:
 0=...OMISSIS.../scripting/JsTest.js
debug:   Reading File Content
File Path: ...OMISSIS.../scripting/JsTest.js 
File Reader: jsdocFileReader
info:    running processor: extractJSDocCommentsProcessor
info:    running processor: parseTagsProcessor
info:    running processor: extractTagsProcessor
debug:   extracting tags - doc (js)  - from file "scripting/JsTest.js" - starting at line 3, ending at line 8
debug:   extracting tags - doc (js)  - from file "scripting/JsTest.js" - starting at line 30, ending at line 35
info:    running processor: codeNameProcessor
info:    running processor: computeIdsProcessor
debug:   computed id for: "resizeAspectFit" (js)
debug:   computed id for: "anotherFunc" (js)
info:    running processor: computePathsProcessor
debug:   path: resizeAspectFit; outputPath: resizeAspectFit.html - doc "resizeAspectFit" (js)  - from file "scripting/JsTest.js" - starting at line 3, ending at line 8
debug:   path: anotherFunc; outputPath: anotherFunc.html - doc "anotherFunc" (js)  - from file "scripting/JsTest.js" - starting at line 30, ending at line 35
info:    running processor: renderDocsProcessor
debug:   Rendering doc: resizeAspectFit
debug:   template found ...OMISSIS.../templates/common.template.html
debug:   Rendering doc: anotherFunc
debug:   template found ...OMISSIS.../common.template.html
info:    running processor: unescapeCommentsProcessor
info:    running processor: inlineTagProcessor
info:    running processor: writeFilesProcessor
debug:   written file ...OMISSIS.../build/anotherFunc.html
debug:   written file ...OMISSIS.../build/resizeAspectFit.html
info:    running processor: checkAnchorLinksProcessor
[ ... a lot of info... ]  // <-- now the 'docs' variable has all the expected info.

Probably I do something wrong when I define myPackage. Could you tell me what's wrong?

[petebacondarwin]

Strange! I've no idea. Can you try debugging into that service? First of all to see if it is being triggered and secondly to see how (and whether) the output is being used. Did you upgrade to a new version of espree? I wonder if the return value from the function call has changed?

[lmores]

Yes, you are right! I was using "espree": "^7.3.0" for my custom implementation of jsParser(). The return value of espree.parse(...) is somehow different from the one of "espree": "^2.2.3" shipped with "dgeni-packages": "^0.28.4". Downgrading espree made my custom implementation work!

I'll play a little more with my dgeni project and then I'll close this issue if everything works.

Thanks a lot.

[petebacondarwin]

Great! It may be that you could change your jsParser implementation to understand the new return value from the latest version of espree too?

[lmores]

Quick update: using espree 2.2.3 my custom jsParser works fine, but is not able to parse "recent" keywords like async/await (and it is reasonable since that espree version is likely to be older than the introduction of async/await as javascript keyword). On the other hand espree 7.3.0 breaks something inside dgeni (or some dgeni package) and no output is produced.

Here is a diff between the output of espree 2.2.3 and espree 7.3.0 when the following js code is parsed:

'use strict'

/**
 * Resize an image to fit the given dimensions and preserving aspect ratio
 *
 * @param {string} b64Image
 * @param {number[]} dimensions
 * @returns {number}
 */
function resizeAspectFit(b64Image, dimensions) {
  return 1;
}

/**
 * Another comment
 *
 * @param {string} arg1
 * @param {number[]} arg2
 * @returns {any}
 */
function anotherFunc(arg1, arg2) {}

Maybe you can see at a glance what's wrong.

By trial and errors I found out that espree 4.1.0 works fine with dgeni and is able to parse async/await and the spread operator, whereas espree 5.0.0 breaks something in dgeni (possibly the same as in 7.3.0). As you can read here there is one breaking change from v4.1.0 to 5.0.0.

At the moment I'm in a bit of a rush at work, but when I'll have some spare time (and I'll be more familiar with dgeni and espree) I may come back and try to submit a PR to make dgeni work with the latest version of espree (if you are interested in it).

[petebacondarwin]

That would be grand! Thanks @sir-ion

[petebacondarwin]

From your diff, it looks like none of the comment nods are being returned in the result from espree. Clearly this is essential for the jsdoc package to work. I wonder if there is some additional config option that is needed?

[petebacondarwin]

Are you setting comment: true, in the options passed to espree.parse()?

[petebacondarwin]

I notice that in dgen-packages jsParser we are passing comments:true rather than comment: true: https://github.com/angular/dgeni-packages/blob/master/jsdoc/services/jsParser.js#L14

[lmores]

I think it's some kind of typo. Here you use comments instead of comment as in the official espree documentation.

Now my custom parser works with espree 7.3.0!

[petebacondarwin]

🎉

[lmores]

Well, this settle the issue! Sooner or later (if I do not find other issues using 7.3.0) I'll send a PR to update the espree version and fix comment (if you don't do it before I do).

[petebacondarwin]

Thanks, that would be great!