Rethinking the scope of the antlr4ts-cli tool

[BurtHarris]

This idea needs further development, but I wanted to float it early for feedback. I'm still not sure with how it applies to separate parser and lexer grammars, or how to treat grammars which are used only as sub-grammar.

Preface: this may make more sense if we consider the fact the overall package may have a slightly misleading name. Antlr4ts is really not really ANTLR for TypeScript, it is ANTLR for JavaScript with TypeScript being an implementation detail to enhance quality.

I suggest we rethink what our CLI does: instead of only at emitting files in Typescript, the CLI should go further, and invoke the compiler typescript on them so the output is an immediately a usable JavaScript module, with a module declaration file (.d.ts). If a programmer is writing in JavaScript rather than Typescript, he shouldn't need to worry about tsc, the CLI can take care of that for him.

For example, given the Expr.g4 grammar from the book, invoking the CLI tool on that file, without any other switches, should:

1) Invoke the Java antlr tool specifying the command-line switches to ensure the output goes into a directory named Expr/.antlr4ts. 2) Generate an index.ts file into Expr that (among other things) re-exports the lexer, parser, and context classes from the Expr/.antlr4ts folder. 3) Generate a tsconfig.json into Expr with just the right settings to use. (This can address issues like making sure to use "target": "ES2015" 4) And invoke tsc to on Expr/index.ts to generate the .js and .d.ts outputs needed to make the Expr folder a separately transpiled JavaScript module.

Yes, a directory containing an index.js file is automatically considered a module by the JavaScript module loader, so the larger application can gain access to the Expr.g4 implementation, ignoring original language the module started in, using:

const Expr = require('./Expr')

Or one of the equivalent import statements;

[BurtHarris]

@mike-lischke, I'd really appreciate your thought on this.

[mike-lischke]

Tbh. I don't use the antlr4ts cli at all and never have. Instead I directly invoke the antrl4ts jar. The cli is just another wrapper whose parameters and handling you have to learn. I believe it would make more sense to focus on providing that jar and even more sense to include it in the main ANTLR4 line. This separation of an ANTLR4 target is not doing any good, but is probably required since it is in a rather early state with almost no dev capacity to improve it.

There's also that request for quite some time already to replace the current JS target by antlr4ts.

[BurtHarris]

Thanks @mike-lischke for the input.

Re: merging this back into the main ANTLR4 line. It had been years since I looked at this, but yesterday before reading your comments I cloned the main ANTLR4 line, and had a look around again. Some good things emerge: it helps me to understand the intent of the directory structure, and I'm working on a PR with reorganizing to fit. But it also brought me to remember some of the reason why we forked in the first place. For example, the build times there suck, and it requires a lot of Maven/Java knowledge to navigate the tools. I guess at the heart of it is that I've never been a Java dev, but learned a good bit doing this work, and I'm certainly not opposed to merging this back. @sharwell: are there technical challenges related to the standard vs optimized tooling?

To me, a slow dev cycle takes the fun out of working on a codebase, perhaps there are tricks within antlr/antlr4 for isolating one's work to a specific target.

Eventually I'd like to see a target that didn't requiring the JRE, but that's certainly not a goal for current iterations.

[BurtHarris]

@mike-lischke: From experience, it seems that the TypeScript team sometimes makes breaking changes that don't get reflected well in the typescript package's version number. For this reason, if a developer uses a different version of TypeScript to transpile antlr generated .ts files, they may get confusing errors after they upgrade their local version of TypeScript.

By expanding the scope of that antlr4ts-cli does to include running TypeScript on the generated files, we can help our users avoid this headache.

In some cases there can be workaround that might involve using specific TypeScript transpiler options (switches), again those options are something that seem better controlled by the version of the ANTLR tool used to generate the files, not necessarily the version of TypeScript the developer has installed locally, or options he/she uses for the consuming code.