Closed HelgeG closed 6 years ago
I created a "pure" TypeScript project and used antlr4ts-cli
to create the parser and lexer classes, and I am not seeing this issue.
I am also seeing this same issue when using the antlr4
js package with a JavaScript
antlr target.
Using:
My grammar test (based on the docs):
grammar Expr;
expression: assignment | simpleExpression;
assignment: (VAR | LET) ID EQUAL simpleExpression;
simpleExpression
: simpleExpression (PLUS | MINUS) simpleExpression
| simpleExpression (MULTIPLY | DIVIDE) simpleExpression
| variableRef
| functionRef
;
variableRef: ID;
functionRef: ID OPEN_PAR CLOSE_PAR;
VAR: [vV] [aA] [rR];
LET: [lL] [eE] [tT];
PLUS: '+';
MINUS: '-';
MULTIPLY: '*';
DIVIDE: '/';
EQUAL: '=';
OPEN_PAR: '(';
CLOSE_PAR: ')';
ID: [a-zA-Z] [a-zA-Z0-9_]*;
WS: [ \n\r\t] -> channel(HIDDEN);
The grammar is compiled using antlr4 -Dlanguage=JavaScript Expr.g4
per https://github.com/antlr/antlr4/blob/master/doc/javascript-target.md
My code (based on the docs):
import antlr4 from 'antlr4';
import * as c3 from 'antlr4-c3';
const suggestExpression = (input) => {
const inputStream = new antlr4.InputStream(input);
const lexer = new ExprLexer(inputStream);
const tokenStream = new antlr4.CommonTokenStream(lexer);
const parser = new ExprParser(tokenStream);
const errorListener = new antlr4.error.ErrorListener();
parser.addErrorListener(errorListener);
const tree = parser.expression();
const core = new c3.CodeCompletionCore(parser);
return core.collectCandidates(0);
};
The error:
CodeCompletionCore.js:67 Uncaught TypeError: Cannot read property 'index' of undefined
at CodeCompletionCore.collectCandidates (CodeCompletionCore.js:67)
The lines:
I've tried adding the inputStream to the parser via parser.inputStream
, but this results in an infinite loop in the while (true)
block just below. It seems that the tokenStream.LT(offset++)
call is returning -1
without a .type
so the loop just keeps going.
Is antlr4ts
required? Seems like using the antlr4
lib should work, too.
I modified my parser manually and was able to prevent the errors from being thrown. It appears that the inputStream
might be the tokenStream
, so I attached that to the parser. The ATN states don't have a getTransitions()
method on them, even though c3 expects to always retrieve transitions this way, so I mapped a getTransitions
method onto each state which returns the state.transitions
that are defined.
const suggestExpression = (input) => {
const inputStream = new antlr4.InputStream(input);
const lexer = new ExprLexer(inputStream);
const tokenStream = new antlr4.CommonTokenStream(lexer);
const parser = new ExprParser(tokenStream);
const errorListener = new antlr4.error.ErrorListener();
parser.addErrorListener(errorListener);
parser.inputStream = tokenStream;
parser.atn.states.forEach((state) => {
state.getTransitions = () => state.transitions;
});
const tree = parser.expression();
const core = new c3.CodeCompletionCore(parser);
return core.collectCandidates(input - 1);
};
I think this is a bit of progress, however I'm still getting 0 candidates back. I assume this has to do with the lack of CodeCompletionCore.preferredRules
defined, but the test suggests otherwise:
let inputStream = new ANTLRInputStream("var c = a + b()");
let lexer = new ExprLexer(inputStream);
let tokenStream = new CommonTokenStream(lexer);
let parser = new ExprParser(tokenStream);
let errorListener = new ErrorListener();
parser.addErrorListener(errorListener);
let tree = parser.expression();
expect(errorListener.errorCount, "Test 1").equals(0);
let core = new c3.CodeCompletionCore(parser);
// 1) At the input start.
let candidates = core.collectCandidates(0);
expect(candidates.tokens.size, "Test 2").to.equal(3);
expect(candidates.tokens.has(ExprLexer.VAR), "Test 3").to.equal(true);
expect(candidates.tokens.has(ExprLexer.LET), "Test 4").to.equal(true);
expect(candidates.tokens.has(ExprLexer.ID), "Test 5").to.equal(true);
expect(candidates.tokens.get(ExprLexer.VAR), "Test 6").to.eql([ExprLexer.ID, ExprLexer.EQUAL]);
expect(candidates.tokens.get(ExprLexer.LET), "Test 7").to.eql([ExprLexer.ID, ExprLexer.EQUAL]);
expect(candidates.tokens.get(ExprLexer.ID), "Test 8").to.eql([]);
Given the other errors I was seeing, I'm assuming there's something fundamentally different about the JavaScript parser/lexer than the ones generated by antlr4ts-cli
and something is still missing. Any help or insight as to what might be different would be useful.
@kaidjohnson You know you are writing to a closed issue? Please don't hijack already finished issues but create a new one, if required.
Indeed there are differences between the JS and TS runtime versions. The authors of @antlr4ts insist on implementing their runtime in a way that is optimal for the TS language, thereby sacrificing compatibility. Best would be you open an issue against @antlr4ts, asking for more compatibility. However, I remember a discussion where people spoke about making the TS target the main one and generating the JS target from it. The current handwritten JS target would then be obsolete.
@mike-lischke It wasn't my intention to "hijack" anything. The issue I was seeing was the same as the user that posted the original issue here, and I thought I might get some additional insight into what the options were to get around it. Every git repo owner seems to have a different preference for creating tickets -- some complain about opening duplicates, some complain about extending closed tickets. It's always a 50/50 gamble; didn't mean to ruffle any feathers.
My current JS target was not handwritten, but rather generated via the core antlr4 library.
FWIW, I did work around the issue after a bit more trial-and-error by generating my parser and lexer via antlr4ts, converting the resulting ts (es2015 target) to js (using babel) -- the typescript es5 target wasn't handling the class conversion well -- and then using the core antlr4 runtime (not the ts version) for use in the browser. The parser needed to patched slightly to be compatible with the core Parser class from antlr4ts, but after that small patch (see below), I am getting tokens back.
Here's the patch, which essentially just copies over the missing method from antlr4ts:
import BaseExprParser from './ExprParser.ts';
class ExprParser extends BaseExprParser {
constructor(...args) {
super(...args);
// The antlr4ts Parser expects `tryLT` to be defined on the token stream
// and `LT` to defer to the `tryLT`.
// @see 'node_modules/antlr4ts/Parser.js
// @see 'node_modules/antlr4ts/BufferedTokenStream.js:203
this._input.tryLT = this._input.LT;
this._input.LT = function (k) {
const result = this.tryLT(k);
if (result === undefined) {
throw new RangeError('requested lookback index out of range');
}
return result;
};
}
}
For now, this resolves the issue for me and I am able to run the antlr4-c3 engine in the browser. Should I run into any other issues, I'll be sure to open a new ticket. Thanks!
Hi,
I am trying to start a project to try out the c3 engine, but I am running into a little obstacle early on. I am setting up the scaffold as outlined in the
Getting Started
section of the README, but I get the following error when I do thecore.collectCandidates(0)
call:I am assuming that there is some initialization that I haven't done, but I cannot figure it out. Below is the setup leading up to the failing call.
This is using a parser generated by
antlr 4.7.1
with a JavaScript target andantlr4-c3 1.1.8
. Hopefully someone can glance at the code and tell me where I am going astray.Stepping through the code, it looks like when I call
c3.CodeCompletionCore(parser)
the member variables get set (includingthis.parser = parser;
), but when I callcore.collectCandidates
, theparser.inputStream
variable is undefined:It looks to me like I am not setting the
tokenStream
correctly since there is no definition of the inputStream on the parser object.