Improved type safety for token names

[LaurensRietveld]

Take this code block:

const Filter = createToken({
  name: "Filter" as const,
  pattern: new RegExp("filter", "i"),
}) 
const functionName = this.CONSUME(Filter);
return { type: functionName.tokenType.name };

Considering the name of the TokenType interface is always string, the return type of this function is {type: string}.

Where the above snippet is simple/small, this creates overhead for larger grammars where better type information is wanted, and adds the burden of specifying more accurate types to the developer. Ideally, the types are derived automatically, making the returntype of the MWE type: "Filter"

[LaurensRietveld]

I managed to work around this by overwriting the CONSUME function, and using my own createToken function. See here:

function createToken<N extends string>(config: Omit<ITokenConfig, "name"> & { name: N }) {
  return origCreateToken(config) as Omit<TokenType, "name"> & { name: N };
}
class ... {
  public CONSUME<S extends TokenType>(token:S) {
    return super.CONSUME(token) as Omit<IToken, "tokenType"> & {tokenType: S}
  }
}

[khill-fbmc]

This is what I ended up adding to your example:

import type { ConsumeMethodOpts, IToken, TokenType } from "chevrotain";

public override CONSUME<S extends TokenType>(
  token: S,
  options?: ConsumeMethodOpts
) {
  return super.CONSUME(token, options) as Omit<IToken, "tokenType"> & {
    tokenType: S;
  };
}