as token type: Identifier -> Punctuator

[conartist6]

In the code:

import { foo as bar } from 'source'; 

as is an Identifier token. I believe this is incorrect, since as can never refer to a value of any kind. I would expect as to be a Punctuator token.

Here is the example in astexplorer.

[conartist6]

Interestingly of is considered an identifier too in for (const foo of bar).

I presume this means that the keyword token type is reserved purely for words that are always keywords, i.e. they can never be identifiers. Unfortunately this makes it impossible to look at a token and know if it is actually being used as a keyword or as an identifier.

[nzakas]

Yup, this is just one of the quirks of working with JavaScript tokenization. We follow the original approach that Esprima took, which is that known keywords are of type Keyword, other words are type Identifier, and only non-word symbols are Punctuators. We can’t change that now without breaking the ecosystem.

[conartist6]

Alrighty. I think I'm going to do something different in cst-tokens since I don't have any ecosystem-breaking burden. Maybe I'll introduce something like type: 'Word' to avoid diluting the meaning of type: 'Keyword'.