search

bablr-lang / bablr-vm

A VM for enforcing language rules on agAST trees
MIT License
40 stars 2 forks source link

Rename `node.tokens` #4

Closed conartist6 closed 2 years ago

conartist6 commented 2 years ago

We need to carefully consider the best way to attach tokens to nodes.

node.tokens property has existing usages which would cause conflicts and breakages.

node[Symbol.for('cst/tokens')] would make the data format less usable -- harder to inspect, serialize, deserialize, etc.

node.cstTokens or node.ownTokens seems like it would probably be best.

gibson042 commented 2 years ago

The ECMAScript specification uses the term "input element" (and note that tokens are a strict subset of input elements, with line terminators, linear whitespace, and comments being included in the latter category but not the former). In https://github.com/estree/estree/issues/41#issuecomment-131636382 , I used sourceElements as the closest synonym I could find that didn't read strangely when removed from the context of that specification.

conartist6 commented 2 years ago

I'm leaning towards node.cstTokens. If I can't use a name directly from the spec I don't think cstTokens has an appreciably different meaning than sourceElements.