The difference between 'text' and 'value' fields in token info?

[gavvvr]

Hi. I was wondering what is the difference between 'text' and 'value' in token info? From my experience they are always equal. Are there any situations where they can have any difference? Please describe the difference in ReadMe. 'text' field is not mentioned here

[nathan]

The fields differ if you provide a transform, e.g.:

const moo = require('moo')

const lexer = moo.compile({
  ws: /[ \t]+/,
  string: {match: /"(?:\\["\\]|[^\n"\\])*"/, value: s => s.slice(1, -1)},
})

lexer.reset('"test"')
lexer.next() /* { value: 'test', text: '"test"', ... } */

We should update the Token Info section to include text and explain the difference.

[gavvvr]

Thanks for the explanation. Agree, it should be added to readme

[tjvr]

I've added a new section to the Readme.

Thanks for your question!