Transformer is not applied for tokens in standalone parser

[BlankSpruce]

Describe the bug Standalone parser doesn't invoke methods on tokens defined in supplied transformer

To Reproduce Given:

• grammar example.lark:

  start: IDENTIFIER "(" text ")"
  IDENTIFIER: /[A-Za-z_][A-Za-z0-9_]*/
  text: IDENTIFIER

• standalone parser generated with: python3 -m lark.tools.standalone example.lark > example.py

Then:

• this program:

  
  import lark

class DowncaseIdentifiers(lark.Transformer): def IDENTIFIER(self, token): return lark.Token("IDENTIFIER", token.lower())

def text(self, children):
    return lark.Tree("text", [self.IDENTIFIER(children[0])])

parser = lark.Lark.open( grammar_filename="example.lark", transformer=DowncaseIdentifiers(), parser="lalr" ) to_parse = "FOO(BAR)" result = parser.parse(to_parse) print(result.pretty())

produces:

start foo text bar

- this program:
```python
import example

class DowncaseIdentifiers(example.Transformer):
    def IDENTIFIER(self, token):
        return example.Token("IDENTIFIER", token.lower())

    def text(self, children):
        return example.Tree("text", [self.IDENTIFIER(children[0])])

parser = example.Lark_StandAlone(transformer=DowncaseIdentifiers())
to_parse = "FOO(BAR)"
result = parser.parse(to_parse)
print(result.pretty())

produces:

start
  FOO
  text  bar

Expected: Both programs produce the same output, namely:

start
  foo
  text  bar

[techpixel]

Your import from example probably dosen't have the module Transformer.

Also, I might know your issue. Make sure you have visit_tokens set to true. This lets the Transformer visit both tokens AND trees

[MegaIng]

Here is the culprit: https://github.com/lark-parser/lark/blob/6925b9be0d9e7a546336e4e5e87b28291c53dea3/lark/common.py#L21