There's no mechanism to define token visitors.

[willow-ahrens]

Also, the current code to visit tokens is creating a lot of Val types, since it passes the data of the tokens into the Val instead of the contents of the type_ field, as one might expect. Perhaps this is why visiting tokens is currently so slow. The code in question is https://github.com/jamesrhester/Lerche.jl/blob/380abe9f1da529e656e0e9613efdd3edf9c99a80/src/visitors.jl#L128 I think the issue is that Token iterates the data and not the name of the token, so the iterator constructor for Symbol gets data.

Anyways, to make things work, I wrote something like this:

Lerche._call_userfunc_token(t::Transformer,token) = begin
    return Lerche.token_func(t,Val(Symbol(token.type_)),token)
end
macro terminal(s)
    if s.head != :(=) || s.args[1].head != :call
        error("A rule must be a function definition")
    end    
    rule_name = QuoteNode(s.args[1].args[1])
    if s.args[1].args[2].head != :(::)
        error("Type must be included in the first argument to define a rule: $s")
    end
    rule_type = s.args[1].args[2].args[2] # the type name
    esc(quote
        Lerche.token_func($(s.args[1].args[2]),::Val{Symbol($rule_name)},$(s.args[1].args[3])) = $(s.args[2])
    end )
end

which allows me to call a terminal macro analogous to the rule macro.

Thanks for writing this package by the way! I'm quite enjoying my parsing experience so far :)

[jamesrhester]

Thanks for the suggestion, you are right, forming a Julia Symbol from a token is silly as the _type entry is the correct thing to switch on, and that is what Lark does. I'll make a fix and do some benchmarking.

I'll also add the @terminal macro, thanks for the suggestion.

[jamesrhester]

Commit https://github.com/jamesrhester/Lerche.jl/commit/342065a64e7dc55bbf7cfc407ffc9786ce32e824 includes the fix and terminal macro definition that you supplied. I see your handle is now up in lights on the code page, hope that's not a problem! Will release this as version 0.5.0

[willow-ahrens]

Awesome! Everything looks like it's working in version 0.5.0, I'll close this issue.