search

elixir-makeup / makeup_erlang

Erlang lexer for Makeup
3 stars 6 forks source link

Fix error when trying to lex text with the binary sintax in it #3

Closed mracos closed 5 years ago

mracos commented 5 years ago

Why?

Before this change, it was raising a FunctionClauseError in the post process function Makeup.Lexers.ErlangLexer.__as_erlang_language/1 when trying to lex text with binary syntax in it (<< and >>).

iex(1)> Makeup.Lexers.ErlangLexer.Testing.lex("<<>>")
** (FunctionClauseError) no function clause matching in Makeup.Lexers.ErlangLexer.__as_erlang_language__/1

    The following arguments were given to Makeup.Lexers.ErlangLexer.__as_erlang_language__/1:

        # 1
        "<<"

    Attempted function clauses (showing 1 out of 1):

        def __as_erlang_language__({ttype, meta, value})

    (makeup_erlang) lib/makeup/lexers/erlang_lexer.ex:201: Makeup.Lexers.ErlangLexer.__as_erlang_language__/1
    (elixir) lib/enum.ex:1336: Enum."-map/2-lists^map/1-0-"/2
    (makeup_erlang) Makeup.Lexers.ErlangLexer.root_element__2/6
    (makeup_erlang) Makeup.Lexers.ErlangLexer.root__2/6
    (makeup_erlang) Makeup.Lexers.ErlangLexer.root/2
    (makeup_erlang) lib/makeup/lexers/erlang_lexer.ex:355: Makeup.Lexers.ErlangLexer.lex/2
    (makeup_erlang) lib/makeup/lexers/erlang_lexer/testing.ex:13: Makeup.Lexers.ErlangLexer.Testing.lex/1

How?

It was erroring out because we were not wrapping the binary syntax combinators with the Makeup.Lexer.Combinators.token/2 combinator, which returns the result that __as_erlang_language/1 expects.

It was being passed a literal "<<" instead of the token tuple that is expected. e.g. {:token_name, attrs = %{}, literal = "<<"}.

Implementation

Just adding both << and >> to the punctuation combinator made it work, since it was already being wrapped by the token/2 combinator.

I'm just not that sure if it really belongs to the punctuations one, WDYT? The comment that was there also raises the same question.