rsmenon / pygments-mathematica

Mathematica/Wolfram Language lexer and highlighter for Pygments
MIT License
59 stars 8 forks source link

Highlight is wrong if a string's closing quote is at the start of a line #18

Open magv opened 1 year ago

magv commented 1 year ago

Hi, all. There's a highlight problem with multi-line strings: if the closing quote is at the start of the line, it is ignored and the string continues past it.

Here's a test case:

import pygments
import mathematica
print(list(pygments.lex('"string\n"variable', mathematica.MathematicaLexer())))

... which prints:

[(Token.Literal.String, '"'),
 (Token.Literal.String, 'string\n'),
 (Token.Literal.String, '"'),
 (Token.Literal.String, 'variable\n')]

As you can see, variable is marked as a string. The same doesn't happen if the closing quote is preceded by a character:

import pygments
import mathematica
print(list(pygments.lex('"string\nx"variable', mathematica.MathematicaLexer())))

This prints:

[(Token.Literal.String, '"'),
 (Token.Literal.String, 'string\nx'),
 (Token.Literal.String, '"'),
 (Token.Name.Variable, 'variable'),
 (Token.Text.Whitespace, '\n')]

(Not sure where the final '\n' whitespace is coming from in this case, but this is harmless).