Closed joeced closed 6 years ago
Original comment by Mark Mikofski (Bitbucket: bwanamarko, GitHub: Unknown):
setup.py edited online with Bitbucket
Original comment by Mark Mikofski (Bitbucket: bwanamarko, GitHub: Unknown):
@lukas_drude, added Pygments-2.0.1 to setup.py requires
attribute. Is this sufficient to resolve this issue? if not feel free to reopen it. thx!
Original comment by Mark Mikofski (Bitbucket: bwanamarko, GitHub: Unknown):
setup.py edited online with Bitbucket
Original comment by Lukas Drude (Bitbucket: lukas_drude, GitHub: Unknown):
Ideed, pygments is available in a new version 2.0.1.
You get your pygments version number with this:
#!bash
pip freeze | grep Pygm
You can update it with:
#!bash
sudo pip install -u Pygments
Now the output of the tokenizer looks like this:
#!
(Token.Name, u'a')
(Token.Text, u' ')
(Token.Punctuation, u'=')
(Token.Text, u' ')
(Token.Punctuation, u'[')
(Token.Literal.Number.Integer, u'1')
(Token.Text, u' ')
(Token.Literal.Number.Integer, u'2')
(Token.Punctuation, u']')
(Token.Punctuation, u'.')
(Token.Operator, u"'")
(Token.Punctuation, u';')
(Token.Comment, u"%'")
(Token.Text, u'\n')
Original comment by Lukas Drude (Bitbucket: lukas_drude, GitHub: Unknown):
I found this resolved bug on their list: Matlab highlighting inconsistencies: after the transpose .' the text becomes a string.
So do we just need to wait for the next pygments release?
Original comment by Mark Mikofski (Bitbucket: bwanamarko, GitHub: Unknown):
Note: From MATLAB docs
The dot-apostrophe operator (A.'), transposes without affecting the sign of complex elements. For matrices containing all real elements, the two operators return the same result.
Original comment by Mark Mikofski (Bitbucket: bwanamarko, GitHub: Unknown):
The issue is that Pygments does not recognize .'
(dot apostrophe) as an operator. Remove the dot and it works, otherwise, you are correct, it interprets the apostrophe as literal text. If you want create an issue on Pygments. Otherwise maybe we can try use regex-replace to try to replace the dot-apostrophe with just the apostrophe.
Here is your code sample parsed by Pygments.
>>> from pygments.lexers import MatlabLexer
>>> ml = Matlablexer()
>>> cd = """classdef Minimal example...""" # put your code example here
>>> for tk in ml.get_tokens(cd):
>>> print tk
(Token.Keyword, u'classdef')
(Token.Text, u' ')
(Token.Name, u'MinimalExample')
(Token.Text, u'\n ')
(Token.Keyword, u'methods')
(Token.Text, u'\n ')
(Token.Keyword, u'function')
(Token.Text.Whitespace, u' ')
(Token.Name.Function, u'checkSizeOfOutput')
(Token.Punctuation, u'(')
(Token.Text, u'this')
(Token.Punctuation, u')')
(Token.Text.Whitespace, u'\n ')
(Token.Comment, u'% Working')
(Token.Text, u'\n ')
(Token.Name, u'a')
(Token.Text, u' ')
(Token.Punctuation, u'=')
(Token.Text, u' ')
(Token.Punctuation, u'[')
(Token.Literal.Number.Integer, u'1')
(Token.Text, u' ')
(Token.Literal.Number.Integer, u'2')
(Token.Punctuation, u']')
(Token.Punctuation, u'.')
(Token.Literal.String, u"'") # <-- apostrophe misinterpreted as literal text
(Token.Literal.String, u";%'")
(Token.Text, u'\n ')
(Token.Name, u'b')
(Token.Text, u' ')
(Token.Punctuation, u'=')
(Token.Text, u' ')
(Token.Punctuation, u'(')
(Token.Name, u'a')
(Token.Operator, u"'") # <-- apostrophe works correctly if no dot
(Token.Punctuation, u')')
(Token.Punctuation, u';')
(Token.Text, u'\n\n ')
(Token.Comment, u'% Broken')
(Token.Text, u'\n ')
(Token.Name, u'a')
(Token.Text, u' ')
(Token.Punctuation, u'=')
(Token.Text, u' ')
(Token.Punctuation, u'[')
(Token.Literal.Number.Integer, u'1')
(Token.Text, u' ')
(Token.Literal.Number.Integer, u'2')
(Token.Punctuation, u']')
(Token.Punctuation, u'.')
(Token.Literal.String, u"'") # <-- apostrophe misinterpreted as literal text
(Token.Literal.String, u";\n b = (a'")
(Token.Punctuation, u')')
(Token.Punctuation, u';')
(Token.Text, u'\n\n ')
(Token.Comment, u'% Working')
(Token.Text, u'\n ')
(Token.Name, u'a')
(Token.Text, u' ')
(Token.Punctuation, u'=')
(Token.Text, u' ')
(Token.Punctuation, u'[')
(Token.Literal.Number.Integer, u'1')
(Token.Text, u' ')
(Token.Literal.Number.Integer, u'2')
(Token.Punctuation, u']')
(Token.Punctuation, u'.')
(Token.Literal.String, u"'") # <-- apostrophe misinterpreted as literal text
(Token.Literal.String, u";\n b = (a'")
(Token.Punctuation, u';')
(Token.Text, u'\n ')
(Token.Keyword, u'end')
(Token.Text, u'\n ')
(Token.Keyword, u'end')
(Token.Text, u'\n')
(Token.Keyword, u'end')
(Token.Text, u'\n')
Originally reported by: Lukas Drude (Bitbucket: lukas_drude, GitHub: Unknown)
Matlab uses
to indicate a transposition of a vector.
Sphinx seems to handle this as the beginning of a string when written within a class definition:
The first example is working because the apostrophe in the comment seems to close the accidentally detected beginning of a string.
The second one does not work, because the first parenthesis is within the accidentally detected string. Thus, the closing parenthesis results in an error.
The third example is not valid Matlab code but does not cause a Sphinx-error, because the opening parenthesis is again of the accidentally detected string.
When the code is not within a class, all three cases work fine.