Python-like string detection collides with Matlab transpose

[joeced]

Originally reported by: Lukas Drude (Bitbucket: lukas_drude, GitHub: Unknown)

---

Matlab uses

#!matlab

.'

to indicate a transposition of a vector.

Sphinx seems to handle this as the beginning of a string when written within a class definition:

#!matlab

classdef MinimalExample
    methods
        function checkSizeOfOutput(this)
            % Working
            a = [1 2].';%'
            b = (a');

            % Broken
            a = [1 2].';
            b = (a');

            % Working
            a = [1 2].';
            b = (a';
        end
    end
end

The first example is working because the apostrophe in the comment seems to close the accidentally detected beginning of a string.

The second one does not work, because the first parenthesis is within the accidentally detected string. Thus, the closing parenthesis results in an error.

The third example is not valid Matlab code but does not cause a Sphinx-error, because the opening parenthesis is again of the accidentally detected string.

When the code is not within a class, all three cases work fine.

---

• Bitbucket: https://bitbucket.org/bwanamarko/sphinxcontrib-matlabdomain/issue/31

[joeced]

Original comment by Mark Mikofski (Bitbucket: bwanamarko, GitHub: Unknown):

---

setup.py edited online with Bitbucket

• require Pygments 2.0.1 and Sphinx 1.2
• fixes #31

[joeced]

Original comment by Lukas Drude (Bitbucket: lukas_drude, GitHub: Unknown):

---

Yes, this resolves it.

[joeced]

Original comment by Mark Mikofski (Bitbucket: bwanamarko, GitHub: Unknown):

---

@lukas_drude, added Pygments-2.0.1 to setup.py requires attribute. Is this sufficient to resolve this issue? if not feel free to reopen it. thx!

[joeced]

Original comment by Mark Mikofski (Bitbucket: bwanamarko, GitHub: Unknown):

---

setup.py edited online with Bitbucket

• require Pygments 2.0.1 and Sphinx 1.2
• fixes #31

[joeced]

Original comment by Lukas Drude (Bitbucket: lukas_drude, GitHub: Unknown):

---

Ideed, pygments is available in a new version 2.0.1.

You get your pygments version number with this:

#!bash

pip freeze | grep Pygm

You can update it with:

#!bash

sudo pip install -u Pygments

Now the output of the tokenizer looks like this:

#!

(Token.Name, u'a')
(Token.Text, u' ')
(Token.Punctuation, u'=')
(Token.Text, u' ')
(Token.Punctuation, u'[')
(Token.Literal.Number.Integer, u'1')
(Token.Text, u' ')
(Token.Literal.Number.Integer, u'2')
(Token.Punctuation, u']')
(Token.Punctuation, u'.')
(Token.Operator, u"'")
(Token.Punctuation, u';')
(Token.Comment, u"%'")
(Token.Text, u'\n')

[joeced]

Original comment by Lukas Drude (Bitbucket: lukas_drude, GitHub: Unknown):

---

I found this resolved bug on their list: Matlab highlighting inconsistencies: after the transpose .' the text becomes a string.

So do we just need to wait for the next pygments release?

[joeced]

Original comment by Mark Mikofski (Bitbucket: bwanamarko, GitHub: Unknown):

---

Note: From MATLAB docs

The dot-apostrophe operator (A.'), transposes without affecting the sign of complex elements. For matrices containing all real elements, the two operators return the same result.

[joeced]

Original comment by Mark Mikofski (Bitbucket: bwanamarko, GitHub: Unknown):

---

The issue is that Pygments does not recognize .' (dot apostrophe) as an operator. Remove the dot and it works, otherwise, you are correct, it interprets the apostrophe as literal text. If you want create an issue on Pygments. Otherwise maybe we can try use regex-replace to try to replace the dot-apostrophe with just the apostrophe.

Here is your code sample parsed by Pygments.

>>> from pygments.lexers import MatlabLexer
>>> ml = Matlablexer()
>>> cd = """classdef Minimal example...""" # put your code example here
>>> for tk in ml.get_tokens(cd):
>>>     print tk
(Token.Keyword, u'classdef')
(Token.Text, u' ')
(Token.Name, u'MinimalExample')
(Token.Text, u'\n    ')
(Token.Keyword, u'methods')
(Token.Text, u'\n        ')
(Token.Keyword, u'function')
(Token.Text.Whitespace, u' ')
(Token.Name.Function, u'checkSizeOfOutput')
(Token.Punctuation, u'(')
(Token.Text, u'this')
(Token.Punctuation, u')')
(Token.Text.Whitespace, u'\n            ')
(Token.Comment, u'% Working')
(Token.Text, u'\n            ')
(Token.Name, u'a')
(Token.Text, u' ')
(Token.Punctuation, u'=')
(Token.Text, u' ')
(Token.Punctuation, u'[')
(Token.Literal.Number.Integer, u'1')
(Token.Text, u' ')
(Token.Literal.Number.Integer, u'2')
(Token.Punctuation, u']')
(Token.Punctuation, u'.')
(Token.Literal.String, u"'")  # <-- apostrophe misinterpreted as literal text
(Token.Literal.String, u";%'")
(Token.Text, u'\n            ')
(Token.Name, u'b')
(Token.Text, u' ')
(Token.Punctuation, u'=')
(Token.Text, u' ')
(Token.Punctuation, u'(')
(Token.Name, u'a')
(Token.Operator, u"'")  # <-- apostrophe works correctly if no dot
(Token.Punctuation, u')')
(Token.Punctuation, u';')
(Token.Text, u'\n\n            ')
(Token.Comment, u'% Broken')
(Token.Text, u'\n            ')
(Token.Name, u'a')
(Token.Text, u' ')
(Token.Punctuation, u'=')
(Token.Text, u' ')
(Token.Punctuation, u'[')
(Token.Literal.Number.Integer, u'1')
(Token.Text, u' ')
(Token.Literal.Number.Integer, u'2')
(Token.Punctuation, u']')
(Token.Punctuation, u'.')
(Token.Literal.String, u"'")  # <-- apostrophe misinterpreted as literal text
(Token.Literal.String, u";\n            b = (a'")
(Token.Punctuation, u')')
(Token.Punctuation, u';')
(Token.Text, u'\n\n            ')
(Token.Comment, u'% Working')
(Token.Text, u'\n            ')
(Token.Name, u'a')
(Token.Text, u' ')
(Token.Punctuation, u'=')
(Token.Text, u' ')
(Token.Punctuation, u'[')
(Token.Literal.Number.Integer, u'1')
(Token.Text, u' ')
(Token.Literal.Number.Integer, u'2')
(Token.Punctuation, u']')
(Token.Punctuation, u'.')
(Token.Literal.String, u"'")  # <-- apostrophe misinterpreted as literal text
(Token.Literal.String, u";\n            b = (a'")
(Token.Punctuation, u';')
(Token.Text, u'\n        ')
(Token.Keyword, u'end')
(Token.Text, u'\n    ')
(Token.Keyword, u'end')
(Token.Text, u'\n')
(Token.Keyword, u'end')
(Token.Text, u'\n')