Open shanemcq18 opened 9 months ago
Sorry for the late reply and thanks for the report.
It's related to #44, which has been there for many years. It has not been easy to solve, but I'll try again next week.
This is just for reference. I tried the "textmate grammar" instead of pygments to converting some MATLAB code into tokens to compare the output.
Using your example class
Pygments output (removed consecutive Token.Text: ' '
for readability.
Token.Keyword: 'classdef'
Token.Text: ' '
Token.Name: 'My_Class'
Token.Text: ' '
Token.Operator: '<'
Token.Text: ' '
Token.Name: 'handle'
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Comment: '% Docstring for the class.'
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Keyword: 'methods'
Token.Text: ' '
Token.Punctuation: '('
Token.Name: 'Abstract'
Token.Punctuation: ')'
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Comment: '% How can we document this method?'
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Name: 'val'
Token.Text: ' '
Token.Punctuation: '='
Token.Text: ' '
Token.Name: 'abstractmethod1'
Token.Punctuation: '('
Token.Name: 'arg'
Token.Punctuation: ')'
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Comment: '% Perhaps a docstring here?'
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Comment: '% This is also not documented.'
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Punctuation: '['
Token.Name: 'val1'
Token.Punctuation: ','
Token.Text: ' '
Token.Name: 'val2'
Token.Punctuation: ']'
Token.Text: ' '
Token.Punctuation: '='
Token.Text: ' '
Token.Name: 'abstractmethod2'
Token.Punctuation: '('
Token.Name: 'arg1'
Token.Punctuation: ','
Token.Text: ' '
Token.Name: 'arg2'
Token.Punctuation: ')'
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Keyword: 'end'
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Keyword: 'methods'
Token.Text.Whitespace: '\n'
Token.Keyword: ' \n function'
Token.Text.Whitespace: ' '
Token.Text: '[out] '
Token.Punctuation: '='
Token.Text.Whitespace: ' '
Token.Name.Function: 'publicmethod'
Token.Punctuation: '('
Token.Text: 'arg'
Token.Punctuation: ')'
Token.Text.Whitespace: '\n '
Token.Comment: '% This is documented.'
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Name: 'out'
Token.Text: ' '
Token.Punctuation: '='
Token.Text: ' '
Token.Literal.Number.Integer: '2'
Token.Text: ' '
Token.Operator: '*'
Token.Text: ' '
Token.Name: 'arg'
Token.Punctuation: ';'
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Keyword: 'end'
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Text.Whitespace: '\n'
Token.Text: ' '
...
Token.Text: ' '
Token.Keyword: 'end'
Token.Text.Whitespace: '\n'
Token.Keyword: 'end'
Token.Text.Whitespace: '\n'
Textmate Grammar Output with to_dict
:
{
"token": "source.matlab",
"children": [
{
"token": "meta.class.matlab",
"begin": [{"token": "storage.type.class.matlab", "content": "classdef"}],
"end": [{"token": "storage.type.class.end.matlab", "content": "end"}],
"children": [
{
"token": "meta.class.declaration.matlab",
"children": [
{
"token": "entity.name.type.class.matlab",
"content": "My_Class",
},
{
"token": "punctuation.separator.lt.inheritance.matlab",
"content": "<",
},
{
"token": "meta.inherited-class.matlab",
"children": [
{
"token": "entity.other.inherited-class.matlab",
"content": "handle",
}
],
},
],
},
{
"token": "punctuation.whitespace.comment.leading.matlab",
"content": " ",
},
{
"token": "comment.line.percentage.matlab",
"begin": [
{
"token": "punctuation.definition.comment.matlab",
"content": "%",
}
],
"content": "% Docstring for the class.",
},
{
"token": "meta.methods.matlab",
"begin": [
{
"token": "keyword.control.methods.matlab",
"content": "methods",
},
{
"token": "storage.modifier.methods.matlab",
"content": "Abstract",
},
],
"end": [
{
"token": "keyword.control.end.methods.matlab",
"content": "end",
}
],
"children": [
{
"token": "punctuation.whitespace.comment.leading.matlab",
"content": " ",
},
{
"token": "comment.line.percentage.matlab",
"begin": [
{
"token": "punctuation.definition.comment.matlab",
"content": "%",
}
],
"content": "% How can we document this method?",
},
{
"token": "meta.assignment.variable.single.matlab",
"children": [
{
"token": "variable.other.readwrite.matlab",
"content": "val",
}
],
},
{"token": "keyword.operator.assignment.matlab", "content": "="},
{
"token": "meta.function-call.parens.matlab",
"begin": [
{
"token": "entity.name.function.matlab",
"content": "abstractmethod1",
},
{
"token": "punctuation.section.parens.begin.matlab",
"content": "(",
},
],
"end": [
{
"token": "punctuation.section.parens.end.matlab",
"content": ")",
}
],
"children": [
{
"token": "variable.other.readwrite.matlab",
"content": "arg",
}
],
},
{
"token": "punctuation.whitespace.comment.leading.matlab",
"content": " ",
},
{
"token": "comment.line.percentage.matlab",
"begin": [
{
"token": "punctuation.definition.comment.matlab",
"content": "%",
}
],
"content": "% Perhaps a docstring here?",
},
{
"token": "punctuation.whitespace.comment.leading.matlab",
"content": " ",
},
{
"token": "comment.line.percentage.matlab",
"begin": [
{
"token": "punctuation.definition.comment.matlab",
"content": "%",
}
],
"content": "% This is also not documented.",
},
{
"token": "meta.assignment.variable.group.matlab",
"begin": [
{
"token": "punctuation.section.assignment.group.begin.matlab",
"content": "[",
}
],
"end": [
{
"token": "punctuation.section.assignment.group.end.matlab",
"content": "]",
}
],
"children": [
{
"token": "variable.other.readwrite.matlab",
"content": "val1",
},
{
"token": "punctuation.separator.comma.matlab",
"content": ",",
},
{
"token": "variable.other.readwrite.matlab",
"content": "val2",
},
],
},
{"token": "keyword.operator.assignment.matlab", "content": "="},
{
"token": "meta.function-call.parens.matlab",
"begin": [
{
"token": "entity.name.function.matlab",
"content": "abstractmethod2",
},
{
"token": "punctuation.section.parens.begin.matlab",
"content": "(",
},
],
"end": [
{
"token": "punctuation.section.parens.end.matlab",
"content": ")",
}
],
"children": [
{
"token": "variable.other.readwrite.matlab",
"content": "arg1",
},
{
"token": "punctuation.separator.comma.matlab",
"content": ",",
},
{
"token": "variable.other.readwrite.matlab",
"content": "arg2",
},
],
},
],
},
{
"token": "meta.methods.matlab",
"begin": [
{
"token": "keyword.control.methods.matlab",
"content": "methods",
}
],
"end": [
{
"token": "keyword.control.end.methods.matlab",
"content": "end",
}
],
"children": [
{
"token": "meta.function.matlab",
"begin": [
{
"token": "storage.type.function.matlab",
"content": "function",
}
],
"end": [
{
"token": "storage.type.function.end.matlab",
"content": "end",
}
],
"children": [
{
"token": "meta.function.declaration.matlab",
"children": [
{
"token": "meta.assignment.variable.output.matlab",
"end": [
{
"token": "keyword.operator.assignment.matlab",
"content": "=",
}
],
"children": [
{
"token": "punctuation.section.assignment.group.begin.matlab",
"content": "[",
},
{
"token": "variable.parameter.output.matlab",
"content": "out",
},
{
"token": "punctuation.section.assignment.group.end.matlab",
"content": "]",
},
],
},
{
"token": "entity.name.function.matlab",
"content": "publicmethod",
},
{
"token": "meta.parameters.matlab",
"begin": [
{
"token": "punctuation.definition.parameters.begin.matlab",
"content": "(",
}
],
"end": [
{
"token": "punctuation.definition.parameters.end.matlab",
"content": ")",
}
],
"children": [
{
"token": "variable.parameter.input.matlab",
"content": "arg",
}
],
},
],
},
{
"token": "punctuation.whitespace.comment.leading.matlab",
"content": " ",
},
{
"token": "comment.line.percentage.matlab",
"begin": [
{
"token": "punctuation.definition.comment.matlab",
"content": "%",
}
],
"content": "% This is documented.",
},
{
"token": "meta.assignment.variable.single.matlab",
"children": [
{
"token": "variable.other.readwrite.matlab",
"content": "out",
}
],
},
{
"token": "keyword.operator.assignment.matlab",
"content": "=",
},
{
"token": "constant.numeric.decimal.matlab",
"content": "2",
},
{
"token": "keyword.operator.arithmetic.matlab",
"content": "*",
},
{
"token": "variable.other.readwrite.matlab",
"content": "arg",
},
{
"token": "punctuation.terminator.semicolon.matlab",
"content": ";",
},
],
}
],
},
],
}
],
}
The nice thing with "textmate-grammar" is that tokens are more rich. Still it will require quite a lot a work to convert to "textmate-grammar" parsing.
Abstract methods – which have a signature but not a body – are currently not documented. It would be nice to be able to document them though, since they exist to define an interface for users to extend ("if you inherit from this class, you must write a function that...\<precise explanation that should live in a docstring>").
The trouble is that these methods are special cases lexically because they aren't bookended by
function
andend
. Here's an example.Now with this RST block somewhere,
we get documentation for
publicmethod()
but notabstractmethod1()
orabstractmethod2()
.This was discussed a little way back in #28. Is this something that can be added without too much trouble? Adding
function
andend
is not allowed for abstract methods, so it would have to be handled as a special case. It seems pretty natural to allow docstrings with this style:Thanks for writing this extension, it's a huge help for our project.