[WIP] Further improvements to textmate based parser

apozharski commented 3 months ago

Better classdef attributes and validator function parsing based on improvements to MATLAB-language-grammar in:

and a textmate-grammar-python pr which bumps the MATLAB-language-grammar (which will be created once Mathworks pulls in the improvements).

Further improvements:

parsing docstrings for functions

joeced commented 3 months ago

The tests, (see https://github.com/sphinx-contrib/matlabdomain/actions/runs/10058995696/job/27803269388?pr=258#step:4:106) are failing as setup.py does not require the textmate grammar package.

apozharski commented 3 months ago

Yep I have just been running pytest locally to get through the true errors. Unfortunately it looks like there are fundamental problems with the textmate grammar that I am not sure are solvable due to the limitations of the textmate grammar. In particular the syntax for multiline defaults is I believe unparseable. @watermarkhu I am unsure if you have any insights that I do not here.

An example of the problem:

prop = [1;
              2;
              3]; % a prop

in which one cannot unambiguously identify which of the ; actually ends the property default.

joeced commented 3 months ago

It's ok for the test to fail for now. We'll get there eventually. For the default value of properties the current code starts here: https://github.com/sphinx-contrib/matlabdomain/blob/b081e7c3473097ff41f4d9dc1f666cb6bdfebd3e/sphinxcontrib/mat_types.py#L1275 and uses more than 50 lines to collect it. I hope that textmate grammar can do it in less :)

watermarkhu commented 3 months ago

Yep I have just been running pytest locally to get through the true errors. Unfortunately it looks like there are fundamental problems with the textmate grammar that I am not sure are solvable due to the limitations of the textmate grammar. In particular the syntax for multiline defaults is I believe unparseable. @watermarkhu I am unsure if you have any insights that I do not here.

An example of the problem:
prop = [1;
              2;
              3]; % a prop
in which one cannot unambiguously identify which of the ; actually ends the property default.

There are two ways to tackle this.

Further edits to the MATLAB grammar: The MATLAB grammar can be further improved to catch these cases. In this case, we can add a dummy newline scope to the array and cell scopes.
Pre-parser:. Similar to the current case, there is MATLAB's line continuations ..., which is allowed to be inserted almost anywhere. I started on adding the line-continuation scope to places that I encountered them, but there are many, many more.
Just for the sake of getting the tokens, I added a pre-parser to simply remove line-continuations from the source file. While this method is far from ideal, it is another possibility.

apozharski commented 2 months ago

This is replaced with #261

sphinx-contrib / matlabdomain

[WIP] Further improvements to textmate based parser #258