Tooling support

[viridia]

Assuming this PEP gets approved and adopted, it is going to have a major impact on the dozens of support tools out there: linters, IDE plugins, code formatters, syntax highlighters and so on. The introduction of a new statement type and two new "soft" keywords (match and case) will require adjustment to these tools.

Although I don't think this group needs to do anything about this or make any actual decisions, I think it's worth noting what the impacts will be.

As a way of thinking about this, I will divide the world of Python source tools into two groups, which I'll call "shallow" and "deep".

• Shallow tools do not have a complete understanding of Python grammar, but instead scan the text looking for patterns, often using regular expressions. Examples of this kind of tool are TextMate and the Python plugin for VSCode: they understand the basic keywords, as well as indent/dedent and lines ending with a colon, but don't actually parse expressions or statement structure. This makes sense considering that the source code is frequently syntactically invalid while being edited, so a strict approach would fail most of the time.

• Deep tools are ones that completely parse the Python grammar. An example of this is black, which we discussed in another context.

For the shallow tools, adding support for match basically consists of adding two new entries to the keyword table. Depending on how they scan the text, they may or may not have to add support for distinguishing soft keywords at the start of a line from other uses of that name.

Here's an example from the vscode-python repo: https://github.com/microsoft/vscode-python/blob/master/src/client/language/languageConfiguration.ts#L52

For the deep tools, obviously they will have to add the match statement syntax to their grammar definition, including support for soft keywords.

[viridia]

One thing that might be helpful is to document the complete grammar of match in an appendix to the PEP. (Currently the PEP contains a section containing a simplified version of the grammar, but a deep tool implementer is going to want the complete grammar.)

[gvanrossum]

I wonder if this is going to be a show-stopper? We may be putting a big burden on the deep tools in particular.

Note that certain deep tools will be affected less, or at least not in their parser: those that use ast.parse() (like mypy). They will still have to learn how to visit Match and match_case nodes though, but at least the syntax definition is covered.

We would do well to put some effort into converting LibCST to PEG, though.

I created #65 with a task to add the full grammar to the PEP.

[viridia]

This shouldn't be a show-stopper, otherwise you'd never be able to add a new Python statement again.

In fact, I think the choices we have made have been, overall, fairly friendly to third-party tools - like for example, making the indentation of the case clauses consistent with other statement forms.

Also, even if this feature lands in some future Python version, most users won't upgrade to that version for a while. That will give the third-party tools some time to catch up. Users who want to live on the bleeding edge will have to either suck it up, or pester their favorite tools for support, or possibly submit PRs.

[gvanrossum]

That sounds like a fair assessment.

As an experiment, I updated my favorite python-mode.el for Emacs to support this. I found 9 (!) separate lists of keywords, but after updating all of them (in the hope that some of the changes mattered) it worked flawlessly for match and case.

[viridia]

How much effort do you think is involved in moving LibCST to PEG? It seems like a good idea if you plan to have any PEG-related functionality in Python in the future.

[gvanrossum]

It would be a significant effort. But we have a good starting point: the pegen parser generator can generate Python code too (with Python expressions in the actions of course). Maybe it wouldn't be that much effort to just investigate how LibCST currently constructs a parse tree? IIUC it currently uses 'parso', which itself is a fork of lib2to3.

[viridia]

This seems like an ideal Summer of Code project.