switch to a different tool for python syntax highlighting

DetachHead commented 6 months ago

MagicPython doesn't seem to have been updated in 2 years, meaning it doesn't support any of the new syntax in python 3.12:

it seems very odd that these new features came out 6 months ago and are still not supported properly in vscode.

there's also the issue with raw strings being incorrectly treated as regex which has been raised many times:

i think this is a big source of confusion for users (as you can see from how many times it's been raised), and should be addressed. i think @gilbertohasnofb makes a good argument in https://github.com/MagicStack/MagicPython/issues/259#issuecomment-1879727479:

AFAIK this is not a standard defined in any PEP, and popular code formatter such as black will convert any R'' strings into r'' by default.

I get the reasoning behind this decision, and there were some arguments for it since TextMate and Sublime Text both highlight Python syntax this way by default, but with the widespread use of VSCode these days (which does not highlight raw strings in Python this way by default) this argument seems outdated. Ideally this idiosyncratic highlighting could be made optional, preferably off by default.

rchiodo commented 6 months ago

Thanks for the issue. We actually have an internal issue already tracking this: https://github.com/microsoft/pyrx/issues/4026, but we can use this issue as the external facing problem.

JelleZijlstra commented 6 months ago

AFAIK this is not a standard defined in any PEP, and popular code formatter such as black will convert any R'' strings into r'' by default.

Just noting that this part isn't true; Black does not convert between R'' and r'', specifically because MagicPython syntax highlights them differently. I don't have a strong opinion on whether Pylance should continue to treat the two prefixes differently: on the one hand it's a surprising and non-obvious behavior, on the other hand it is nice to get a way to syntax-highlight regexes.

DetachHead commented 6 months ago

i see this issue was moved to the pylance repo. from my understanding the syntax highlighting provided by magicpython is bundled with vscode and separate to the semantic highlighting provided by pylance. is the plan to remove that basic syntax highlighting entirely and instead have it handled entirely by pylance? if so, it would be a downgrade for users who are not using pylance (eg. vscodium/basedpyright users)

rchiodo commented 5 months ago

The plan is to fix the colors of text in Python code. We haven't really started investigating how or where we'd implement such a thing, but we realize the textmate solution that uses MagicPython's tmlanguage isn't really working out.

debonte commented 5 months ago

VS Code is exploring using Tree-sitter for syntax highlighting. If they move forward with that, the current thinking is that each language could choose either TextMate or Tree-sitter depending on which implementation is more up-to-date / better maintained -- https://github.com/microsoft/vscode/issues/210475

For internal people, there was a discussion about this work at the beginning of the 4/30 "Language Teams and LSP Weekly Sync" meeting.

heejaechang commented 5 months ago

did we drop the idea of supplying textmate rule in pylance rather than using one supplied by vscode?

debonte commented 5 months ago

did we drop the idea of supplying textmate rule in pylance rather than using one supplied by vscode?

No, I think that's still on the table. Although Dirk mentioned in the LSP sync that fixing bugs / enhancing existing TextMate grammars can be incredibly painful.

JackismyShephard commented 1 week ago

What is the ETA on getting these issues fixed? The no syntax highlighting on type definitions and generics is really a thorn in my side 😋

microsoft / pylance-release

switch to a different tool for python syntax highlighting #5824