Rollback the syntax highlighter and plot a more comprehensive and community engaged course for its future

[bcardarella]

The latest version of elixir-ls introduced what I'm finding to be a very annoying syntax highlighting experience. The new highlighter is far too aggressive in its tokenization and perhaps not nuanced enough in its token grouping. Here is the before/after of my editor using the Atom One Dark there:

Before:

After:

After engaging with @lukaszsamson on the Elixir slack on the topic he seems dismissive of how annoying and jarring these changes can be for people. While there can be something said about improving the syntax highlighter I'd put forward that such changes should be done with greater care than just a release. For example, now we have different syntax highlighting methods on GitHub and Livebook than in elixir-ls:

elixir-ls:

GitHub:

Livebook:

The token highlighting is consistent between GitHub and Livebook but now elixir-ls is inconsistent.

This may feel like a small thing to some but to others it is a jarring experience and impacts productivity.

What I would suggest:

1. patch release to rollback the syntax highlighting recently introduced to v0.16
2. solicit community feedback on these changes, present on the proposed changes and show the before/after. Break down the tokenization and the groupings
3. work with other syntax highlighting implementations also derived from Atom such at GitHub and Livebook to ensure consistency across the ecosystem.

[lukaszsamson]

After engaging with @lukaszsamson on the Elixir slack on the topic he seems dismissive

On the contrary. The issue was reported 5 days ago https://github.com/elixir-lsp/vscode-elixir-ls/issues/382 and after healthy discussion I released a first patch release.

For example, now we have different syntax highlighting methods on GitHub and Livebook than in elixir-ls

Those are separate projects using different grammars (textmate vs treesitter). It's unrealistic to expect they would look the same.

it is a jarring experience and impacts productivity

There is a known and easy workaround - theme customisation - already suggested by other folks in https://github.com/elixir-lsp/vscode-elixir-ls/issues/382#issuecomment-1758112397. BTW some VSCode extensions upfront declare a list of themes they look good on and don't care about other.

patch release to rollback the syntax highlighting recently introduced to v0.16

I'm against rollback to reintroduce broken things. Changes are necessary to make way for new features. Patch releases may come if there are concrete things to improve.

solicit community feedback on these changes, present on the proposed changes and show the before/after. Break down the tokenization and the groupings

The discussion is already happening in https://github.com/elixir-lsp/vscode-elixir-ls/issues/382. Please head there and propose concrete improvements.

work with other syntax highlighting implementations also derived from Atom such at GitHub and Livebook to ensure consistency across the ecosystem.

I'm afraid you have unrealistic expectations from what is basically one person driven OSS project. If you have the resources to help with the effort then please do.

[bcardarella]

@lukaszsamson I've already moved over to next-ls. I encourage others to do the same.

[bcardarella]

@lukaszsamson and just for the historical record, the suggestion here that you've dismissed is that you shouldn't be introducing such changes without community engagement. It's irresponsible and you lose people when you do such things.