How to debug syntax highlighting?

[ambv]

Hi, first of all I'm really impressed with the project so far. The idea is awesome, the progress so far is impressive. I'm seriously rooting for it.

Problem

I have a syntax theme I built that is largely inspired by Vim's built-in Delek. You can find it here: https://open-vsx.org/extension/ambv/theme-chromodynamics-delek

It works fine in VScode with no warnings AFAICT. In Onivim2 the window is colored right but no syntax highlighting at all can be seen. This is true for all file types I tried (Python, Lua, JS, C++, HTML, etc.)

Compare VScode:

Onivim2:

How to debug?

Now, given the amount of work that's ahead of you I'd like to debug this myself and fix the theme (I guess there's something wrong about it, the other themes I tried worked.) However, I need some pointers as to how I can debug this. I tried running oni2 -f but it's pretty chatty. oni2 -f --quiet is less so but I didn't see anything syntax highlighting related apart from the proper registration of my theme as a plugin and some unhandled messages from the host around $registerTextEditorDecorationType.

[CrossR]

Thanks for the kind words!

The easiest way to debug this issue would probably be with my PR (https://github.com/onivim/oni2/pull/2255) which adds a debug tool to check the tags (from textmate) and colours (textmate -> theme). It was originally added to track down a specific issue, but I should probably tidy/update it/merge it, since its useful generally for these sorts of issues. I also show in there the way to Compare against VSCode.

That would give us the source of the error at least, is it an error going from code to syntax tags (i.e. a textmate error, which is unlikely) or a textmate to colours error, which implies the theme is being loaded/setup wrong.

Its probably useful for us to generally hook up the same debug floats that Code has, to query the scopes and colours dynamically.

The second way is using the debug log, though you'll probably want to include --log-filter "Syntax", but I don't know how much info we currently log about this sort of thing.

I expect this is going to be some form of fallback issue, where we've not fully sorted out the "if no colour, fallback to C then B then A". If that is the case, it will cause issues with a decent number of themes.

[ambv]

The second way is using the debug log, though you'll probably want to include --log-filter "Syntax", but I don't know how much info we currently log about this sort of thing.

There's not much indeed.

log with `--trace --log-filter "Syntax"`

[INFO]  [ 0.141s] Oni2.Service_Syntax : Init called
[DEBUG] [ 0.142s] Oni2.Syntax.Client : Starting executable: /Applications/Onivim2.app/Contents/MacOS/Oni2_editor and parentPid: 5646 with args: --syntax-highlight-service=5646:/var/folders/17/pdb9_z8x1wx9xrpmb514tmx00000gp/T/exthost-8097bf-sock2
[INFO]  [ 0.153s] Oni2.Syntax.Client : Notifying theme changed.
[INFO]  [ 0.153s] Oni2.Syntax.Client : Notifying treeSitter changed: false
2020-11-01 17:07:07.251 Oni2_editor[5646:8933329] Setting Updater automatic checks: 1
2020-11-01 17:07:07.251 Oni2_editor[5646:8933329] Setting Updater feed URL: https://v2.onivim.io/appcast?channel=master&licenseKey=&platform=macos
2020-11-01 17:07:07.251 Oni2_editor[5646:8933329] Setting Updater feed URL: https://v2.onivim.io/appcast?channel=master&licenseKey=&platform=macos
[INFO]  [ 0.291s] Oni2.Syntax.Client : Connected to server
[INFO]  [ 0.291s] Oni2.Syntax.Client : Initialized
[INFO]  [ 0.291s] Oni2.Service_Syntax : onConnected
[INFO]  [ 0.291s] Oni2.Service_Syntax : Starting buffer subscription for: 2
[INFO]  [ 0.350s] Oni2.Syntax.Client : Notifying theme changed.
[INFO]  [ 6.367s] Oni2.Service_Syntax : Starting buffer subscription for: 3
[INFO]  [ 6.368s] Oni2.Service_Syntax : Stopping buffer subscription for: 2
[INFO]  [ 6.666s] Oni2.Syntax.Client : Received token update
[INFO]  [ 6.666s] Oni2.Syntax.Client : Received token update
...

Reading through the textmate debug helper issue it looks promising that this is just a difference in scope matching or scope priority, etc. I will have to figure out how to build my own Onivim 2 first to debug this.