JSDoc comments syntax highlighting does not match parsed state of `@type` / `@typedef` blocks

[myfonj]

Type: Bug

JS/TS language service is quite permissive for JSDoc comment block structure and does not have problems with line breaks and optional leading asterisks (good), but sadly the JavaScript.tmLanguage definitions are more strict and fall back to regular comment block in any sign of irregularity in the structure of the comment. For example simple line break before closing */ of "line" JSDoc comment makes it appear like a normal insignificant comment:

/** @type {string} */
var a = "OK, highlighted as JSDoc.";
/** @type {string}
*/
var b = "ERROR, highlighted like a regular comment.";

See this this abomination of a JSDoc @typedef that is still correctly processed (despite inconsistent usage of leading asterisks and line breaks) yet not highlighted as such. Notice errors on the bottom and the tooltip that shows that all JSDoc blocks are processed:

// @ts-check

/** @typedef
    * {
    typeof
    * ABC [
         number
    * ]
    }
           *       Ch -
   this
 - still
 * works
*/
// ↑ ERROR, highlighted as a comment, but processed

const ABC = /** @type {const} */(["a", "b", "c"]);

/** @type {Ch}
*/
// ↑ ERROR, highlighted as a comment, but processed
var x = 'd';

/** @type {Ch} */
// ↑ OK, highlighted
var y = 'd';

---

On a personal note, I still do not fully understand why is highlighting done separately from language services. It feels that two parts are doing same work, parsing and tokenizing, but one does it for real and the other just fakes it. Is the Text TextMate grammar processing significantly faster or what is the reason that allows issues such as this one to happen?

---

VS Code version: Code - Insiders 1.88.0-insider (15a0f3c48f9315669c97f101a5d954464285e6ac, 2024-02-29T05:47:41.883Z) OS version: Windows_NT x64 10.0.19045 Modes:

System Info

|Item|Value| |---|---| |CPUs|Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz (8 x 2112)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: enabled_on
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_graphite: disabled_off
video_decode: enabled
video_encode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled
webgpu: enabled| |Load (avg)|undefined| |Memory (System)|15.76GB (2.26GB free)| |Process Argv|| |Screen Reader|no| |VM|0%|

Extensions (1)

Extension|Author (truncated)|Version ---|---|--- errorlens|use|3.16.0

A/B Experiments

``` vsliv368cf:30146710 vspor879:30202332 vspor708:30202333 vspor363:30204092 vscod805:30301674 vsaa593cf:30376535 py29gd2263:30784851 c4g48928:30535728 2i9eh265:30646982 962ge761:30841072 pythongtdpath:30726887 welcomedialog:30812478 pythonidxpt:30768918 pythonnoceb:30776497 asynctok:30898717 dsvsc013:30777762 dsvsc014:30777825 dsvsc015:30821418 pythontestfixt:30866404 pythonregdiag2:30926734 pyreplss1:30879911 pythonmypyd1:30859725 pythoncet0:30859736 pythontbext0:30879054 accentitlementst:30870582 dsvsc016:30879898 dsvsc017:30880771 dsvsc018:30880772 bf62j303:30968145 pydisww2:30959801 cppperfcontrol:30963551 d34g3935:30961436 edj9j583:30969294 fegfb526:30952798 e3gdj431:30946825 ccp1r3:30958150 pythonait:30973460 ```

[RedCMD]

TextMate runs on the render thread which must stay fast any lag will cause cursor freezing etc

for performance reasons, TextMate only parses one line at a time making it rather difficult to do a context aware multiline tokenizing

however it is possible to make the grammar more lenient allowing it to follow the LSP more closely

[myfonj]

however it is possible to make the grammar more lenient

I see. So some digging revealed among swarm of other JSDoc related bugs microsoft/TypeScript-TmLanguage#467 (closed at this moment), what seems like the most fitting place to continue.

---

OT/cont

TextMate runs on the render thread which must stay fast any lag will cause cursor freezing etc

Understood. But still sounds a bit scary: IIUC the "cursor freezing" caveat, the syntax highlighter in the render thread is sorta-kinda synchronous and blocking process, right? What basically means it has even higher priority than keyboard input? If so, it sounds like an odd design choice for software what's primary purpose is to … edit text files by typing or similar user input. (And yes, I have experienced keyboard lag in VSC, but always thought it is result of certain extension/file combination (humongous markdown with several MD-related extensions, for example) gobbing CPU so much that OS itself runs into problems with HW signals/interrupts, or something, not that VSC waits for information what colour should the last character have.)