microsoft / TypeScript-TmLanguage

TextMate grammar files for TypeScript for VS Code, Sublime Text, and Atom.
MIT License
398 stars 125 forks source link

[JavaScript] Tokenization breaks when new line starts with less-than operator #904

Open joshlongerbeam opened 2 years ago

joshlongerbeam commented 2 years ago

Does this issue occur when all extensions are disabled?: Yes

VS Code version: Code 1.66.0 (e18005f0f1b33c29e81d732535d8c0e47cafb0b5, 2022-03-30T05:50:41.156Z) OS version: Linux x64 4.15.0-175-generic Restricted Mode: No

System Info |Item|Value| |---|---| |CPUs|Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz (8 x 3541)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: disabled_off
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
oop_rasterization: disabled_off
opengl: enabled_on
rasterization: disabled_software
raw_draw: disabled_off_ok
skia_renderer: enabled_on
video_decode: disabled_software
video_encode: disabled_software
vulkan: disabled_off
webgl: enabled
webgl2: enabled| |Load (avg)|0, 0, 1| |Memory (System)|31.26GB (23.84GB free)| |Process Argv|--unity-launch --crash-reporter-id c990702e-9bfc-48d9-b2b9-78bab7fdb823| |Screen Reader|no| |VM|0%| |DESKTOP_SESSION|ubuntu| |XDG_CURRENT_DESKTOP|Unity| |XDG_SESSION_DESKTOP|ubuntu| |XDG_SESSION_TYPE|x11|
Extensions: none # Simple example 1. In a new editor, set the language mode to plain `JavaScript` 2. Type the following ```js let obj = {} let bad = 1 < obj.two let unexpected = 1 /> let nowFixed = 1 ``` 3. The `<` on line 4 is treated as a JSX tag opener (even when the self-closing tag on line 7 is removed) Note how GitHub's highlighting in 2. and this screenshot disagree ![Screenshot from 2022-04-01 11-22-56](https://user-images.githubusercontent.com/11166706/161293970-6445f7ae-0a1c-40e2-834a-e07b6a08ae5b.png) 4. Declaring `two` slightly changes highlighting, but notice the `.` in `obj.two` is still green. Using "Developer: Inspect Editor Token and Scopes" shows it's still considered a JSX tag. ![Screenshot from 2022-04-01 11-23-48](https://user-images.githubusercontent.com/11166706/161294747-810cb57e-d986-4694-a4b4-ae8c30e3a185.png) # Extended example To show all the specific conditions required to trigger this issue ![Screenshot from 2022-04-01 11-17-14](https://user-images.githubusercontent.com/11166706/161295522-6f572375-0610-4354-a5fc-909248d45971.png)
Show code snippet for copy-paste ```js let expected, notExpected, obj = {} let expectedFunc = () => { return (true) } obj.twenty = 20 expected = 10 < obj.twenty expected = 10 < 20 expected = 30 > obj.twenty expected = 10 < obj.twenty; expected = (10 < obj.twenty) expected = 10 < obj.twenty notExpected = 10 < obj.twenty // <-- `<` at start of line misinterpreted as JSX // syntax highlighting continues to be wonky let notExpectedFunc = () => { return (true) } /> // until a JSX closing tag is added let expectedFunc = () => { return (true) } ```
# Real-world example In my repo, using `prettier` for formatting (specifically, a `prettier` fork that places operators at start of line instead of end) yields the following output, with the problem starting at **line 77**. Note how placing the cursor on the less-than op highlights the greater-than op on **line 84**, as if lines 77-84 are just one giant JSX opening tag declaration. Also note that this is a `.js` file with no mention nor import of `React`, and the prior examples also had language type explicitly set to JavaScript. ![Screenshot from 2022-04-01 11-43-10](https://user-images.githubusercontent.com/11166706/161297376-fd99bdd9-7a78-43c2-b04f-1ad64c9f4d91.png) ---- My preemptive apologies if this issue is a duplicate (I searched but found none) or if it belongs in a more specific repo.