Error highlighting `source.regexp`

[appleton]

I'm seeing an error with the source.regexp language, it happens with any input value except for an empty string and seems to originate from inside oniguruma. I'm not sure how to go about debugging this but thought a bug report might be useful.

import { all, createStarryNight } from "@wooorm/starry-night";

createStarryNight(all).then((instance) => instance.highlight("hi", "source.regexp"));

Uncaught Error: invalid code point value
    at @wooorm_starry-night.js?v=d538df45:95:21
    at new u (@wooorm_starry-night.js?v=d538df45:96:14)
    at t2.createOnigScanner (@wooorm_starry-night.js?v=d538df45:257:18)
    at m.createOnigScanner (@wooorm_starry-night.js?v=d538df45:885:36)
    at new g (@wooorm_starry-night.js?v=d538df45:2499:69)
    at m._resolveAnchors (@wooorm_starry-night.js?v=d538df45:2493:22)
    at m.compileAG (@wooorm_starry-night.js?v=d538df45:2489:228)
    at u.compileAG (@wooorm_starry-night.js?v=d538df45:2257:58)
    at l (@wooorm_starry-night.js?v=d538df45:1468:128)
    at @wooorm_starry-night.js?v=d538df45:1397:85

[wooorm]

hi! weird; looking into it.

first off, example doesn’t work, instance and highlighter are mixed up

[wooorm]

Second, it seems to (not) occur with source.sy:

import sy from '@wooorm/starry-night/source.sy'
import regex from '@wooorm/starry-night/source.regexp'
import {createStarryNight} from '@wooorm/starry-night'

const highlighter = await createStarryNight([sy, regex])

console.log(highlighter.missingScopes())

const result = highlighter.highlight('hi', 'source.regexp')

console.log(result)

[wooorm]

I can make the crash appear/disappear by changing this line: https://github.com/wooorm/starry-night/blob/746f62ebe93d891acae7526dd983f7bd3354c3be/lang/source.sy.js#L387-L392.

-      begin: '\\A(?:\\xC2\\xAD|\\xAD){2}',
+      begin: '\\A(?:\\xC2\\xAD){2}',

This line stems from: https://github.com/Alhadis/language-regexp/blob/75535265c171ad2dbaba1a5792194894c6b51986/grammars/syon.cson#L456.

@Alhadis could you perhaps advise about this line and its purpose? Due to that line, the example in my previous comment causes a crash in vscode-oniguruma. It appears/disappears by changing it. Perhaps these (control?) characters are not needed in your grammar, or, perhaps I should raise this with vscode-oniguruma?

The crash:

Error: invalid code point value
    at /Users/tilde/Projects/oss/starry-night/node_modules/vscode-oniguruma/release/main.js:1:3320
    at new u (/Users/tilde/Projects/oss/starry-night/node_modules/vscode-oniguruma/release/main.js:1:3369)
    at t.createOnigScanner (/Users/tilde/Projects/oss/starry-night/node_modules/vscode-oniguruma/release/main.js:1:7559)
    at m.createOnigScanner (/Users/tilde/Projects/oss/starry-night/node_modules/vscode-textmate/release/main.js:1:5022)
    at new g (/Users/tilde/Projects/oss/starry-night/node_modules/vscode-textmate/release/main.js:1:49288)
    at m.compile (/Users/tilde/Projects/oss/starry-night/node_modules/vscode-textmate/release/main.js:1:48529)
    at m.compileAG (/Users/tilde/Projects/oss/starry-night/node_modules/vscode-textmate/release/main.js:1:49063)
    at u.compileAG (/Users/tilde/Projects/oss/starry-night/node_modules/vscode-textmate/release/main.js:1:40508)
    at l (/Users/tilde/Projects/oss/starry-night/node_modules/vscode-textmate/release/main.js:1:24135)
    at /Users/tilde/Projects/oss/starry-night/node_modules/vscode-textmate/release/main.js:1:20316

Node.js v22.0.0

[Alhadis]

@wooorm I've applied your fix, thanks. 🤔 Not sure why I attempted to match both the UTF-8 and Latin-1 encodings of U+00AD SOFT HYPHEN; perhaps I was trying to cover all my bases.

[wooorm]

Oh sweet, thanks!

BTW: I wasn‘t saying this was the correct solution for you. You know way more about grammars than me. I just figured out what causes the crash 😅 So I am not sure about the word “fix” haha.

But if you’re comfortable without it, perfect!

I’ll close this then, as an upstream thing, that will resolve itself when GH and then me pulls in the grammar updates!

[appleton]

Awesome, thank you so much @wooorm and @Alhadis!