Use of Unicode escapes confuses `generateRecipes`

[pragdave]

The following source

JQ {
  lineTerminator
    = "\n" | "\r" | "\u2028" | "\u2029"
}

Generates a recipe file that compiles fine under node, but fails to compile with ts-node. If I remove the two Unicode escapes, it compiles fine under both.

~/P/jq|main ⚡☡
14∙37∙48≻ npx ohm  generateRecipes -t tmp/jq.ohm
tmp/jq.ohm-recipe.js
tmp/jq.ohm-recipe.d.ts

~/P/jq|main ⚡☡
14∙38∙11≻ ts-node tmp/jq.ohm-recipe.js

/usr/local/lib/node_modules/ts-node/src/index.ts:245
    return new TSError(diagnosticText, diagnosticCodes)
           ^
TSError: ⨯ Unable to compile TypeScript:
tmp/jq.ohm-recipe.js:1:416 - error TS1002: Unterminated string literal.

1 'use strict';module.exports=require('ohm-js').makeRecipe(["grammar",{"source":"JQ {\n\nlineTerminator\n  = \"\\n\" | \"\\r\" | \"\\u2028\" | \"\\u2029\"\n  \n}"},"JQ",null,"lineTerminator",{"lineTerminator":["define",{"sourceInterval":[6,58]},null,[],["alt",{"sourceInterval":[25,58]},["terminal",{"sourceInterval":[25,29]},"\n"],["terminal",{"sourceInterval":[32,36]},"\r"],["terminal",{"sourceInterval":[39,47]},"

tmp/jq.ohm-recipe.js:2:1 - error TS1005: ',' expected.

2 "],["terminal",{"sourceInterval":[50,58]},"
  ----
tmp/jq.ohm-recipe.js:2:6 - error TS1005: ',' expected.

2 "],["terminal",{"sourceInterval":[50,58]},"
       ----
tmp/jq.ohm-recipe.js:2:14 - error TS1005: ',' expected.

2 "],["terminal",{"sourceInterval":[50,58]},"
              ----
tmp/jq.ohm-recipe.js:2:18 - error TS1005: ',' expected.

2 "],["terminal",{"sourceInterval":[50,58]},"
                   ----
tmp/jq.ohm-recipe.js:2:32 - error TS1005: ',' expected.

2 "],["terminal",{"sourceInterval":[50,58]},"
                                ----
tmp/jq.ohm-recipe.js:3:9 - error TS1002: Unterminated string literal.

3 "]]]}]);

tmp/jq.ohm-recipe.js:3:1 - error TS1005: ',' expected.

3 "]]]}]);
  ----
tmp/jq.ohm-recipe.js:3:9 - error TS1005: ']' expected.

3 "]]]}]);

  tmp/jq.ohm-recipe.js:1:190
    1 'use strict';module.exports=require('ohm-js').makeRecipe(["grammar",{"source":"JQ {\n\nlineTerminator\n  = \"\\n\" | \"\\r\" | \"\\u2028\" | \"\\u2029\"\n  \n}"},"JQ",null,"lineTerminator",{"lineTerminator":["define",{"sourceInterval":[6,58]},null,[],["alt",{"sourceInterval":[25,58]},["terminal",{"sourceInterval":[25,29]},"\n"],["terminal",{"sourceInterval":[32,36]},"\r"],["terminal",{"sourceInterval":[39,47]},"
                                                                                                                                                                                                   ~
    The parser expected to find a '}' to match the '{' token here.

    at createTSError (/usr/local/lib/node_modules/ts-node/src/index.ts:245:12)
    at reportTSError (/usr/local/lib/node_modules/ts-node/src/index.ts:249:19)
    at getOutput (/usr/local/lib/node_modules/ts-node/src/index.ts:362:34)
    at Object.compile (/usr/local/lib/node_modules/ts-node/src/index.ts:395:32)
    at Module.m._compile (/usr/local/lib/node_modules/ts-node/src/index.ts:473:43)
    at Module._extensions..js (node:internal/modules/cjs/loader:1124:10)
    at Object.require.extensions.<computed> [as .js] (/usr/local/lib/node_modules/ts-node/src/index.ts:476:12)
    at Module.load (node:internal/modules/cjs/loader:975:32)
    at Function.Module._load (node:internal/modules/cjs/loader:816:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:79:12)

[pdubroy]

Interesting, thanks for reporting. Seems like it's related to this: https://v8.dev/features/subsume-json.

I am curious why there is a difference between JS and TS here. Can you let me know which version of TypeScript you're using? It seems to work for me under 4.4.3:

[ohm-scratch] cat jq.ohm
JQ {
  lineTerminator
    = "\n" | "\r" | "\u2028" | "\u2029"
}
[ohm-scratch] npx ohm generateRecipes jq.ohm && npx ts-node jq.ohm-recipe.js
jq.ohm-recipe.js
[ohm-scratch] grep typescript package.json
    "typescript": "^4.4.3"

A possible fix would be to modify Terminal.outputRecipe so that it always emits the escaped version of those characters.

[pdubroy]

Hmm, here was my first attempt at a fix: https://github.com/harc/ohm/compare/issue-322?expand=1 but it seems it's a bit more complicated than I thought. Will try to tackle it tomorrow.

Thanks @pragdave for taking the new stuff for a spin!

[pragdave]

I am curious why there is a difference between JS and TS here. Can you let me know which version of TypeScript you're using? It seems to work for me under 4.4.3

{
"name": "my-jq",
"version": "1.0.0",
"main": "cli.js",
"license": "MIT",
"devDependencies": {
***@***.***/cli": "^0.1.0",
***@***.***/jest": "^27.0.1",
***@***.***/node": "^16.9.1",
"fast-glob": "^3.2.7",
"jest": "^27.0.6",
"ts-jest": "^27.0.4",
"ts-node": "^10.2.1",
"typescript": "^4.4.3"
},
"dependencies": {
"ohm-js": "^16.0.0-pre.2"
},
"scripts": {
"test": "jest",
"testw": "jest --watch"
}
}

[pdubroy]

Ok, turns out it was simpler than I thought — I was totally overthinking it, due to the multiple levels of escaping.

Thanks again for the bug report!