string interpolation surprise

[vahp]

backtick string interpolation fails depending on what character appears in front of the $ sign - seems to happen whether using curly-braces or not. The docs don't mention this and javascript doesn't have this behavior.

<div _="init set x to 42 then put `test $x test$x test_$x test_${x} test-$x test.$x` into me"></div>

in the https://hyperscript.org/playground/ this gets:

test 42 test$x test$x test${x} test-42 test.42

[BirdeeHub]

The error is in the Lexer.tokenize function. More specifically, the consumeIdentifier function it calls, which should have different behavior if in a template vs not.

When hyperscript finds a template string (strings surrounded with ` chars) it calls tokenize again on that string with template == true.

When it does so, it seems to parse things like test$x as a single token rather than 3, because it parses them as normal language identifiers, which cannot normally contain interpolations

Im a noob, so, no promises, but armed with the following test and some console.logs I can hopefully prevail.

    it("string interpolation isnt surprising", function () {
        var div = make( '<div _="on click set x to 42 then put `test $x test ${x} test$x test_$x test_${x} test-$x test.$x` into my.innerHTML"></div>');
        div.click();
        div.innerHTML.should.equal("test 42 test 42 test42 test_42 test_42 test-42 test.42");
    });

Ill poke at it when I have free time and get tired of doing other stuff this week because language stuff is fun.

[BirdeeHub]

Actually I may have already got it. Ill refine my fix a bit but, I mean... the test is green now, and I didnt make any other tests fail

(there was 1 that was already failing... something about uri schemes when starting sockets. It was failing before I changed anything though.)

[BirdeeHub]

Oh. That other test is only failing because Im running it straight from opening the file with firefox rather than serving it, so it doesnt know if it needs to start websocket with http or https.

Ran the tests after serving with python3 http.server and it passes. So, all good.

Now I need to try to figure out if this test counts as a parser bug or a tokenizer bug so that I know what file to put the test in and I can push it XD

Edit: I think it counts as tokenizer bug, everything I needed to change was in the Lexer class.