tree-sitter / tree-sitter-javascript

Javascript grammar for tree-sitter
MIT License
314 stars 108 forks source link

Handling of `await` and `yield` not context-dependent #303

Open Maxdamantus opened 2 months ago

Maxdamantus commented 2 months ago

The following piece of code is valid but it is parsed incorrectly:

async function foo() {
    await /REGEX/g;
    yield /DIVISION/g;
}

function* bar() {
    await /DIVISION/g;
    yield /REGEX/g;
}

async function* baz() {
    await /REGEX/g;
    yield /REGEX/g;
}

function div3(await, yield, g) {
    return await/yield/g;
}

The output of tree-sitter parse is the following:

(program [0, 0] - [18, 0]
  (function_declaration [0, 0] - [3, 1]
    name: (identifier [0, 15] - [0, 18])
    parameters: (formal_parameters [0, 18] - [0, 20])
    body: (statement_block [0, 21] - [3, 1]
      (expression_statement [1, 4] - [1, 19]
        (await_expression [1, 4] - [1, 18]
          (regex [1, 10] - [1, 18]
            pattern: (regex_pattern [1, 11] - [1, 16])
            flags: (regex_flags [1, 17] - [1, 18]))))
      (expression_statement [2, 4] - [2, 22]
        (yield_expression [2, 4] - [2, 21]
          (regex [2, 10] - [2, 21]
            pattern: (regex_pattern [2, 11] - [2, 19])
            flags: (regex_flags [2, 20] - [2, 21]))))))
  (generator_function_declaration [5, 0] - [8, 1]
    name: (identifier [5, 10] - [5, 13])
    parameters: (formal_parameters [5, 13] - [5, 15])
    body: (statement_block [5, 16] - [8, 1]
      (expression_statement [6, 4] - [6, 22]
        (await_expression [6, 4] - [6, 21]
          (regex [6, 10] - [6, 21]
            pattern: (regex_pattern [6, 11] - [6, 19])
            flags: (regex_flags [6, 20] - [6, 21]))))
      (expression_statement [7, 4] - [7, 19]
        (yield_expression [7, 4] - [7, 18]
          (regex [7, 10] - [7, 18]
            pattern: (regex_pattern [7, 11] - [7, 16])
            flags: (regex_flags [7, 17] - [7, 18]))))))
  (generator_function_declaration [10, 0] - [13, 1]
    name: (identifier [10, 16] - [10, 19])
    parameters: (formal_parameters [10, 19] - [10, 21])
    body: (statement_block [10, 22] - [13, 1]
      (expression_statement [11, 4] - [11, 19]
        (await_expression [11, 4] - [11, 18]
          (regex [11, 10] - [11, 18]
            pattern: (regex_pattern [11, 11] - [11, 16])
            flags: (regex_flags [11, 17] - [11, 18]))))
      (expression_statement [12, 4] - [12, 19]
        (yield_expression [12, 4] - [12, 18]
          (regex [12, 10] - [12, 18]
            pattern: (regex_pattern [12, 11] - [12, 16])
            flags: (regex_flags [12, 17] - [12, 18]))))))
  (function_declaration [15, 0] - [17, 1]
    name: (identifier [15, 9] - [15, 13])
    parameters: (formal_parameters [15, 13] - [15, 30]
      (ERROR [15, 14] - [15, 29]
        (await_expression [15, 14] - [15, 29]
          (ERROR [15, 19] - [15, 20])
          (yield_expression [15, 21] - [15, 29]
            (ERROR [15, 26] - [15, 27])
            (identifier [15, 28] - [15, 29])))))
    body: (statement_block [15, 31] - [17, 1]
      (return_statement [16, 4] - [16, 25]
        (await_expression [16, 11] - [16, 24]
          (regex [16, 16] - [16, 24]
            pattern: (regex_pattern [16, 17] - [16, 22])
            flags: (regex_flags [16, 23] - [16, 24])))))))
test.js 0 ms    (ERROR [15, 14] - [15, 29])

foo and bar are incorrectly parsed such that occurrences of /DIVISION/g are interpreted as regex literals instead of as sequences of division operators. Handling this correctly depends on special treatment of await within async functions (since it's not a reserved word in other contexts), and special treatment of yield within generator functions (since it's otherwise only a reserved word in "use strict" contexts).

div3 incorrectly fails to parse. The body should similarly contain two division operations.