search

google / closure-compiler

A JavaScript checker and optimizer.
https://developers.google.com/closure/compiler/
Apache License 2.0
7.42k stars 1.15k forks source link

java.lang.IllegalStateException: 'yield' expression is not within a generator function. #3897

Open L3P3 opened 2 years ago

L3P3 commented 2 years ago

I tried to transpile down the vscode frontend from es2020 to es2015 in order to use it on a machine that does not support newer browsers. Rare use case but should work in a bug free world. :) For that, I grabbed closure-compiler-v20211107.jar and used it like this:

/usr/lib/code-server/vendor/modules/code-oss-dev/out/vs/workbench# closure-compiler --js workbench.web.api.js --js_output_file workbench.web.api.js --language_in ECMASCRIPT_2020 --language_out ECMASCRIPT_2015
java.lang.IllegalStateException: 'yield' expression is not within a generator function. Reference node:
YIELD 1461:4095  [length: 32] [source_file: workbench.web.api.js]
    CALL 1461:4095  [length: 26] [source_file: workbench.web.api.js]
        GETPROP createTerminal 1461:4095  [length: 14] [source_file: workbench.web.api.js]
            NAME L 1461:4095  [length: 4] [source_file: workbench.web.api.js]
        NAME E 1461:4095  [length: 1] [source_file: workbench.web.api.js]
        NAME M 1461:4095  [length: 1] [source_file: workbench.web.api.js]
        NAME J 1461:4095  [length: 1] [source_file: workbench.web.api.js]

 Parent node:
NAME la 1461:4095  [length: 40] [source_file: workbench.web.api.js] [constant_var_flags: 2]
    YIELD 1461:4095  [length: 32] [source_file: workbench.web.api.js]
        CALL 1461:4095  [length: 26] [source_file: workbench.web.api.js]
            GETPROP createTerminal 1461:4095  [length: 14] [source_file: workbench.web.api.js]
                NAME L 1461:4095  [length: 4] [source_file: workbench.web.api.js]
            NAME E 1461:4095  [length: 1] [source_file: workbench.web.api.js]
            NAME M 1461:4095  [length: 1] [source_file: workbench.web.api.js]
            NAME J 1461:4095  [length: 1] [source_file: workbench.web.api.js]

        at com.google.javascript.jscomp.AstValidator$1.handleViolation(AstValidator.java:86)
        at com.google.javascript.jscomp.AstValidator.violation(AstValidator.java:1995)
        at com.google.javascript.jscomp.AstValidator.validateYieldWithinGeneratorFunction(AstValidator.java:613)
        at com.google.javascript.jscomp.AstValidator.validateYield(AstValidator.java:607)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:440)
        at com.google.javascript.jscomp.AstValidator.validateNameDeclarationChild(AstValidator.java:1262)
        at com.google.javascript.jscomp.AstValidator.validateNameDeclarationHelper(AstValidator.java:1227)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:206)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:153)
        at com.google.javascript.jscomp.AstValidator.validateBlock(AstValidator.java:941)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpressionHelper(AstValidator.java:1052)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpression(AstValidator.java:1034)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:424)
        at com.google.javascript.jscomp.AstValidator.validateCall(AstValidator.java:1119)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:412)
        at com.google.javascript.jscomp.AstValidator.validateBinaryOp(AstValidator.java:1905)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:380)
        at com.google.javascript.jscomp.AstValidator.validateIf(AstValidator.java:1450)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:201)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:153)
        at com.google.javascript.jscomp.AstValidator.validateBlock(AstValidator.java:941)
        at com.google.javascript.jscomp.AstValidator.validateIf(AstValidator.java:1451)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:201)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:153)
        at com.google.javascript.jscomp.AstValidator.validateBlock(AstValidator.java:941)
        at com.google.javascript.jscomp.AstValidator.validateFunctionBody(AstValidator.java:1082)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpressionHelper(AstValidator.java:1058)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpression(AstValidator.java:1034)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:424)
        at com.google.javascript.jscomp.AstValidator.validatePseudoExpression(AstValidator.java:480)
        at com.google.javascript.jscomp.AstValidator.validateCall(AstValidator.java:1122)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:412)
        at com.google.javascript.jscomp.AstValidator.validateReturn(AstValidator.java:1467)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:212)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:153)
        at com.google.javascript.jscomp.AstValidator.validateBlock(AstValidator.java:941)
        at com.google.javascript.jscomp.AstValidator.validateFunctionBody(AstValidator.java:1082)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpressionHelper(AstValidator.java:1058)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpression(AstValidator.java:1034)
        at com.google.javascript.jscomp.AstValidator.validateMemberFunction(AstValidator.java:918)
        at com.google.javascript.jscomp.AstValidator.validateClassMember(AstValidator.java:877)
        at com.google.javascript.jscomp.AstValidator.validateClassMembers(AstValidator.java:867)
        at com.google.javascript.jscomp.AstValidator.validateClassHelper(AstValidator.java:861)
        at com.google.javascript.jscomp.AstValidator.validateClassDeclaration(AstValidator.java:833)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:231)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:153)
        at com.google.javascript.jscomp.AstValidator.validateBlock(AstValidator.java:941)
        at com.google.javascript.jscomp.AstValidator.validateFunctionBody(AstValidator.java:1082)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpressionHelper(AstValidator.java:1058)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpression(AstValidator.java:1034)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:424)
        at com.google.javascript.jscomp.AstValidator.validatePseudoExpression(AstValidator.java:480)
        at com.google.javascript.jscomp.AstValidator.validateCall(AstValidator.java:1122)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:412)
        at com.google.javascript.jscomp.AstValidator.validateExprStmt(AstValidator.java:1460)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:209)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:153)
        at com.google.javascript.jscomp.AstValidator.validateBlock(AstValidator.java:941)
        at com.google.javascript.jscomp.AstValidator.validateFunctionBody(AstValidator.java:1082)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpressionHelper(AstValidator.java:1058)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpression(AstValidator.java:1034)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:424)
        at com.google.javascript.jscomp.AstValidator.validatePropertyReferenceTarget(AstValidator.java:1662)
        at com.google.javascript.jscomp.AstValidator.validateGetProp(AstValidator.java:1644)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:392)
        at com.google.javascript.jscomp.AstValidator.validateCall(AstValidator.java:1119)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:412)
        at com.google.javascript.jscomp.AstValidator.validateExprStmt(AstValidator.java:1460)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:209)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:153)
        at com.google.javascript.jscomp.AstValidator.validateStatements(AstValidator.java:147)
        at com.google.javascript.jscomp.AstValidator.validateScript(AstValidator.java:136)
        at com.google.javascript.jscomp.AstValidator.validateCodeRoot(AstValidator.java:123)
        at com.google.javascript.jscomp.AstValidator.process(AstValidator.java:109)
        at com.google.javascript.jscomp.PhaseOptimizer$NamedPass.process(PhaseOptimizer.java:317)
        at com.google.javascript.jscomp.PhaseOptimizer.process(PhaseOptimizer.java:232)
        at com.google.javascript.jscomp.Compiler.performTranspilationAndOptimizations(Compiler.java:2573)
        at com.google.javascript.jscomp.Compiler.lambda$stage2Passes$5(Compiler.java:939)
        at com.google.javascript.jscomp.CompilerExecutor.lambda$runInCompilerThread$0(CompilerExecutor.java:101)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

From this output, I cannot tell which part of the file causes this, I am sorry. But you can reproduce it if you install code-server. This may help: https://github.com/cdr/code-server/tree/v3.12.0/vendor

L3P3 commented 2 years ago

I tested language_out ECMASCRIPT_2017:

java.lang.IllegalStateException: 'await' expression is not within an async function. Reference node:
AWAIT 1461:4095  [length: 32] [source_file: workbench.web.api.js.old]
    CALL 1461:4095  [length: 26] [source_file: workbench.web.api.js.old]
        GETPROP createTerminal 1461:4095  [length: 14] [source_file: workbench.web.api.js.old]
            THIS 1461:4095  [length: 4] [source_file: workbench.web.api.js.old]
        NAME E 1461:4095  [length: 1] [source_file: workbench.web.api.js.old]
        NAME M 1461:4095  [length: 1] [source_file: workbench.web.api.js.old]
        NAME K 1461:4095  [length: 1] [source_file: workbench.web.api.js.old]

 Parent node:
NAME ca 1461:4095  [length: 40] [source_file: workbench.web.api.js.old] [constant_var_flags: 2]
    AWAIT 1461:4095  [length: 32] [source_file: workbench.web.api.js.old]
        CALL 1461:4095  [length: 26] [source_file: workbench.web.api.js.old]
            GETPROP createTerminal 1461:4095  [length: 14] [source_file: workbench.web.api.js.old]
                THIS 1461:4095  [length: 4] [source_file: workbench.web.api.js.old]
            NAME E 1461:4095  [length: 1] [source_file: workbench.web.api.js.old]
            NAME M 1461:4095  [length: 1] [source_file: workbench.web.api.js.old]
            NAME K 1461:4095  [length: 1] [source_file: workbench.web.api.js.old]

        at com.google.javascript.jscomp.AstValidator$1.handleViolation(AstValidator.java:86)
        at com.google.javascript.jscomp.AstValidator.violation(AstValidator.java:1995)
        at com.google.javascript.jscomp.AstValidator.validateAwaitWithinAsyncFunction(AstValidator.java:630)
        at com.google.javascript.jscomp.AstValidator.validateAwait(AstValidator.java:624)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:444)
        at com.google.javascript.jscomp.AstValidator.validateNameDeclarationChild(AstValidator.java:1262)
        at com.google.javascript.jscomp.AstValidator.validateNameDeclarationHelper(AstValidator.java:1227)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:206)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:153)
        at com.google.javascript.jscomp.AstValidator.validateBlock(AstValidator.java:941)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpressionHelper(AstValidator.java:1052)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpression(AstValidator.java:1034)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:424)
        at com.google.javascript.jscomp.AstValidator.validateCall(AstValidator.java:1119)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:412)
        at com.google.javascript.jscomp.AstValidator.validateBinaryOp(AstValidator.java:1905)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:380)
        at com.google.javascript.jscomp.AstValidator.validateIf(AstValidator.java:1450)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:201)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:153)
        at com.google.javascript.jscomp.AstValidator.validateBlock(AstValidator.java:941)
        at com.google.javascript.jscomp.AstValidator.validateIf(AstValidator.java:1451)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:201)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:153)
        at com.google.javascript.jscomp.AstValidator.validateBlock(AstValidator.java:941)
        at com.google.javascript.jscomp.AstValidator.validateFunctionBody(AstValidator.java:1082)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpressionHelper(AstValidator.java:1058)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpression(AstValidator.java:1034)
        at com.google.javascript.jscomp.AstValidator.validateMemberFunction(AstValidator.java:918)
        at com.google.javascript.jscomp.AstValidator.validateClassMember(AstValidator.java:877)
        at com.google.javascript.jscomp.AstValidator.validateClassMembers(AstValidator.java:867)
        at com.google.javascript.jscomp.AstValidator.validateClassHelper(AstValidator.java:861)
        at com.google.javascript.jscomp.AstValidator.validateClassDeclaration(AstValidator.java:833)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:231)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:153)
        at com.google.javascript.jscomp.AstValidator.validateBlock(AstValidator.java:941)
        at com.google.javascript.jscomp.AstValidator.validateFunctionBody(AstValidator.java:1082)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpressionHelper(AstValidator.java:1058)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpression(AstValidator.java:1034)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:424)
        at com.google.javascript.jscomp.AstValidator.validatePseudoExpression(AstValidator.java:480)
        at com.google.javascript.jscomp.AstValidator.validateCall(AstValidator.java:1122)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:412)
        at com.google.javascript.jscomp.AstValidator.validateExprStmt(AstValidator.java:1460)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:209)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:153)
        at com.google.javascript.jscomp.AstValidator.validateBlock(AstValidator.java:941)
        at com.google.javascript.jscomp.AstValidator.validateFunctionBody(AstValidator.java:1082)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpressionHelper(AstValidator.java:1058)
        at com.google.javascript.jscomp.AstValidator.validateFunctionExpression(AstValidator.java:1034)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:424)
        at com.google.javascript.jscomp.AstValidator.validatePropertyReferenceTarget(AstValidator.java:1662)
        at com.google.javascript.jscomp.AstValidator.validateGetProp(AstValidator.java:1644)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:392)
        at com.google.javascript.jscomp.AstValidator.validateCall(AstValidator.java:1119)
        at com.google.javascript.jscomp.AstValidator.validateExpression(AstValidator.java:412)
        at com.google.javascript.jscomp.AstValidator.validateExprStmt(AstValidator.java:1460)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:209)
        at com.google.javascript.jscomp.AstValidator.validateStatement(AstValidator.java:153)
        at com.google.javascript.jscomp.AstValidator.validateStatements(AstValidator.java:147)
        at com.google.javascript.jscomp.AstValidator.validateScript(AstValidator.java:136)
        at com.google.javascript.jscomp.AstValidator.validateCodeRoot(AstValidator.java:123)
        at com.google.javascript.jscomp.AstValidator.process(AstValidator.java:109)
        at com.google.javascript.jscomp.PhaseOptimizer$NamedPass.process(PhaseOptimizer.java:317)
        at com.google.javascript.jscomp.PhaseOptimizer.process(PhaseOptimizer.java:232)
        at com.google.javascript.jscomp.Compiler.performTranspilationAndOptimizations(Compiler.java:2573)
        at com.google.javascript.jscomp.Compiler.lambda$stage2Passes$5(Compiler.java:939)
        at com.google.javascript.jscomp.CompilerExecutor.lambda$runInCompilerThread$0(CompilerExecutor.java:101)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
frigus02 commented 2 years ago

Thanks for reporting. This is quite hard for us to follow up on because the input file is large and minified. Can you try to provide a smaller example? The AST contains line/column numbers (e.g. YIELD 1461:4095). It might help if you pretty print the file before. That way it should be easier to find which part of the code causes the exception.

L3P3 commented 2 years ago 
java.lang.IllegalStateException: 'await' expression is not within an async function. Reference node:
AWAIT 148056:36  [length: 34] [source_file: workbench.web.api.beauty.js]
    CALL 148056:42  [length: 28] [source_file: workbench.web.api.beauty.js]
        GETPROP createTerminal 148056:47  [length: 14] [source_file: workbench.web.api.beauty.js]
            THIS 148056:42  [length: 4] [source_file: workbench.web.api.beauty.js]
        NAME E 148056:62  [length: 1] [source_file: workbench.web.api.beauty.js]
        NAME M 148056:65  [length: 1] [source_file: workbench.web.api.beauty.js]
        NAME K 148056:68  [length: 1] [source_file: workbench.web.api.beauty.js]
async executeInTerminal(A, P, T, N) {
  let O, F, k, x;
  if (A.configurationProperties.isBackground) {
    const W = await this.resolveMatchers(T, A.configurationProperties.problemMatchers);
    let z = new s.WatchingProblemCollector(W, this.markerService, this.modelService, this.fileService);
    W.length > 0 && !z.isWatching() && (this.appendOutput(y.localize(3, null, A._label)), this.showOutput());
    const H = new p.DisposableStore;
    let U = 0;
    const $ = A.getMapKey();
    H.add(z.onDidStateChange(X => {
      if (X.kind === "backgroundProcessingBegins") U++, this.busyTasks[$] = A, this.fireTaskEvent(n.TaskEvent.create("active", A));
      else if (X.kind === "backgroundProcessingEnds" && (U--, this.busyTasks[$] && delete this.busyTasks[$], this.fireTaskEvent(n.TaskEvent.create("inactive", A)), U === 0 && z.numberOfMatches > 0 && z.maxMarkerSeverity && z.maxMarkerSeverity >= m.MarkerSeverity.Error)) {
        let K = A.command.presentation.reveal;
        A.command.presentation.revealProblems === n.RevealProblemKind.OnProblem ? this.viewsService.openView(c.default.MARKERS_VIEW_ID, !0) : K === n.RevealKind.Silent && (this.terminalService.setActiveInstance(O), this.terminalGroupService.showPanel(!1))
      }
    })), z.aboutToStart();
    let Q;
    if ([O, F, k] = await this.createTerminal(A, T, N), k) return Promise.reject(new Error(k.message));
                    ^^^^^
    if (!O) return Promise.reject(new Error(`Failed to create terminal for task ${A._label}`));
    this.terminalStatusManager.addTerminal(A, O, z);

Looks like await inside if() is the cause here. I thought this has been resolved already. :confused: Also note that I do not want to transpile to pre-async ecmascript version but just es2017. All of this fun stuff should not be changed whatsoever...

frigus02 commented 2 years ago

Thank you. We created an issue to track the work internally.

concavelenz commented 2 years ago

We have a minimal repro, the destructuring is necessary.

async function add(a) {
  let x;
  if ([x] = await a) {

  }
}
console.log(add);
brad4d commented 2 years ago

When the value of a destructuring assignment expression is not a stand-alone statement, Es6RewriteDestructuring wraps it in an arrow function to provide a scope for the variables that must be declared and sub-assignment statements that must be generated. Es6RewriteDestructuring runs before Es6RewriteGenerators, but it fails to notice yield expressions and ends up wrapping them inside that arrow function.

Maybe I could have Es6RewriteDestructuring respond to the presence of yield expressions by making the arrow function a generator and calling it with yield *? I'm going to experiment a bit to see if that works as I hope it does.

brad4d commented 2 years ago

OK, that probably would work, except generator arrow functions aren't allowed by the language.

We're using arrow functions so that access to this is consistent.

I've just realized that wrapping in the arrow function will also break any references to arguments, so that also needs fixing. If we have to fix that, we might as well use a normal function and fix both arguments and this manually.

brad4d commented 2 years ago

nevermind about arguments. I apparently messed up my attempt to test whether arrow functions get their own copy of arguments. They don't.

brad4d commented 2 years ago

I have a fix for this in internal review.

L3P3 commented 2 years ago

I see the point of function wrappers for older targets but for newer, why can't we use normal blocks? Because blocks cannot be placed inside an expression?

brad4d commented 2 years ago

Since blocks cannot be placed inside an expression, it's necessary to decompose expressions into multiple statements and create temporary variables, etc. in order to preserve the order of operations from the original code. The ExpressionDecomposer can do this except for a few edge cases, but apparently we previously opted to use the arrow-function approach instead. Likely this was partly because it seemed a more elegant solution at the time and partly because handling destructuring of for-loops that contain multiple variable declarations (e.g. for (let x = 3, [y, z] = obj; ... gets tricky.

Now it doesn't seem so elegant. I'm thinking the right approach is to first wrap such for-loops in a block and extract the declaration part into the block. Then the compiler can successfully break it into multiple declaration statements, which is a prerequisite for Es6RewriteDestructuring to use ExpressionDecomposer the way we want. Once that's done, I think we can use ExpressionDecomposer in Es6RewriteDestructuring instead of creating IIFEs to transpile the destructuring.

L3P3 commented 2 years ago

Copy that. Function wrappers should be last resort in my opinion.