search

github / codeql-cli-binaries

Binaries for the CodeQL CLI
Other
750 stars 112 forks source link

typescript-parser-wrapper fails for tsconfig.json path check on Windows #163

Open jason-ha opened 1 year ago

jason-ha commented 1 year ago

When javascript is scanned during finalize typescript-parser-wrapper can throw with tsconfig.json path mismatch. This particular situation was found when more than javascript was detected and was on a Windows agent.

D:\a\_work\_temp\codeql3000\github\codeql\codeql.exe database trace-command --index-traceless-dbs --db-cluster D:\a\_work\_temp\codeql3000\d 
Running 2 commands for 3 databases: 
- D:\a\_work\_temp\codeql3000\d\cpp 
- D:\a\_work\_temp\codeql3000\d\javascript 
- D:\a\_work\_temp\codeql3000\d\python 
Running command in D:\a\_work\1\s: [D:\a\_work\_temp\codeql3000\github\codeql\javascript\tools\autobuild.cmd] 
[2023-04-25 22:24:54] [build-stdout] Single-threaded extraction. 
[2023-04-25 22:24:54] [build-stdout] packages\domain\interop\package.json: Main file set to packages\domain\interop\src\index.ts 
[2023-04-25 22:24:54] [build-stdout] packages\domain\shf_validator\package.json: Main file not found 
[2023-04-25 22:24:54] [build-stdout] Found Node.js at: node 
[2023-04-25 22:24:54] [build-stdout] Found Node.js version: v16.15.1 
[2023-04-25 22:24:54] [build-stdout] Opening project D:\a\_work\1\s\build\tsconfig.json 
[2023-04-25 22:24:54] [build-stdout] Memory for TypeScript process: 2000 MB, and 400 MB reserve 
[2023-04-25 22:24:54] [build-stderr] D:\a\_work\_temp\codeql3000\github\codeql\javascript\tools\typescript-parser-wrapper\node_modules\typescript\lib\typescript.js:2572 
[2023-04-25 22:24:54] [build-stderr] throw e; 
[2023-04-25 22:24:54] [build-stderr] ^ 
[2023-04-25 22:24:54] [build-stderr] Error: Debug Failure. Expected D:/a/_work/1/s/build/tsconfig.json === D:\a\_work\1\s\build\tsconfig.json. 
[2023-04-25 22:24:54] [build-stderr] at attachFileToDiagnostic (D:\a\_work\_temp\codeql3000\github\codeql\javascript\tools\typescript-parser-wrapper\node_modules\typescript\lib\typescript.js:20186:18) 
[2023-04-25 22:24:54] [build-stderr] at Object.attachFileToDiagnostics (D:\a\_work\_temp\codeql3000\github\codeql\javascript\tools\typescript-parser-wrapper\node_modules\typescript\lib\typescript.js:20218:42) 
[2023-04-25 22:24:54] [build-stderr] at Object.parseJsonText (D:\a\_work\_temp\codeql3000\github\codeql\javascript\tools\typescript-parser-wrapper\node_modules\typescript\lib\typescript.js:32364:46) 
[2023-04-25 22:24:54] [build-stderr] at Object.parseJsonText (D:\a\_work\_temp\codeql3000\github\codeql\javascript\tools\typescript-parser-wrapper\node_modules\typescript\lib\typescript.js:32092:23) 
[2023-04-25 22:24:54] [build-stderr] at parseConfigFileTextToJson (D:\a\_work\_temp\codeql3000\github\codeql\javascript\tools\typescript-parser-wrapper\node_modules\typescript\lib\typescript.js:41734:33) 
[2023-04-25 22:24:54] [build-stderr] at Object.readConfigFile (D:\a\_work\_temp\codeql3000\github\codeql\javascript\tools\typescript-parser-wrapper\node_modules\typescript\lib\typescript.js:41725:48) 
[2023-04-25 22:24:54] [build-stderr] at loadTsConfig (D:\a\_work\_temp\codeql3000\github\codeql\javascript\tools\typescript-parser-wrapper\main.js:327:23) 
[2023-04-25 22:24:54] [build-stderr] at handleOpenProjectCommand (D:\a\_work\_temp\codeql3000\github\codeql\javascript\tools\typescript-parser-wrapper\main.js:393:14) 
[2023-04-25 22:24:54] [build-stderr] at Interface.<anonymous> (D:\a\_work\_temp\codeql3000\github\codeql\javascript\tools\typescript-parser-wrapper\main.js:620:17) 
[2023-04-25 22:24:54] [build-stderr] at Interface.emit (node:events:527:28) 
[2023-04-25 22:24:55] [build-stderr] at Interface._onLine (node:readline:487:10) 
[2023-04-25 22:24:55] [build-stderr] at Interface._normalWrite (node:readline:661:12) 
[2023-04-25 22:24:55] [build-stderr] at Socket.ondata (node:readline:269:10) 
[2023-04-25 22:24:55] [build-stderr] at Socket.emit (node:events:527:28) 
[2023-04-25 22:24:55] [build-stderr] at addChunk (node:internal/streams/readable:315:12) 
[2023-04-25 22:24:55] [build-stderr] at readableAddChunk (node:internal/streams/readable:289:9) 
[2023-04-25 22:24:55] [build-stderr] at Socket.Readable.push (node:internal/streams/readable:228:10) 
[2023-04-25 22:24:55] [build-stderr] at Pipe.onStreamRead (node:internal/stream_base_commons:190:23) 
[2023-04-25 22:24:55] [build-stderr] com.semmle.util.exception.CatastrophicError: The TypeScript parser wrapper crashed with exit code 1 
[2023-04-25 22:24:55] [ERROR] Spawned process exited abnormally (code 1; tried to run: [D:\a\_work\_temp\codeql3000\github\codeql\tools\win64\tracer.exe, D:\a\_work\_temp\codeql3000\github\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, D:\a\_work\_temp\codeql3000\github\codeql\javascript\tools\autobuild.cmd]) 
A fatal error occurred: Exit status 1 from command: [D:\a\_work\_temp\codeql3000\github\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, D:\a\_work\_temp\codeql3000\github\codeql\javascript\tools\autobuild.cmd]
aibaars commented 1 year ago

Thanks for reporting. The problem seems to be caused by the use of different path component separators: D:/a/_work/1/s/build/tsconfig.json === D:\a\_work\1\s\build\tsconfig.json. We need to investigate why this happens.

As a workaround you could try switching to a Linux agent.

jason-ha commented 1 year ago

Unfortunately, we cannot use a linux agent or at least couldn't without weeks of work. The build for repository in question has complicated setup. The javascript present is a small portion; so, we can wait for a fix. Also worth noting that most of the javascript is generated during the build; so, no good option to do a one-off javascript "build" to capture that portion.

Thanks!

erik-krogh commented 1 year ago

Also worth noting that most of the javascript is generated during the build; so, no good option to do a one-off javascript "build" to capture that portion.

What is the JavaScript code build from? If it's build from other JavaScript/TypeScript in the same repo, then our analysis should work just fine without doing a build-step.
(JS generally doesn't require a build-step, unlike e.g. Java/C#/C++).

jason-ha commented 1 year ago

Mostly Emscripten

erik-krogh commented 1 year ago

Mostly Emscripten

If the source language of that code is one of the other languages covered by CodeQL (C++?), then that language analysis should take care of analyzing the source code.

Analyzing compiled JavaScript, especially from a tool like Emscripten, won't provide you much benefit, as the it's unlikely that our analysis will find anything useful, and even if it does you can't fix the JavaScript code directly.

My recommendation is to do the JavaScript analysis on a simple linux runner that doesn't do any build or setup

jason-ha commented 1 year ago

I'm not sure. I don't think there is a translation of C++ to javascript exactly such that a C++ scan would get results. I think it is more complicated. And even if we can't drive a change directly we'd want to be aware of the issue. In some cases, there might be an alternate pattern. Anyway the tooling isn't meant to just run on Linux, right? Are you considering not fixing or just discussing possible workarounds until this is fixed?

erik-krogh commented 1 year ago

Anyway the tooling isn't meant to just run on Linux, right?

Correct. I'm currently using it on a Mac. And we test most things on Linux+Mac+Windows.

Are you considering not fixing or just discussing possible workarounds until this is fixed?

The latter. I don't know how long it'll take before the issue is fixed.
I use a Mac right now, so I'm trying to pull someone else in that can help fix the issue.

I don't think there is a translation of C++ to javascript exactly such that a C++ scan would get results. I think it is more complicated.

More complicated? What is the source language?

Our analysis is not designed to work on generated code, and it generally performs badly and/or produces results that don't mean much.
This is why we generally tell people not to run any build-steps before analyzing JavaScript code. So I still think you can skip the build-step on your JavaScript code.

Oh, and you won't get in-PR alerts from generated code. The alerts will only appear in the code-scanning tab.

erik-krogh commented 10 months ago

I have tried to replicate it on a non-dev Windows box I have access to. But I can't replicate the issue.
I don't know why.
It might be that the TypeScript compiler no longer crashes on that input (it's been updated since).
Or it could be that the code I tried on didn't experience the above in the first place. (I tried on github.com/microsoft/vscode, which is a complex TypeScript application).

You can see if updating CodeQL fixes the issue 🤷

If not, then I will still recommend that you move your JavaScript analysis to a separate run, where you don't do a build. Because the JavaScript analysis does not need the build to happen, and the build artifacts might confuse the analysis or make the performance worse.

Sibamamri commented 2 weeks ago

D:\a_work_temp\codeql3000\github\codeql\codeql.exe database trace-command --index-traceless-dbs --db-cluster D:\a_work_temp\codeql3000\d Running 2 commands for 3 databases:

erik-krogh commented 2 weeks ago

As stated above I was unable to replicate the issue.

Can you give me a minimal example that cause the crash to happen?