search

github / codeql-cli-binaries

Binaries for the CodeQL CLI
Other
754 stars 112 forks source link

Failure with codeql database create #61

Open cspidave opened 3 years ago

cspidave commented 3 years ago

I am trying to use codeql in support of the MS HLK driver certification. Following their directions, I installed and configured codeql for windows. My problem is when I try to do a database create, and pass the -c "build command" Our build process is slightly more complex than just calling msbuild in a directory - we need to configure the environment first. So I created a script to do just that (both as a cygwin bash and then as that wrapped in a .bat).

When I do that (after carefully testing the scripts in cygwin, cmd, powershell) I get a failure with the log below. If I try the runner.exe command, that produces the correct build. So tracer.exe is failing and I was not able to figure out what I am doing wrong. Suggestions would we welcome :-)

$ cat database-create-20210415.154254.408.log [2021-04-15 15:42:54] This is codeql database create -l cpp -s C:/cygwin/home/cspi/dave/tikiql/driver/ndis6 -c C:/cygwin/bin/bash.exe gocodeql.sh C:/codeql-home/databases/mva -j 0 [2021-04-15 15:42:54] Log file was started late. [2021-04-15 15:42:54] [PROGRESS] database create> Running command [C:/cygwin/bin/bash.exe, gocodeql.sh] in C:\cygwin\home\cspi\dave\tikiql\driver\ndis6. [2021-04-15 15:42:54] Running plumbing command: codeql database trace-command --working-dir=C:\cygwin\home\cspi\dave\tikiql\driver\ndis6 -- C:\codeql-home\databases\mva C:/cygwin/bin/bash.exe gocodeql.sh [2021-04-15 15:42:54] [PROGRESS] database trace-command> C:\codeql-home\databases\mva: Running in C:\cygwin\home\cspi\dave\tikiql\driver\ndis6: [C:/cygwin/bin/bash.exe, gocodeql.sh] [2021-04-15 15:42:54] Setting up tracing from specification at C:\codeql-home\codeql\cpp\tools\win64\compiler-tracing.spec. [2021-04-15 15:42:54] [ERROR] Spawned process exited abnormally (code -1073741790; tried to run: [C:\codeql-home\codeql\tools\win64\tracer.exe, C:\codeql-home\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, C:/cygwin/bin/bash.exe, gocodeql.sh]) [2021-04-15 15:42:54] Exception caught at top level: Exit status -1073741790 from command: [C:\codeql-home\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, C:/cygwin/bin/bash.exe, gocodeql.sh] com.semmle.cli2.database.DatabaseProcessCommandCommon.executeSubcommand(DatabaseProcessCommandCommon.java:99) com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110) com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(SubcommandCommon.java:159) com.semmle.cli2.database.CreateCommand.executeSubcommand(CreateCommand.java:113) com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:430) com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:201) com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:209) com.semmle.cli2.CodeQL.main(CodeQL.java:93)

dbartol commented 3 years ago

If I understand the logs above correctly, you are running codeql database create, passing your script as the build command, and your build script invokes the CodeQL runner.exe. The CodeQL runner.exe is a high-level wrapper for integrating usage of CodeQL into CI systems. If you're just trying to build a CodeQL database locally, try having your script just invoke your build directly, without using the runner as an intermediary. There's a good chance it's the unnecessary invocation of the runner that's causing the problem. If you still have the same problem after removing the runner, though, please reply to this issue with an update and I'll help you figure out the problem.

cspidave commented 3 years ago

David, thanks for the response. If I understood your comments....

The problem I reported was found when I tried to run: codeql database create -l cpp -s C:/cygwin/home/cspi/dave/tikiql/driver/ndis6 -c 'C:/cygwin/bin/bash.exe gocodeql.sh' C:/codeql-home/databases/mva -j 0

with the -c being a script invocation to get the build done. (I also tried a go.bat and several other variation on the theme, all which work from cmd.exe)

If I remove the runner - by which I am assuming here you mean to remove the -c "build script" I get:

(after I prebuild my code)

$ /cygdrive/c/codeql-home/codeql/codeql.exe database create -l cpp -s $(cygpath.exe -m $PWD) $(cygpath.exe -m /cygdrive/c/codeql-home/databases/mva) -j 0 Initializing database at C:\codeql-home\databases\mva. A fatal error occurred: Could not detect autobuild script in C:\codeql-home\codeql\cpp

So after reading up on autobuild again, I recreated a build.bat in my working directory that I tested with:

cmd.exe /c build.bat

and saw that it was good. When I ran the above command again, no change. So then copied that build.bat to /codeql-home/codeql/cpp and again, no change.

Clearly I must be missing something when I read the fine manual :-( So any suggestion as to what I am obviously missing would be appreciated.

One last observation: https://docs.github.com/en/code-security/secure-coding/configuring-the-codeql-workflow-for-compiled-languages

says 1 msbuild.exe on a .sln file (which we have) 2 Invoke a script that looks like a build script—build.bat, build.cmd, and build.exe (in that order)

and I do need to set up a specific env before I end up calling mdbuild.exe so I think I do need an override which I thought the '-c' would do, but that is of course what is failing.

thanks!

hmakholm commented 3 years ago

@dbartol, I don't think this runner.exe is the "CodeQL runner". It seems to be something that actually ships with the CLI, built from semmle-code/c/runner/runner-win.c. It appears to be a command-line wrapper around CreateProcessW with special bells and whistles for privilege management under LGTM. I assumed it was part of the tracer infrastructure?

cspidave commented 3 years ago

Called by tracer.exe yes. The issue I see does not seem to be with \codeql-home\codeql\tools\win64\runner.exe,, but rather tracer failing while calling runner - that is I can execute what seems to be the command line with runner.exe and it works for me.

dbartol commented 3 years ago

@hmakholm Ugh. Thanks for pointing that out. We ship too many things called "runner" 🤦.

@cspidave Sorry for the confusion. Please ignore anything I said above about "runner".

There's no special environment you need to set up for CodeQL, other than what you'd set up for your normal build environment. Basically, if you can go to a cmd/PS/bash/whatever prompt and type C:/cygwin/bin/bash.exe gocodeql.sh and it builds your project correctly, then you should also be able to go to that same prompt and type codeql database create -l cpp -s C:/cygwin/home/cspi/dave/tikiql/driver/ndis6 -c 'C:/cygwin/bin/bash.exe gocodeql.sh' C:/codeql-home/databases/mva -j 0 and have it build your database correctly. We don't care if your build command is msbuild, make, or a bunch of Perl scripts; we just trace every invocation of the compiler as your build process invokes it, and use that to build our database.

Assuming your build script works just fine without running it via codeql database create, it's possible that our tracer has a problem tracing Cygwin binaries. Does your build script also work under git bash, which uses Msys instead of Cygwin? If so, could you try building your database using git bash as your shell interpreter rather than Cywin's bash? If that works, that would narrow down our problem to Cygwin so we can take a closer look.

dbartol commented 3 years ago

Also, is there a file named build-tracer.log in the log directory of C:/codeql-home/databases/mva? That file is the detailed log of our tracer, and may show more detailed errors if it is the tracing code that is hitting the error.

cspidave commented 3 years ago

Learn something new.... git for windows has a bash.exe :-) Looks like the same result. In detail:

Installed windows git, tweaked my build script to work with the slightly different semantics. tried a build.bat from powershell:

"c:/Program Files/Git/bin/bash.exe" gitgo.sh

that worked from powershell. So two trials, trying to avoid cygwin (running from git bash, and using git bash in the invocation)

Trying to wrap it in the above bat

cspi@windave MINGW64 /c/cygwin/home/cspi/dave/tikiql/driver/ndis6 (devel/mva-2.1.0) $ bash -x ./doit.sh

And trying it without the bat (-c is the content of the bat)

cspi@windave MINGW64 /c/cygwin/home/cspi/dave/tikiql/driver/ndis6 (devel/mva-2.1.0) $ bash -x ./doit2.sh

and just to show what my gitgo.sh is:

$ cat gitgo.sh

set up the right env

source ../../mva/wdk_env/export_wlh_x64

and run the right version of msbuild.exe

exec "/d/Program Files/Microsoft Visual Studio/2019/BuildTools/MSBuild/Current/Bin/amd64/msbuild.exe" /target:rebuild

The log from the last run was: $ cat /cygdrive/c/codeql-home/databases/mva/log/database-create-20210421.200648.865.log [2021-04-21 20:06:48] This is codeql database create -l cpp -s C:/cygwin/home/cspi/dave/tikiql/driver/ndis6 -c c:/Program Files/Git/bin/bash.exe gitgo.sh C:/codeql-home/databases/mva -j 0 [2021-04-21 20:06:48] Log file was started late. [2021-04-21 20:06:48] [PROGRESS] database create> Running command [c:/Program, Files/Git/bin/bash.exe, gitgo.sh] in C:\cygwin\home\cspi\dave\tikiql\driver\ndis6. [2021-04-21 20:06:48] Running plumbing command: codeql database trace-command --working-dir=C:\cygwin\home\cspi\dave\tikiql\driver\ndis6 -- C:\codeql-home\databases\mva c:/Program Files/Git/bin/bash.exe gitgo.sh [2021-04-21 20:06:48] [PROGRESS] database trace-command> C:\codeql-home\databases\mva: Running in C:\cygwin\home\cspi\dave\tikiql\driver\ndis6: [c:/Program, Files/Git/bin/bash.exe, gitgo.sh] [2021-04-21 20:06:48] Setting up tracing from specification at C:\codeql-home\codeql\cpp\tools\win64\compiler-tracing.spec. [2021-04-21 20:06:48] [ERROR] Spawned process exited abnormally (code -1073741790; tried to run: [C:\codeql-home\codeql\tools\win64\tracer.exe, C:\codeql-home\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, c:/Program, Files/Git/bin/bash.exe, gitgo.sh]) [2021-04-21 20:06:48] Exception caught at top level: Exit status -1073741790 from command: [C:\codeql-home\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, c:/Program, Files/Git/bin/bash.exe, gitgo.sh] com.semmle.cli2.database.DatabaseProcessCommandCommon.executeSubcommand(DatabaseProcessCommandCommon.java:99) com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110) com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(SubcommandCommon.java:159) com.semmle.cli2.database.CreateCommand.executeSubcommand(CreateCommand.java:113) com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:430) com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:201) com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:209) com.semmle.cli2.CodeQL.main(CodeQL.java:93)

cspidave commented 3 years ago

Another set of data points:

We are using the MICROSOFT ENTERPRISE WINDOWS DRIVER KIT mounted as drive d:

I cloned the Windows echo driver example: https://github.com/microsoft/Windows-driver-samples/tree/master/general/echo

Using the command environment created by LaunchBuildEnv which sets the settings/paths needed for the WDK. I change directories to the target directory on disk. Note: at this point I am using no Cygwin elements, except in the directory names of my working directory.

I checked that the build command: msbuild /t:rebuild /p:Platform=x64 Worked correctly from the command line. I invoked:

C:\cygwin\home\cspi\dave\Windows-driver-samples\general\echo\kmdf>\codeql-home\codeql\codeql.exe database create -l=cpp -c="msbuild /t:rebuild /p:Platform=x64" \codeql-home\databases\echo Initializing database at C:\codeql-home\databases\echo. Running command [msbuild, /t:rebuild, /p:Platform=x64] in C:\cygwin\home\cspi\dave\Windows-driver-samples\general\echo\kmdf. [2021-04-26 13:49:04] [ERROR] Spawned process exited abnormally (code -1073741790; tried to run: [C:\codeql-home\codeql\tools\win64\tracer.exe, C:\codeql-home\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, msbuild, /t:rebuild, /p:Platform=x64]) A fatal error occurred: Exit status -1073741790 from command: [C:\codeql-home\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, msbuild, /t:rebuild, /p:Platform=x64]

Which replicates the problem seen with our driver.

The generated log file:

cspi@windave /cygdrive/c/codeql-home/databases/echo/log $ cat database-create-20210426.134904.173.log [2021-04-26 13:49:04] This is codeql database create -l=cpp -c=msbuild /t:rebuild /p:Platform=x64 \codeql-home\databases\echo [2021-04-26 13:49:04] Log file was started late. [2021-04-26 13:49:04] [PROGRESS] database create> Running command [msbuild, /t:rebuild, /p:Platform=x64] in C:\cygwin\home\cspi\dave\Windows-driver-samples\general\echo\kmdf. [2021-04-26 13:49:04] Running plumbing command: codeql database trace-command --working-dir=C:\cygwin\home\cspi\dave\Windows-driver-samples\general\echo\kmdf -- C:\codeql-home\databases\echo msbuild /t:rebuild /p:Platform=x64 [2021-04-26 13:49:04] [PROGRESS] database trace-command> C:\codeql-home\databases\echo: Running in C:\cygwin\home\cspi\dave\Windows-driver-samples\general\echo\kmdf: [msbuild, /t:rebuild, /p:Platform=x64] [2021-04-26 13:49:04] Setting up tracing from specification at C:\codeql-home\codeql\cpp\tools\win64\compiler-tracing.spec. [2021-04-26 13:49:04] [ERROR] Spawned process exited abnormally (code -1073741790; tried to run: [C:\codeql-home\codeql\tools\win64\tracer.exe, C:\codeql-home\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, msbuild, /t:rebuild, /p:Platform=x64]) [2021-04-26 13:49:04] Exception caught at top level: Exit status -1073741790 from command: [C:\codeql-home\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, msbuild, /t:rebuild, /p:Platform=x64] com.semmle.cli2.database.DatabaseProcessCommandCommon.executeSubcommand(DatabaseProcessCommandCommon.java:99) com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110) com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(SubcommandCommon.java:159) com.semmle.cli2.database.CreateCommand.executeSubcommand(CreateCommand.java:113) com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:430) com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:201) com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:209) com.semmle.cli2.CodeQL.main(CodeQL.java:93)

Shivam60 commented 3 years ago

I have the exact issue for javascript files

Can any give an update on this ?

@cspidave @dbartol