github / codeql-cli-binaries

Binaries for the CodeQL CLI
Other
756 stars 112 forks source link

A fatal error occurred: IOException while executing process with args: [/home/abs4x/CodeQL/cli/tools/linux64/scc, --by-file, --format, json] #150

Closed giimlhd closed 1 year ago

giimlhd commented 1 year ago

When I execute the command of creating a database on LinuxX64 , an error about IOException occurred.

the command is :

codeql database create /home/abs4x/code-ql-test/test-db --overwrite  --language=javascript --source-root=/home/abs4x/code-ql-test/QlComponent/

the tip about error is:

Initializing database at /home/abs4x/code-ql-test/test-db.
A fatal error occurred: IOException while executing process with args: [/home/abs4x/CodeQL/cli/tools/linux64/scc, --by-file, --format, json]
(eventual cause: IOException "error=0, Failed to exec spawn helper: pid: 90818, exit value: 127")

the log is :

[2023-01-05 14:35:29] [PROGRESS] database init> Calculating baseline information in /home/abs4x/code-ql-test/QlComponent
[2023-01-05 14:35:29] Exception caught at top level: IOException while executing process with args: [/home/abs4x/CodeQL/cli/tools/linux64/scc, --by-file, --format, json]
                      (eventual cause: IOException "error=0, Failed to exec spawn helper: pid: 89017, exit value: 127")
                      com.semmle.util.process.AbstractProcessBuilder.execute(AbstractProcessBuilder.java:258)
                      com.semmle.util.process.AbstractProcessBuilder.execute(AbstractProcessBuilder.java:198)
                      com.semmle.cli2.database.baseline.SccBaseline.run(SccBaseline.java:72)
                      com.semmle.cli2.database.baseline.SccBaseline.calculate(SccBaseline.java:116)
                      com.semmle.cli2.database.InitCommand.executeSubcommand(InitCommand.java:200)
                      com.semmle.cli2.picocli.SubcommandCommon.lambda$executeSubcommandWithMessages$2(SubcommandCommon.java:625)
                      com.semmle.cli2.picocli.SubcommandCommon.withCompilationMessages(SubcommandCommon.java:405)
                      com.semmle.cli2.picocli.SubcommandCommon.executeSubcommandWithMessages(SubcommandCommon.java:624)
                      com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110)
                      com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(SubcommandCommon.java:188)
                      com.semmle.cli2.database.CreateCommand.executeSubcommand(CreateCommand.java:84)
                      com.semmle.cli2.picocli.SubcommandCommon.lambda$executeSubcommandWithMessages$2(SubcommandCommon.java:625)
                      com.semmle.cli2.picocli.SubcommandCommon.withCompilationMessages(SubcommandCommon.java:405)
                      com.semmle.cli2.picocli.SubcommandCommon.executeSubcommandWithMessages(SubcommandCommon.java:624)
                      com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:561)
                      com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:223)
                      com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:232)
                      com.semmle.cli2.CodeQL.main(CodeQL.java:100)

                       ... caused by:

                      java.io.IOException: Cannot run program "/home/abs4x/CodeQL/cli/tools/linux64/scc" (in directory "/home/abs4x/code-ql-test/QlComponent"): error=0, Failed to exec spawn helper: pid: 89017, exit value: 127
                      java.base/java.lang.ProcessBuilder.start(Unknown Source)
                      java.base/java.lang.ProcessBuilder.start(Unknown Source)
                      com.semmle.util.process.AbstractProcessBuilder.execute(AbstractProcessBuilder.java:231)
                      com.semmle.util.process.AbstractProcessBuilder.execute(AbstractProcessBuilder.java:198)
                      com.semmle.cli2.database.baseline.SccBaseline.run(SccBaseline.java:72)
                      com.semmle.cli2.database.baseline.SccBaseline.calculate(SccBaseline.java:116)
                      com.semmle.cli2.database.InitCommand.executeSubcommand(InitCommand.java:200)
                      com.semmle.cli2.picocli.SubcommandCommon.lambda$executeSubcommandWithMessages$2(SubcommandCommon.java:625)
                      com.semmle.cli2.picocli.SubcommandCommon.withCompilationMessages(SubcommandCommon.java:405)
                      com.semmle.cli2.picocli.SubcommandCommon.executeSubcommandWithMessages(SubcommandCommon.java:624)
                      com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110)
                      com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(SubcommandCommon.java:188)
                      com.semmle.cli2.database.CreateCommand.executeSubcommand(CreateCommand.java:84)
                      com.semmle.cli2.picocli.SubcommandCommon.lambda$executeSubcommandWithMessages$2(SubcommandCommon.java:625)
                      com.semmle.cli2.picocli.SubcommandCommon.withCompilationMessages(SubcommandCommon.java:405)
                      com.semmle.cli2.picocli.SubcommandCommon.executeSubcommandWithMessages(SubcommandCommon.java:624)
                      com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:561)
                      com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:223)
                      com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:232)
                      com.semmle.cli2.CodeQL.main(CodeQL.java:100)

                       ... caused by:

                      java.io.IOException: error=0, Failed to exec spawn helper: pid: 89017, exit value: 127
                      java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
                      java.base/java.lang.ProcessImpl.<init>(Unknown Source)
                      java.base/java.lang.ProcessImpl.start(Unknown Source)
                      java.base/java.lang.ProcessBuilder.start(Unknown Source)
                      java.base/java.lang.ProcessBuilder.start(Unknown Source)
                      com.semmle.util.process.AbstractProcessBuilder.execute(AbstractProcessBuilder.java:231)
                      com.semmle.util.process.AbstractProcessBuilder.execute(AbstractProcessBuilder.java:198)
                      com.semmle.cli2.database.baseline.SccBaseline.run(SccBaseline.java:72)
                      com.semmle.cli2.database.baseline.SccBaseline.calculate(SccBaseline.java:116)
                      com.semmle.cli2.database.InitCommand.executeSubcommand(InitCommand.java:200)
                      com.semmle.cli2.picocli.SubcommandCommon.lambda$executeSubcommandWithMessages$2(SubcommandCommon.java:625)
                      com.semmle.cli2.picocli.SubcommandCommon.withCompilationMessages(SubcommandCommon.java:405)
                      com.semmle.cli2.picocli.SubcommandCommon.executeSubcommandWithMessages(SubcommandCommon.java:624)
                      com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110)
                      com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(SubcommandCommon.java:188)
                      com.semmle.cli2.database.CreateCommand.executeSubcommand(CreateCommand.java:84)
                      com.semmle.cli2.picocli.SubcommandCommon.lambda$executeSubcommandWithMessages$2(SubcommandCommon.java:625)
                      com.semmle.cli2.picocli.SubcommandCommon.withCompilationMessages(SubcommandCommon.java:405)
                      com.semmle.cli2.picocli.SubcommandCommon.executeSubcommandWithMessages(SubcommandCommon.java:624)
                      com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:561)
                      com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:223)
                      com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:232)
                      com.semmle.cli2.CodeQL.main(CodeQL.java:100)

I do not know how to solve this problem.

giimlhd commented 1 year ago

the version of codeql-cli is 2.11.3

hmakholm commented 1 year ago

Hmm, haven't seen that one before. I'd recommend the following troubleshooting steps:

  1. Does the problem still happen for you with the latest released CodeQL version (2.11.5)?
  2. Run codeql version --format json to verify that the unpackedLocation is what you expect it to be.
  3. Verify that [unpackedLocation]/tools/linux64/java/lib/jspawnhelper exists and has execute permissions.
  4. Check that you don't have the environment variable $CODEQL_JAVA_HOME set.
  5. Does giving a -j-Djdk.lang.Process.launchMechanism=vfork help (or produce a different error)?

By the way, which Linux distribution are you running, and can you find out which version of glibc?

giimlhd commented 1 year ago

@hmakholm Thank you for your suggestions.

The Linux distribution is :
Linux version 3.10.0-862.el7.x86_64 (mockbuild@x86-034.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) ) ;

the glibc : glibc-2.17-222.el7

The problem still happen with the latest released CodeQL version (2.11.5) and the unpackedLocation is also what I expect it to be.

In fact, I have to run codeql-cli under a non root account, and have improved execute permissions of some files running chmod -R 777 codeql[ java、jspawnhelper ] in these directories one by one: [unpackedLocation]/codeql[unpackedLocation]/tools/linux64/java/bin/java[unpackedLocation]/tools/linux64/java/lib/jspawnhelper , and then , the IOException error mentioned above doesn't occurs, but another different error occurs instead when I run the command to create a database .

The command is the same as before:

codeql database create /home/abs4x/code-ql-test/test-db --overwrite  --language=javascript --source-root=/home/abs4x/code-ql-test/QlComponent/

The error is :

Initializing database at /home/abs4x/code-ql-test/test-db.
Running build command: []
Finalizing database at /home/abs4x/code-ql-test/test-db.
No source code was seen and extracted to /home/abs4x/code-ql-test/test-db.
This can occur if the specified build commands failed to compile or process any code.
 - Confirm that there is some source code for the specified language in the project.
 - For codebases written in Go, JavaScript, TypeScript, and Python, do not specify
   an explicit --command.
 - For other languages, the --command must specify a "clean" build which compiles
   all the source code files without reusing existing build artefacts.

How to solve this error ?

Can't the codeql-cli run under a non root account ?

hmakholm commented 1 year ago

CodeQL definitely doesn't require root -- I don't think we ever run or test it with root privileges ourselves.

It sounds like something is going wrong when you're unzipping the CLI download. On my Ubuntu 22.04 machine, all of the executables you mention unzip with permissions -rwxr-xr-x for me. Perhaps you were running the unzipper with a restrictive umask in effect? Or, if you let a web browser do the unzipping for you, it might have tampered with the permissions in the name of security.

The No source code was seen and extracted error appears to be fallout from a similar permission problems with another internal CodeQL script -- I can reproduce your symptom by removing the execute permission from codeql/javascript/tools/autobuild.sh after unzipping. (We seem to lack error checking for that case; I'll file an internal issue).

Thus the cure for that would be to make sure you're unzipping CodeQL with the correct permissions to begin with -- playing whack-a-mole with individual components doesn't feel promising.

giimlhd commented 1 year ago

Thank you for your reminder very much !

These permission problems did occur if I unzip the Cli download in a window system and then recompress to get a new zip file which will be uploaded to my linux server.(I admit my operations are a little silly).

Now I know the correct way is to upload the released Cli download without any operations from a window system to a linux one and it works.

good !