search

github / codeql

CodeQL: the libraries and queries that power security researchers around the world, as well as code scanning in GitHub Advanced Security
https://codeql.github.com
MIT License
7.67k stars 1.54k forks source link

Error in creating a new java pro database #15198

Open Xieyufengdd opened 10 months ago

Xieyufengdd commented 10 months ago

when excuting "codeql database create .\compiler-db\ --language=java --source-root=E:\JavaPro\Compiler\src" in command line, it reports:

[2023-12-23 18:49:50] [build-stderr] ERROR: Could not detect a suitable build command for the source checkout.
[2023-12-23 18:49:50] [ERROR] Spawned process exited abnormally (code 1; tried to run: [E:\C_Pro\codeql\tools\win64\tracer.exe, E:\C_Pro\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, E:\C_Pro\codeql\java\tools\autobuild.cmd])
A fatal error occurred: Exit status 1 from command: [E:\C_Pro\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, E:\C_Pro\codeql\java\tools\autobuild.cmd]

build-tracer.log is here:

[T 18:49:49 31224] Initializing tracer.
[T 18:49:49 31224] Initialising tags...
[T 18:49:49 31224] CodeQL CLI version 2.15.5
[T 18:49:49 31224] Initializing tracer.
[T 18:49:49 31224] Initialising tags...
[T 18:49:49 31224] Allocated ID 26194622000079F8_0000000000000001 (parent )
[T 18:49:49 31224] ==== Candidate to intercept: E:\C_Pro\codeql\tools\win64\runner.exe (8100) ====
[T 18:49:49 31224] Lua: === Intercepted call to e:\c_pro\codeql\tools\win64\runner.exe ===
[T 18:49:49 31224] Executing the following tracer actions:
[T 18:49:49 31224] Tracer actions:
[T 18:49:49 31224] pre_invocations(0)
[T 18:49:49 31224] post_invocations(0)
[T 18:49:49 31224] trace_languages(1): [java]
[T 18:49:49 8100] Initializing tracer.
[T 18:49:49 8100] Initialising tags...
[T 18:49:49 8100] ID set to 26194622000079F8_0000000000000001 (parent root)
[T 18:49:49 8100] Allocated ID 727A9A4500001FA4_0000000000000001 (parent 26194622000079F8_0000000000000001)
[T 18:49:49 8100] ==== Candidate to intercept: C:\Windows\System32\cmd.exe (26852) ====
[T 18:49:49 8100] Lua: === Intercepted call to c:\windows\system32\cmd.exe ===
[T 18:49:49 8100] Executing the following tracer actions:
[T 18:49:49 8100] Tracer actions:
[T 18:49:49 8100] pre_invocations(0)
[T 18:49:49 8100] post_invocations(0)
[T 18:49:49 8100] trace_languages(1): [java]
[T 18:49:49 26852] Initializing tracer.
[T 18:49:49 26852] Initialising tags...
[T 18:49:49 26852] ID set to 727A9A4500001FA4_0000000000000001 (parent 26194622000079F8_0000000000000001)
[T 18:49:50 26852] Allocated ID D6BA57A5000068E4_0000000000000001 (parent 727A9A4500001FA4_0000000000000001)
[T 18:49:50 26852] ==== Candidate to intercept: E:\C_Pro\codeql\tools\win64\java\bin\java.exe (1828) ====
[T 18:49:50 26852] Lua: === Intercepted call to e:\c_pro\codeql\tools\win64\java\bin\java.exe ===
[T 18:49:50 26852] Executing the following tracer actions:
[T 18:49:50 26852] Tracer actions:
[T 18:49:50 26852] pre_invocations(0)
[T 18:49:50 26852] post_invocations(0)
[T 18:49:50 26852] trace_languages(1): [java]
[T 18:49:50 1828] Initializing tracer.
[T 18:49:50 1828] Initialising tags...
[T 18:49:50 1828] ID set to D6BA57A5000068E4_0000000000000001 (parent 727A9A4500001FA4_0000000000000001)
[T 18:49:50 1828] Passing through -javaagent:E:\C_Pro\codeql\java\tools\codeql-java-agent.jar=ignore-project,java,kotlin to underlying JVM.
[T 18:49:50 1828] Passing through -Xbootclasspath/a:E:\C_Pro\codeql\java\tools\codeql-java-agent.jar to underlying JVM.
[T 18:49:50 1828] Intercepted JVM creation with extra hidden args: ['-javaagent:E:\C_Pro\codeql\java\tools\codeql-java-agent.jar=ignore-project,java,kotlin' '-Xbootclasspath/a:E:\C_Pro\codeql\java\tools\codeql-java-agent.jar' ] (result: 1, 0).
Xieyufengdd commented 10 months ago

here is another log file database-create-20231223.184948.990.log :


  [2023-12-23 18:49:49] This is codeql database create .\compiler-db\ --language=java --source-root=E:\JavaPro\Compiler\src --overwrite
[2023-12-23 18:49:49] Log file was started late.
[2023-12-23 18:49:49] [PROGRESS] database create> Initializing database at E:\JavaPro\Compiler\compiler-db.
[2023-12-23 18:49:49] Running plumbing command: codeql database init --overwrite --language=java --extractor-options-verbosity=1 --qlconfig-file=E:\JavaPro\Compiler\qlconfig.yml --source-root=E:\JavaPro\Compiler\src --allow-missing-source-root=false --allow-already-existing -- E:\JavaPro\Compiler\compiler-db
[2023-12-23 18:49:49] Calling plumbing command: codeql resolve languages --extractor-options-verbosity=1 --format=betterjson
[2023-12-23 18:49:49] [DETAILS] resolve languages> Scanning for [codeql-extractor.yml] from E:\C_Pro\codeql\.codeqlmanifest.json
[2023-12-23 18:49:49] [DETAILS] resolve languages> Parsing E:\C_Pro\codeql\cpp\codeql-extractor.yml.
[2023-12-23 18:49:49] [DETAILS] resolve languages> Parsing E:\C_Pro\codeql\csharp\codeql-extractor.yml.
[2023-12-23 18:49:49] [DETAILS] resolve languages> Parsing E:\C_Pro\codeql\csv\codeql-extractor.yml.
[2023-12-23 18:49:49] [DETAILS] resolve languages> Parsing E:\C_Pro\codeql\go\codeql-extractor.yml.
[2023-12-23 18:49:49] [DETAILS] resolve languages> Parsing E:\C_Pro\codeql\html\codeql-extractor.yml.
[2023-12-23 18:49:49] [DETAILS] resolve languages> Parsing E:\C_Pro\codeql\java\codeql-extractor.yml.
[2023-12-23 18:49:49] [DETAILS] resolve languages> Parsing E:\C_Pro\codeql\javascript\codeql-extractor.yml.
[2023-12-23 18:49:49] [DETAILS] resolve languages> Parsing E:\C_Pro\codeql\properties\codeql-extractor.yml.
[2023-12-23 18:49:49] [DETAILS] resolve languages> Parsing E:\C_Pro\codeql\python\codeql-extractor.yml.
[2023-12-23 18:49:49] [DETAILS] resolve languages> Parsing E:\C_Pro\codeql\ruby\codeql-extractor.yml.
[2023-12-23 18:49:49] [DETAILS] resolve languages> Parsing E:\C_Pro\codeql\swift\codeql-extractor.yml.
[2023-12-23 18:49:49] [DETAILS] resolve languages> Parsing E:\C_Pro\codeql\xml\codeql-extractor.yml.
[2023-12-23 18:49:49] [DETAILS] resolve languages> Parsing E:\C_Pro\codeql\yaml\codeql-extractor.yml.
[2023-12-23 18:49:49] Plumbing command codeql resolve languages completed:
                      {
                        "aliases" : {
                          "c" : "cpp",
                          "c++" : "cpp",
                          "c-c++" : "cpp",
                          "c-cpp" : "cpp",
                          "c#" : "csharp",
                          "java-kotlin" : "java",
                          "kotlin" : "java",
                          "javascript-typescript" : "javascript",
                          "typescript" : "javascript"
                        },
                        "extractors" : {
                          "cpp" : [
                            {
                              "extractor_root" : "E:\\C_Pro\\codeql\\cpp"
                            }
                          ],
                          "csharp" : [
                            {
                              "extractor_root" : "E:\\C_Pro\\codeql\\csharp",
                              "extractor_options" : {
                                "trap" : {
                                  "title" : "Options pertaining to TRAP.",
                                  "description" : "Options pertaining to TRAP.",
                                  "type" : "object",
                                  "properties" : {
                                    "compression" : {
                                      "title" : "Controls compression for the TRAP files written by the extractor.",
                                      "description" : "This option is only intended for use in debugging the extractor. Accepted values are 'brotli' (the default, to write brotli-compressed TRAP), 'gzip', and 'none' (to write uncompressed TRAP).\n",
                                      "type" : "string",
                                      "pattern" : "^(none|gzip|brotli)$"
                                    }
                                  }
                                },
                                "buildless" : {
                                  "title" : "Whether to use buildless (standalone) extraction.",
                                  "description" : "A value indicating, which type of extraction the autobuilder should perform. If 'true', then the standalone extractor will be used, otherwise tracing extraction will be performed. The default is 'false'. Note that buildless extraction will generally yield less accurate analysis results, and should only be used in cases where it is not possible to build the code (for example if it uses inaccessible dependencies).\n",
                                  "type" : "string",
                                  "pattern" : "^(false|true)$"
                                },
                                "cil" : {
                                  "title" : "Whether to enable CIL extraction.",
                                  "description" : "A value indicating, whether CIL extraction should be enabled. The default is 'true'.\n",
                                  "type" : "string",
                                  "pattern" : "^(false|true)$"
                                }
                              }
                            }
                          ],
                          "csv" : [
                            {
                              "extractor_root" : "E:\\C_Pro\\codeql\\csv"
                            }
                          ],
                          "go" : [
                            {
                              "extractor_root" : "E:\\C_Pro\\codeql\\go"
                            }
                          ],
                          "html" : [
                            {
                              "extractor_root" : "E:\\C_Pro\\codeql\\html"
                            }
                          ],
                          "java" : [
                            {
                              "extractor_root" : "E:\\C_Pro\\codeql\\java",
                              "extractor_options" : {
                                "exclude" : {
                                  "title" : "A glob excluding files from analysis.",
                                  "description" : "A glob indicating what files to exclude from the analysis.\n",
                                  "type" : "string"
                                },
                                "add_prefer_source" : {
                                  "title" : "Whether to always prefer source files over class files.",
                                  "description" : "A value indicating whether source files should be preferred over class files. If set to 'true', the extraction adds '-Xprefer:source' to the javac command line. If set to 'false', the extraction uses the default javac behavior ('-Xprefer:newer'). The default is 'true'.\n",
                                  "type" : "string",
                                  "pattern" : "^(false|true)$"
                                },
                                "buildless" : {
                                  "title" : "Whether to use buildless (standalone) extraction (experimental).",
                                  "description" : "A value indicating, which type of extraction the autobuilder should perform. If 'true', then the standalone extractor will be used, otherwise tracing extraction will be performed. The default is 'false'. Note that buildless extraction will generally yield less accurate analysis results, and should only be used in cases where it is not possible to build the code (for example if it uses inaccessible dependencies).\n",
                                  "type" : "string",
                                  "pattern" : "^(false|true)$"
                                }
                              }
                            }
                          ],
                          "javascript" : [
                            {
                              "extractor_root" : "E:\\C_Pro\\codeql\\javascript",
                              "extractor_options" : { }
                            }
                          ],
                          "properties" : [
                            {
                              "extractor_root" : "E:\\C_Pro\\codeql\\properties"
                            }
                          ],
                          "python" : [
                            {
                              "extractor_root" : "E:\\C_Pro\\codeql\\python",
                              "extractor_options" : {
                                "logging" : {
                                  "title" : "Options pertaining to logging.",
                                  "description" : "Options pertaining to logging.",
                                  "type" : "object",
                                  "properties" : {
                                    "verbosity" : {
                                      "title" : "Python extractor logging verbosity level.",
                                      "description" : "Controls the level of verbosity of the CodeQL Python extractor.\nThe supported levels are (in order of increasing verbosity):\n\n  - off\n  - errors\n  - warnings\n  - info or progress\n  - debug or progress+\n  - trace or progress++\n  - progress+++\n",
                                      "type" : "string",
                                      "pattern" : "^(off|errors|warnings|(info|progress)|(debug|progress\\+)|(trace|progress\\+\\+)|progress\\+\\+\\+)$"
                                    }
                                  }
                                }
                              }
                            }
                          ],
                          "ruby" : [
                            {
                              "extractor_root" : "E:\\C_Pro\\codeql\\ruby",
                              "extractor_options" : {
                                "trap" : {
                                  "title" : "Options pertaining to TRAP.",
                                  "description" : "Options pertaining to TRAP.",
                                  "type" : "object",
                                  "properties" : {
                                    "compression" : {
                                      "title" : "Controls compression for the TRAP files written by the extractor.",
                                      "description" : "This option is only intended for use in debugging the extractor. Accepted values are 'gzip' (the default, to write gzip-compressed TRAP) and 'none' (to write uncompressed TRAP).\n",
                                      "type" : "string",
                                      "pattern" : "^(none|gzip)$"
                                    }
                                  }
                                }
                              }
                            }
                          ],
                          "swift" : [
                            {
                              "extractor_root" : "E:\\C_Pro\\codeql\\swift"
                            }
                          ],
                          "xml" : [
                            {
                              "extractor_root" : "E:\\C_Pro\\codeql\\xml"
                            }
                          ],
                          "yaml" : [
                            {
                              "extractor_root" : "E:\\C_Pro\\codeql\\yaml"
                            }
                          ]
                        }
                      }
[2023-12-23 18:49:49] [PROGRESS] database init> Calculating baseline information in E:\JavaPro\Compiler\src
[2023-12-23 18:49:49] [SPAMMY] database init> Ignoring the following directories when processing baseline information: .git, .hg, .svn.
[2023-12-23 18:49:49] [DETAILS] database init> Running command in E:\JavaPro\Compiler\src: E:\C_Pro\codeql\tools\win64\scc.exe --by-file --exclude-dir .git,.hg,.svn --format json --no-large --no-min .
[2023-12-23 18:49:49] [SPAMMY] database init> Found 3 baseline files for java.
[2023-12-23 18:49:49] [PROGRESS] database init> Calculated baseline information for languages: java (76ms).
[2023-12-23 18:49:49] [PROGRESS] database init> Resolving extractor java.
[2023-12-23 18:49:49] [DETAILS] database init> Found candidate extractor root for java: E:\C_Pro\codeql\java.
[2023-12-23 18:49:49] [PROGRESS] database init> Successfully loaded extractor Java/Kotlin (java) from E:\C_Pro\codeql\java.
[2023-12-23 18:49:49] [PROGRESS] database init> Created skeleton CodeQL database at E:\JavaPro\Compiler\compiler-db. This in-progress database is ready to be populated by an extractor.
[2023-12-23 18:49:49] Plumbing command codeql database init completed.
[2023-12-23 18:49:49] Using autobuild script E:\C_Pro\codeql\java\tools\autobuild.cmd.
[2023-12-23 18:49:49] [DETAILS] database create> Detected autobuild command for Java/Kotlin: E:\C_Pro\codeql\java\tools\autobuild.cmd.
[2023-12-23 18:49:49] [PROGRESS] database create> Running build command: [E:\C_Pro\codeql\java\tools\autobuild.cmd]
[2023-12-23 18:49:49] Running plumbing command: codeql database trace-command --working-dir=E:\JavaPro\Compiler\src --index-traceless-dbs --no-db-cluster -- E:\JavaPro\Compiler\compiler-db E:\C_Pro\codeql\java\tools\autobuild.cmd
[2023-12-23 18:49:49] [PROGRESS] database trace-command> Running command in E:\JavaPro\Compiler\src: [E:\C_Pro\codeql\java\tools\autobuild.cmd]
[2023-12-23 18:49:50] [build-stderr] ERROR: Could not detect a suitable build command for the source checkout.
[2023-12-23 18:49:50] [ERROR] Spawned process exited abnormally (code 1; tried to run: [E:\C_Pro\codeql\tools\win64\tracer.exe, E:\C_Pro\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, E:\C_Pro\codeql\java\tools\autobuild.cmd])
[2023-12-23 18:49:50] Exception caught at top level: Exit status 1 from command: [E:\C_Pro\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, E:\C_Pro\codeql\java\tools\autobuild.cmd]
                      com.semmle.cli2.Commandline.executeAndCheckResult(Commandline.java:170)
                      com.semmle.cli2.Commandline.runWithoutReturn(Commandline.java:123)
                      com.semmle.cli2.database.DatabaseProcessCommandCommon.executeSubcommand(DatabaseProcessCommandCommon.java:226)
                      com.semmle.cli2.database.TraceCommandCommand.executeSubcommand(TraceCommandCommand.java:110)
                      com.semmle.cli2.picocli.SubcommandCommon.lambda$executeSubcommandWithMessages$5(SubcommandCommon.java:803)
                      com.semmle.cli2.picocli.SubcommandCommon.withCompilationMessages(SubcommandCommon.java:442)
                      com.semmle.cli2.picocli.SubcommandCommon.executeSubcommandWithMessages(SubcommandCommon.java:801)
                      com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:119)
                      com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(SubcommandCommon.java:199)
                      com.semmle.cli2.database.CreateCommand.executeSubcommand(CreateCommand.java:180)
                      com.semmle.cli2.picocli.SubcommandCommon.lambda$executeSubcommandWithMessages$5(SubcommandCommon.java:803)
                      com.semmle.cli2.picocli.SubcommandCommon.withCompilationMessages(SubcommandCommon.java:442)
                      com.semmle.cli2.picocli.SubcommandCommon.executeSubcommandWithMessages(SubcommandCommon.java:801)
                      com.semmle.cli2.picocli.SubcommandCommon.toplevelMain(SubcommandCommon.java:685)
                      com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:666)
                      com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:237)
                      com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:247)
                      com.semmle.cli2.CodeQL.main(CodeQL.java:115)
mbg commented 10 months ago

Hi @Xieyufengdd 👋

Thanks for raising this question about the error you're getting! The "Could not detect a suitable build command for the source checkout" error means that CodeQL has not been able to figure out how to build your project. If your project uses Maven, Gradle, Ant, or a custom build script, we should be able to figure out how to build your project in most cases. Can you confirm what type of project you have?

If CodeQL cannot figure out how to build your project or you're not using any of the above build systems, you can specify a custom build command. See https://docs.github.com/en/code-security/codeql-cli/getting-started-with-the-codeql-cli/preparing-your-code-for-codeql-analysis#running-codeql-database-create for details on how to do that.

Xieyufengdd commented 10 months ago

Hi @Xieyufengdd 👋

Thanks for raising this question about the error you're getting! The "Could not detect a suitable build command for the source checkout" error means that CodeQL has not been able to figure out how to build your project. If your project uses Maven, Gradle, Ant, or a custom build script, we should be able to figure out how to build your project in most cases. Can you confirm what type of project you have?

If CodeQL cannot figure out how to build your project or you're not using any of the above build systems, you can specify a custom build command. See https://docs.github.com/en/code-security/codeql-cli/getting-started-with-the-codeql-cli/preparing-your-code-for-codeql-analysis#running-codeql-database-create for details on how to do that.

Thanks for reply. I used IntelliJ to create it, sorry I didn't find it didn't support, but later I used it to create a database of a single cpp file, it report nearly the same problem "Could not detect a suitable build command", whether it is the same problem? Thanks !!!

mbg commented 10 months ago

Hi @Xieyufengdd,

For all compiled languages, we need to know how to build your project. If your projects are just single source files and you don't make use of a build system like Maven/Gradle (for Java) or something like a makefile (for C++), then you should follow the instructions in the documentation I linked to, specifying the command you use to compile your source files as the custom build command.