search

github / codeql-cli-binaries

Binaries for the CodeQL CLI
Other
754 stars 112 forks source link

Spawned process exited abnormally #125

Closed BobbyTRU closed 2 years ago

BobbyTRU commented 2 years ago

Hello there I have been trying to create a database for my python code but i keep getting an error. Log files:

[2022-06-15 14:51:04] This is codeql database create ../something3 --language=python
[2022-06-15 14:51:04] Log file was started late.
[2022-06-15 14:51:04] [PROGRESS] database create> Initializing database at C:\Users\Robert\Desktop\WarnIssue\something3.
[2022-06-15 14:51:05] Running plumbing command: codeql database init --language=python --source-root=C:\Users\Robert\Desktop\WarnIssue\testfolder --allow-missing-source-root=false --allow-already-existing -- C:\Users\Robert\Desktop\WarnIssue\something3
[2022-06-15 14:51:05] [PROGRESS] database init> Counting lines of code in C:\Users\Robert\Desktop\WarnIssue\testfolder
[2022-06-15 14:51:07] [PROGRESS] database init> Resolving extractor python.
[2022-06-15 14:51:07] Calling plumbing command: codeql resolve languages --format=json
[2022-06-15 14:51:07] [DETAILS] resolve languages> Scanning for [codeql-extractor.yml] from C:\Users\Robert\Desktop\codeql-win64\codeql\.codeqlmanifest.json
[2022-06-15 14:51:07] [DETAILS] resolve languages> Parsing C:\Users\Robert\Desktop\codeql-win64\codeql\cpp\codeql-extractor.yml.
[2022-06-15 14:51:07] [DETAILS] resolve languages> Parsing C:\Users\Robert\Desktop\codeql-win64\codeql\csharp\codeql-extractor.yml.
[2022-06-15 14:51:07] [DETAILS] resolve languages> Parsing C:\Users\Robert\Desktop\codeql-win64\codeql\csv\codeql-extractor.yml.
[2022-06-15 14:51:08] [DETAILS] resolve languages> Parsing C:\Users\Robert\Desktop\codeql-win64\codeql\go\codeql-extractor.yml.
[2022-06-15 14:51:08] [DETAILS] resolve languages> Parsing C:\Users\Robert\Desktop\codeql-win64\codeql\html\codeql-extractor.yml.
[2022-06-15 14:51:08] [DETAILS] resolve languages> Parsing C:\Users\Robert\Desktop\codeql-win64\codeql\java\codeql-extractor.yml.
[2022-06-15 14:51:08] [DETAILS] resolve languages> Parsing C:\Users\Robert\Desktop\codeql-win64\codeql\javascript\codeql-extractor.yml.
[2022-06-15 14:51:08] [DETAILS] resolve languages> Parsing C:\Users\Robert\Desktop\codeql-win64\codeql\properties\codeql-extractor.yml.
[2022-06-15 14:51:08] [DETAILS] resolve languages> Parsing C:\Users\Robert\Desktop\codeql-win64\codeql\python\codeql-extractor.yml.
[2022-06-15 14:51:08] [DETAILS] resolve languages> Parsing C:\Users\Robert\Desktop\codeql-win64\codeql\ruby\codeql-extractor.yml.
[2022-06-15 14:51:08] [DETAILS] resolve languages> Parsing C:\Users\Robert\Desktop\codeql-win64\codeql\xml\codeql-extractor.yml.
[2022-06-15 14:51:08] Plumbing command codeql resolve languages completed:
                      {
                        "extractors" : {
                          "cpp" : [
                            {
                              "extractor_root" : "C:\\Users\\Robert\\Desktop\\codeql-win64\\codeql\\cpp"
                            }
                          ],
                          "csharp" : [
                            {
                              "extractor_root" : "C:\\Users\\Robert\\Desktop\\codeql-win64\\codeql\\csharp",
                              "extractor_options" : {
                                "trap" : {
                                  "title" : "Options pertaining to TRAP.",
                                  "description" : "Options pertaining to TRAP.",
                                  "type" : "object",
                                  "properties" : {
                                    "compression" : {
                                      "title" : "Controls compression for the TRAP files written by the extractor.",
                                      "description" : "This option is only intended for use in debugging the extractor. Accepted values are 'brotli' (the default, to write brotli-compressed TRAP), 'gzip', and 'none' (to write uncompressed TRAP).\n",
                                      "type" : "string",
                                      "pattern" : "^(none|gzip|brotli)$"
                                    }
                                  }
                                },
                                "buildless" : {
                                  "title" : "Whether to use buildless (standalone) extraction.",
                                  "description" : "A value indicating, which type of extraction the autobuilder should perform. If 'true', then the standalone extractor will be used, otherwise tracing extraction will be performed. The default is 'false'. Note that buildless extraction will generally yield less accurate analysis results, and should only be used in cases where it is not possible to build the code (for example if it uses inaccessible dependencies).\n",
                                  "type" : "string",
                                  "pattern" : "^(false|true)$"
                                }
                              }
                            }
                          ],
                          "csv" : [
                            {
                              "extractor_root" : "C:\\Users\\Robert\\Desktop\\codeql-win64\\codeql\\csv"
                            }
                          ],
                          "go" : [
                            {
                              "extractor_root" : "C:\\Users\\Robert\\Desktop\\codeql-win64\\codeql\\go"
                            }
                          ],
                          "html" : [
                            {
                              "extractor_root" : "C:\\Users\\Robert\\Desktop\\codeql-win64\\codeql\\html"
                            }
                          ],
                          "java" : [
                            {
                              "extractor_root" : "C:\\Users\\Robert\\Desktop\\codeql-win64\\codeql\\java",
                              "extractor_options" : {
                                "exclude" : {
                                  "title" : "A glob excluding files from analysis.",
                                  "description" : "A glob indicating what files to exclude from the analysis.\n",
                                  "type" : "string"
                                }
                              }
                            }
                          ],
                          "javascript" : [
                            {
                              "extractor_root" : "C:\\Users\\Robert\\Desktop\\codeql-win64\\codeql\\javascript"
                            }
                          ],
                          "properties" : [
                            {
                              "extractor_root" : "C:\\Users\\Robert\\Desktop\\codeql-win64\\codeql\\properties"
                            }
                          ],
                          "python" : [
                            {
                              "extractor_root" : "C:\\Users\\Robert\\Desktop\\codeql-win64\\codeql\\python"
                            }
                          ],
                          "ruby" : [
                            {
                              "extractor_root" : "C:\\Users\\Robert\\Desktop\\codeql-win64\\codeql\\ruby"
                            }
                          ],
                          "xml" : [
                            {
                              "extractor_root" : "C:\\Users\\Robert\\Desktop\\codeql-win64\\codeql\\xml"
                            }
                          ]
                        }
                      }
[2022-06-15 14:51:08] [DETAILS] database init> Found candidate extractor root for python: C:\Users\Robert\Desktop\codeql-win64\codeql\python.
[2022-06-15 14:51:08] [PROGRESS] database init> Successfully loaded extractor Python (python) from C:\Users\Robert\Desktop\codeql-win64\codeql\python.
[2022-06-15 14:51:08] [PROGRESS] database init> Created skeleton CodeQL database at C:\Users\Robert\Desktop\WarnIssue\something3. This in-progress database is ready to be populated by an extractor.
[2022-06-15 14:51:08] Plumbing command codeql database init completed.
[2022-06-15 14:51:08] [PROGRESS] database create> Running build command: []
[2022-06-15 14:51:08] Running plumbing command: codeql database trace-command --working-dir=C:\Users\Robert\Desktop\WarnIssue\testfolder --index-traceless-dbs --no-db-cluster -- C:\Users\Robert\Desktop\WarnIssue\something3
[2022-06-15 14:51:08] Using autobuild script C:\Users\Robert\Desktop\codeql-win64\codeql\python\tools\autobuild.cmd.
[2022-06-15 14:51:08] [PROGRESS] database trace-command> Running command in C:\Users\Robert\Desktop\WarnIssue\testfolder: [C:\Users\Robert\Desktop\codeql-win64\codeql\python\tools\autobuild.cmd]
[2022-06-15 14:51:09] [build-stderr] Traceback (most recent call last):
[2022-06-15 14:51:09] [build-stderr]   File "C:\Users\Robert\Desktop\codeql-win64\codeql\python\tools\index.py", line 25, in <module>
[2022-06-15 14:51:09] [build-stderr]     buildtools.index.main()
[2022-06-15 14:51:09] [build-stderr]   File "C:\Users\Robert\Desktop\codeql-win64\codeql\python\tools\python3src.zip\buildtools\index.py", line 191, in main
[2022-06-15 14:51:09] [build-stderr]   File "C:\Users\Robert\Desktop\codeql-win64\codeql\python\tools\python3src.zip\buildtools\index.py", line 181, in site_flag
[2022-06-15 14:51:09] [build-stderr]   File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 364, in check_call
[2022-06-15 14:51:09] [build-stderr]     retcode = call(*popenargs, **kwargs)
[2022-06-15 14:51:10] [build-stderr]   File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 345, in call
[2022-06-15 14:51:10] [build-stderr]     with Popen(*popenargs, **kwargs) as p:
[2022-06-15 14:51:10] [build-stderr]   File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 969, in __init__
[2022-06-15 14:51:10] [build-stderr]     self._execute_child(args, executable, preexec_fn, close_fds,
[2022-06-15 14:51:11] [build-stderr]   File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 1438, in _execute_child
[2022-06-15 14:51:11] [build-stderr]     hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
[2022-06-15 14:51:11] [build-stderr] FileNotFoundError: [WinError 2] The system cannot find the file specified
[2022-06-15 14:51:12] [ERROR] Spawned process exited abnormally (code 1; tried to run: [C:\Users\Robert\Desktop\codeql-win64\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, C:\Users\Robert\Desktop\codeql-win64\codeql\python\tools\autobuild.cmd])
[2022-06-15 14:51:12] Exception caught at top level: Exit status 1 from command: [C:\Users\Robert\Desktop\codeql-win64\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, C:\Users\Robert\Desktop\codeql-win64\codeql\python\tools\autobuild.cmd]
                      com.semmle.cli2.database.DatabaseProcessCommandCommon.executeSubcommand(DatabaseProcessCommandCommon.java:215)
                      com.semmle.cli2.database.TraceCommandCommand.executeSubcommand(TraceCommandCommand.java:86)
                      com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110)
                      com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(SubcommandCommon.java:162)
                      com.semmle.cli2.database.CreateCommand.executeSubcommand(CreateCommand.java:151)
                      com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:500)
                      com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:205)
                      com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:214)
                      com.semmle.cli2.CodeQL.main(CodeQL.java:98)

Tried creating the database inside and outside of the source folder but it did not make a difference. For testing purposes I tried creating a db for a javascript file and it worked without any issues.

pip --version = 22.0.4 python --version = 3.10.5 virtualenv --version = 20.14.1

OS is Windows 11

Python code is just a simple "print("henlo")".

hmakholm commented 2 years ago

Thanks for the report. Our Python engineers suspect you're hitting a known Python setup problem, but the strategy our code employs to detect it might be backfiring in your environment.

Can you try seeing if it makes any difference to set the environment variable CODEQL_EXTRACTOR_PYTHON_ENABLE_SITE to some nonempty value before starting codeql?

RasmusWL commented 2 years ago

Just to confirm, does py -3 work on your command-line? (spawning a Python 3 interpreter)

This is an option when installing Python on windows, and is currently required to make CodeQL work locally

image

If py -3 does work, one possible explanation is that the extractor thinks this should be analyzed as Python 2 code, but that a Python 2 executable is not available. Could you try setting LGTM_PYTHON_SETUP_VERSION=3 in your environment?

BobbyTRU commented 2 years ago

py -3 did not work on my command-line and after installing it, the database creation works. Thanks for the help :D