[build err] Python not found + questions

[mxrch]

Hello ! I have spent my day on understanding CodeQL, to use it in one of my project, but I couldn't make it work on Windows. I'm trying to analyze my repo (https://github.com/mxrch/ghunt) in local with Python queries, but when I create the database of the project, I get this error :

But you can see that I have Python installed. I also tried with Go but I don't have installed it so it throws an error.

So here is my question in addition of my issue : Does it requires any language binary ? Like installing Python, Go, GCC, MVN... ?

Thanks in advance ! Very cool project btw, I discovered it in the Github Actions.

[mxrch]

For the bugged french error, it means "Python can't be found. Try to execute it without arguments to proceed to [?]"

[adityasharad]

Hello, and glad to see your interest in using CodeQL!

It looks here like py.exe (the Windows Python Launcher) is on your PATH, but python.exe itself may not be on the PATH. Could you try running python or where python from the terminal and see whether it can be found? If not, please add the Python installation directory that contains python.exe to your PATH, and then try the codeql command again.

See https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/#python for other requirements to check when building a Python database.

[adityasharad]

I also tried with Go but I don't have installed it so it throws an error

Does it requires any language binary ? Like installing Python, Go, GCC, MVN... ?

To analyse code written in a compiled language (C/C++, C#, Go, Java), you must have installed the compiler and build tools that are relevant to your project. For example, if you're analysing a Go project, you'll need Go installed, and if you're analysing a Java project built with Maven, you'll need both Java and Maven. You can choose between providing an explicit build command via the --command argument (telling CodeQL exactly how to build your project), or you can leave out the --command argument and have CodeQL attempt to 'autobuild' your code. Either way you need the relevant tools installed: you should be able to build your code outside of CodeQL on the same machine. See https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/#creating-databases-for-compiled-languages for the details.

To analyse code written in a language that isn't compiled, the requirements are simpler, and you don't need a build command. See https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/#creating-databases-for-non-compiled-languages for the details.

[mxrch]

Hello, and glad to see your interest in using CodeQL!

It looks here like py.exe (the Windows Python Launcher) is on your PATH, but python.exe itself may not be on the PATH. Could you try running python or where python from the terminal and see whether it can be found? If not, please add the Python installation directory that contains python.exe to your PATH, and then try the codeql command again.

See https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/#python for other requirements to check when building a Python database.

Alright I copied the py.exe file to python.exe and now it doesn't throw error but it's been 10 minutes it's blocking here and the folder size doesn't increase, so I don't know if it's okay or not ☹️

And now the python binary on Windows is "py" by default, typing "python" or "python3" into the terminal redirect us into Python on the Microsoft Store, so I think you should add the "py" name when fetching the python executable in the PATH env !

[mxrch]

I also tried with Go but I don't have installed it so it throws an error

Does it requires any language binary ? Like installing Python, Go, GCC, MVN... ?

To analyse code written in a compiled language (C/C++, C#, Go, Java), you must have installed the compiler and build tools that are relevant to your project. For example, if you're analysing a Go project, you'll need Go installed, and if you're analysing a Java project built with Maven, you'll need both Java and Maven. You can choose between providing an explicit build command via the --command argument (telling CodeQL exactly how to build your project), or you can leave out the --command argument and have CodeQL attempt to 'autobuild' your code. Either way you need the relevant tools installed: you should be able to build your code outside of CodeQL on the same machine. See https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/#creating-databases-for-compiled-languages for the details.

To analyse code written in a language that isn't compiled, the requirements are simpler, and you don't need a build command. See https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/#creating-databases-for-non-compiled-languages for the details.

Okay, so to integrate it in a ready-to-use app, the best to do is to setup a docker with all the build tools for all the supported languages I guess ? :)

[adityasharad]

Alright I copied the py.exe file to python.exe and now it doesn't throw error but it's been 10 minutes it's blocking here and the folder size doesn't increase, so I don't know if it's okay or not ☹️

Looks like it's extracting your code. If you'd like to verify that with more detailed logging, try adding -vvv or -vvvv to your codeql command to increase the verbosity.

And now the python binary on Windows is "py" by default, typing "python" or "python3" into the terminal redirect us into Python on the Microsoft Store, so I think you should add the "py" name when fetching the python executable in the PATH env !

Thanks. Looking for py is a good suggestion and I've passed that on to our Python analysis team.

To unblock you here in the short term: the reason python or python3 redirects you to the Microsoft Store in this case is because the actual python.exe/python3.exe executables aren't directly on the PATH (docs here). So for now I suggest finding where those executables were installed on your system, adding their folder manually to your PATH, then trying again.

Okay, so to integrate it in a ready-to-use app, the best to do is to setup a docker with all the build tools for all the supported languages I guess ? :)

If you have multiple languages used in a codebase you wish to analyse, then an image with all the build tools sounds like a convenient way to do this. You may also wish to look into https://docs.github.com/en/free-pro-team@latest/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning if you want to run CodeQL analysis on your open-source project via GitHub Actions, rather than on your local system.

[mxrch]

@adityasharad Thanks for your response ! It's for a use in local yes, with possible languages.

Here is the logs before the freeze :

[2021-01-22 23:59:46] This is codeql database create ./databases/ghunt --language=python --source-root=/Users/thher/Desktop/labs/ghuntfree -vvvv
[2021-01-22 23:59:46] Log file was started late.
[2021-01-22 23:59:46] Using autobuild script C:\Users\thher\Desktop\labs\codeql-home\python\tools\autobuild.cmd.
[2021-01-22 23:59:46] [DETAILS] database create> Detected autobuild command: C:\Users\thher\Desktop\labs\codeql-home\python\tools\autobuild.cmd.
[2021-01-22 23:59:46] [PROGRESS] database create> Running command [C:\Users\thher\Desktop\labs\codeql-home\python\tools\autobuild.cmd] in C:\Users\thher\Desktop\labs\ghuntfree.
[2021-01-22 23:59:46] Running plumbing command: codeql database trace-command --working-dir=C:\Users\thher\Desktop\labs\ghuntfree -- C:\Users\thher\Desktop\labs\codeql-home\databases\ghunt C:\Users\thher\Desktop\labs\codeql-home\python\tools\autobuild.cmd
[2021-01-22 23:59:46] [PROGRESS] database trace-command> C:\Users\thher\Desktop\labs\codeql-home\databases\ghunt: Running in C:\Users\thher\Desktop\labs\ghuntfree: [C:\Users\thher\Desktop\labs\codeql-home\python\tools\autobuild.cmd]

Before running now with more verbosity I've been waiting for 40 minutes so I think it's really freezing, since the project is not very big !

[mxrch]

On my Parrot VM it's working very well ! In 1 minute