Codeql not recognising IAC code

akhiloscar commented 3 months ago

What happened?

I tried to initialise database for an iac code but it is not recognising the files eventhough there are .tf files within the directory

Version

0.1.0

Relevant log output

oscar@DESKTOP:~/test3$ gh codeql database create testdb12 --language=iac --source-root=/home/oscar/test3/
Initializing database at /home/oscar/test3/testdb12.
Running build command: []
Finalizing database at /home/oscar/test3/testdb12.
CodeQL did not detect any code written in languages supported by CodeQL. This can occur if the specified build commands failed to compile or process any code.
 - Confirm that there is some source code for the specified language in the project.
 - For codebases written in Go, JavaScript, TypeScript, and Python, do not specify
   an explicit --command.
 - For other languages, the --command must specify a "clean" build which compiles
   all the source code files without reusing existing build artefacts.

GeekMasher commented 3 months ago

@akhiloscar To be able to use the CodeQL IaC Extractor, you will need to download the release archive, unzip it, and either do the following:

Install in the same place as CodeQL
- Run gh codeql version --format json and find the unpackedLocation
Install it anywhere on the system and use --search-path with every command

# same place
gh codeql resolve languages
# different place
gh codeql resolve languages --search-path ~/path/to/unzipped/archive

You should see iac in the list.

I might need to write up better docs on this subject.

akhiloscar commented 3 months ago

@GeekMasher , iac is showing up while resolving language

akhiloscar commented 3 months ago

@GeekMasher , If possible can you share the repo for advanced-security/iac-queries

GeekMasher commented 3 months ago

They are CodeQL Packs (they are public)

https://github.com/orgs/advanced-security/packages?repo_name=codeql-extractor-iac

You can install them using the following:

gh codeql pack download advanced-security/iac-queries

akhiloscar commented 3 months ago

Thanks @GeekMasher , but getting cert error when downloading it

GeekMasher commented 3 months ago

Oh very interesting. That might be an issue with the GITHUB_TOKEN or GH_TOKEN being passed to CodeQL via the gh-cli.

You might have to login to github.com using gh auth login.

akhiloscar commented 3 months ago

@GeekMasher , Tried re-authenticating but facing the same error

GeekMasher commented 3 months ago

@akhiloscar Even more interesting. If you are using a Proxy, you might need to add your CA to the CodeQL's Java certificate store.

If this fails, try cloning the repository and running the following:

gh codeql pack install ./ql/src

This will install the pack manually from the source code (will compile so might take 1-2mins).

You might need to checkout a particular tag as I seem to have broken main in the latest commit 🤔

akhiloscar commented 3 months ago

@GeekMasher , Installed advanced queries/iac using "gh codeql pack install ./ql/src" but it seems like missing linux64/extractor directory when tried to initialise database

root@test-linux-01-IAC-POC:/home/adminuser/iac-codeql# gh codeql database create db1 --language=iac --source-root=/home/adminuser/iac-codeql

Initializing database at /home/adminuser/iac-codeql/db1. Running build command: [] [2024-03-24 07:18:59] [build-stderr] /root/.local/share/gh/extensions/gh-codeql/dist/release/v2.16.5/codeql-extractor-iac/tools/autobuild.sh: 3: exec: /root/.local/share/gh/extensions/gh-codeql/dist/release/v2.16.5/codeql-extractor-iac/tools/linux64/extractor: not found [2024-03-24 07:18:59] [ERROR] Spawned process exited abnormally (code 127; tried to run: [/root/.local/share/gh/extensions/gh-codeql/dist/release/v2.16.5/codeql-extractor-iac/tools/autobuild.sh]) A fatal error occurred: Exit status 127 from command: [/root/.local/share/gh/extensions/gh-codeql/dist/release/v2.16.5/codeql-extractor-iac/tools/autobuild.sh]

root@test-linux-01-IAC-POC:/home/adminuser/iac-codeql# cd /root/.local/share/gh/extensions/gh-codeql/dist/release/v2.16.5/codeql-extractor-iac/tools/

root@test-linux-01-IAC-POC:~/.local/share/gh/extensions/gh-codeql/dist/release/v2.16.5/codeql-extractor-iac/tools# ls autobuild.cmd autobuild.sh index-files.cmd index-files.sh pre-finalize.cmd pre-finalize.sh qltest.cmd qltest.sh

GeekMasher commented 3 months ago

@akhiloscar Sorry for the delay. I found out what the issue is.

When scanning configuration files, the extractor expects at least 1 HCL file (this is the main language supported). You can fix this but running a touch random.tf file before running the extractor but I plan on fixing this. I'll be raising an issue and trying to fix this tomorrow.

advanced-security / codeql-extractor-iac