search

github / codeql

CodeQL: the libraries and queries that power security researchers around the world, as well as code scanning in GitHub Advanced Security
https://codeql.github.com
MIT License
7.61k stars 1.52k forks source link

Help setting up the local repo so I can make and test changes to the open source shared queries #17061

Closed RobbingDaHood closed 3 months ago

RobbingDaHood commented 3 months ago

Description of the issue

Following this guide: https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/advanced-setup-of-the-codeql-cli#checking-out-the-codeql-source-code-directly

Then I want to make changes to the open source shared queries but I cannot get codeql to use the packs from the repo. So I cannot test my changes to the shared queries.

Reproduction

I am on a linux machine, I did something like this: 

mkdir ~/Projects/codeql-home 
cd ~/Projects/codeql-home
git clone https://github.com/github/codeql.git codeql-repo
wget https://github.com/github/codeql-action/releases/download/codeql-bundle-v2.18.1/codeql-bundle-linux64.tar.gz
tar -zxvf codeql-bundle-linux64.tar.gz
codeql/codeql resolve qlpacks
Output of the resolve qlpacks, not showing the repo and only showing what is shipped in the bundle ``` > codeql/codeql resolve qlpacks codeql/controlflow is found in 6 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/controlflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-examples/0.0.0/.codeql/libraries/codeql/controlflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/swift-queries/1.0.4/.codeql/libraries/codeql/controlflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-queries/1.0.4/.codeql/libraries/codeql/controlflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-examples/0.0.0/.codeql/libraries/codeql/controlflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-queries/1.0.4/.codeql/libraries/codeql/controlflow/1.0.4 codeql/cpp-all is found in 3 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-all/1.3.0 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-queries/1.1.0/.codeql/libraries/codeql/cpp-all/1.3.0 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-examples/0.0.0/.codeql/libraries/codeql/cpp-all/1.3.0 codeql/cpp-examples (/home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-examples/0.0.0) codeql/cpp-queries (/home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-queries/1.1.0) codeql/csharp-all is found in 3 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-examples/0.0.0/.codeql/libraries/codeql/csharp-all/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-queries/1.0.4/.codeql/libraries/codeql/csharp-all/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-all/1.0.4 codeql/csharp-examples (/home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-examples/0.0.0) codeql/csharp-queries (/home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-queries/1.0.4) codeql/dataflow is found in 16 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/dataflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-queries/1.1.1/.codeql/libraries/codeql/dataflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-examples/0.0.0/.codeql/libraries/codeql/dataflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-queries/1.0.4/.codeql/libraries/codeql/dataflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-examples/0.0.0/.codeql/libraries/codeql/dataflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/swift-queries/1.0.4/.codeql/libraries/codeql/dataflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-queries/1.0.4/.codeql/libraries/codeql/dataflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-examples/0.0.0/.codeql/libraries/codeql/dataflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-queries/1.0.4/.codeql/libraries/codeql/dataflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-queries/1.0.4/.codeql/libraries/codeql/dataflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-examples/0.0.0/.codeql/libraries/codeql/dataflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-queries/1.1.0/.codeql/libraries/codeql/dataflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-examples/0.0.0/.codeql/libraries/codeql/dataflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-queries/1.1.0/.codeql/libraries/codeql/dataflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-examples/0.0.0/.codeql/libraries/codeql/dataflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-examples/0.0.0/.codeql/libraries/codeql/dataflow/1.0.4 codeql/go-all is found in 3 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-queries/1.0.4/.codeql/libraries/codeql/go-all/1.1.3 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-examples/0.0.0/.codeql/libraries/codeql/go-all/1.1.3 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-all/1.1.3 codeql/go-examples (/home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-examples/0.0.0) codeql/go-queries (/home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-queries/1.0.4) codeql/java-all is found in 3 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-queries/1.1.1/.codeql/libraries/codeql/java-all/2.0.0 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-all/2.0.0 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-examples/0.0.0/.codeql/libraries/codeql/java-all/2.0.0 codeql/java-examples (/home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-examples/0.0.0) codeql/java-queries (/home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-queries/1.1.1) codeql/javascript-all is found in 3 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-examples/0.0.0/.codeql/libraries/codeql/javascript-all/1.1.1 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-all/1.1.1 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-queries/1.1.0/.codeql/libraries/codeql/javascript-all/1.1.1 codeql/javascript-examples (/home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-examples/0.0.0) codeql/javascript-queries (/home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-queries/1.1.0) codeql/mad is found in 16 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-queries/1.1.1/.codeql/libraries/codeql/mad/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-examples/0.0.0/.codeql/libraries/codeql/mad/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-queries/1.0.4/.codeql/libraries/codeql/mad/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-examples/0.0.0/.codeql/libraries/codeql/mad/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/swift-queries/1.0.4/.codeql/libraries/codeql/mad/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-queries/1.0.4/.codeql/libraries/codeql/mad/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-examples/0.0.0/.codeql/libraries/codeql/mad/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-queries/1.0.4/.codeql/libraries/codeql/mad/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-queries/1.0.4/.codeql/libraries/codeql/mad/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/mad/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-examples/0.0.0/.codeql/libraries/codeql/mad/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-queries/1.1.0/.codeql/libraries/codeql/mad/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-examples/0.0.0/.codeql/libraries/codeql/mad/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-queries/1.1.0/.codeql/libraries/codeql/mad/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-examples/0.0.0/.codeql/libraries/codeql/mad/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-examples/0.0.0/.codeql/libraries/codeql/mad/1.0.4 codeql/python-all is found in 3 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-queries/1.0.4/.codeql/libraries/codeql/python-all/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-all/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-examples/0.0.0/.codeql/libraries/codeql/python-all/1.0.4 codeql/python-examples (/home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-examples/0.0.0) codeql/python-queries (/home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-queries/1.0.4) codeql/rangeanalysis is found in 5 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-queries/1.1.1/.codeql/libraries/codeql/rangeanalysis/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/rangeanalysis/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-queries/1.1.0/.codeql/libraries/codeql/rangeanalysis/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-examples/0.0.0/.codeql/libraries/codeql/rangeanalysis/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-examples/0.0.0/.codeql/libraries/codeql/rangeanalysis/1.0.4 codeql/regex is found in 10 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-queries/1.1.1/.codeql/libraries/codeql/regex/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-examples/0.0.0/.codeql/libraries/codeql/regex/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-queries/1.0.4/.codeql/libraries/codeql/regex/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/swift-queries/1.0.4/.codeql/libraries/codeql/regex/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/regex/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-examples/0.0.0/.codeql/libraries/codeql/regex/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-queries/1.0.4/.codeql/libraries/codeql/regex/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-examples/0.0.0/.codeql/libraries/codeql/regex/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-queries/1.1.0/.codeql/libraries/codeql/regex/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-examples/0.0.0/.codeql/libraries/codeql/regex/1.0.4 codeql/ruby-all is found in 3 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-examples/0.0.0/.codeql/libraries/codeql/ruby-all/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-queries/1.0.4/.codeql/libraries/codeql/ruby-all/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-all/1.0.4 codeql/ruby-examples (/home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-examples/0.0.0) codeql/ruby-queries (/home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-queries/1.0.4) codeql/ssa is found in 16 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-queries/1.1.1/.codeql/libraries/codeql/ssa/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-examples/0.0.0/.codeql/libraries/codeql/ssa/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-queries/1.0.4/.codeql/libraries/codeql/ssa/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-examples/0.0.0/.codeql/libraries/codeql/ssa/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/swift-queries/1.0.4/.codeql/libraries/codeql/ssa/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ssa/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-queries/1.0.4/.codeql/libraries/codeql/ssa/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-examples/0.0.0/.codeql/libraries/codeql/ssa/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-queries/1.0.4/.codeql/libraries/codeql/ssa/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-queries/1.0.4/.codeql/libraries/codeql/ssa/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-examples/0.0.0/.codeql/libraries/codeql/ssa/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-queries/1.1.0/.codeql/libraries/codeql/ssa/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-examples/0.0.0/.codeql/libraries/codeql/ssa/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-queries/1.1.0/.codeql/libraries/codeql/ssa/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-examples/0.0.0/.codeql/libraries/codeql/ssa/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-examples/0.0.0/.codeql/libraries/codeql/ssa/1.0.4 codeql/suite-helpers is found in 9 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-queries/1.1.1/.codeql/libraries/codeql/suite-helpers/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-queries/1.0.4/.codeql/libraries/codeql/suite-helpers/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/suite-helpers/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/swift-queries/1.0.4/.codeql/libraries/codeql/suite-helpers/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-queries/1.0.4/.codeql/libraries/codeql/suite-helpers/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-queries/1.0.4/.codeql/libraries/codeql/suite-helpers/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-queries/1.0.4/.codeql/libraries/codeql/suite-helpers/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-queries/1.1.0/.codeql/libraries/codeql/suite-helpers/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-queries/1.1.0/.codeql/libraries/codeql/suite-helpers/1.0.4 codeql/swift-all is found in 2 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/swift-all/1.1.0 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/swift-queries/1.0.4/.codeql/libraries/codeql/swift-all/1.1.0 codeql/swift-queries (/home/USER/Projects/codeql-home/codeql/qlpacks/codeql/swift-queries/1.0.4) codeql/threat-models is found in 7 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-queries/1.1.1/.codeql/libraries/codeql/threat-models/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-examples/0.0.0/.codeql/libraries/codeql/threat-models/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-queries/1.0.4/.codeql/libraries/codeql/threat-models/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-queries/1.0.4/.codeql/libraries/codeql/threat-models/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/threat-models/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-examples/0.0.0/.codeql/libraries/codeql/threat-models/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-examples/0.0.0/.codeql/libraries/codeql/threat-models/1.0.4 codeql/tutorial is found in 16 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-queries/1.1.1/.codeql/libraries/codeql/tutorial/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/tutorial/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-examples/0.0.0/.codeql/libraries/codeql/tutorial/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-queries/1.0.4/.codeql/libraries/codeql/tutorial/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-examples/0.0.0/.codeql/libraries/codeql/tutorial/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/swift-queries/1.0.4/.codeql/libraries/codeql/tutorial/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-queries/1.0.4/.codeql/libraries/codeql/tutorial/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-examples/0.0.0/.codeql/libraries/codeql/tutorial/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-queries/1.0.4/.codeql/libraries/codeql/tutorial/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-queries/1.0.4/.codeql/libraries/codeql/tutorial/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-examples/0.0.0/.codeql/libraries/codeql/tutorial/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-queries/1.1.0/.codeql/libraries/codeql/tutorial/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-examples/0.0.0/.codeql/libraries/codeql/tutorial/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-queries/1.1.0/.codeql/libraries/codeql/tutorial/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-examples/0.0.0/.codeql/libraries/codeql/tutorial/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-examples/0.0.0/.codeql/libraries/codeql/tutorial/1.0.4 codeql/typeflow is found in 5 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-queries/1.1.1/.codeql/libraries/codeql/typeflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/typeflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-queries/1.1.0/.codeql/libraries/codeql/typeflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-examples/0.0.0/.codeql/libraries/codeql/typeflow/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-examples/0.0.0/.codeql/libraries/codeql/typeflow/1.0.4 codeql/typetracking is found in 16 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-queries/1.1.1/.codeql/libraries/codeql/typetracking/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-examples/0.0.0/.codeql/libraries/codeql/typetracking/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-queries/1.0.4/.codeql/libraries/codeql/typetracking/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-examples/0.0.0/.codeql/libraries/codeql/typetracking/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/typetracking/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/swift-queries/1.0.4/.codeql/libraries/codeql/typetracking/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-queries/1.0.4/.codeql/libraries/codeql/typetracking/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-examples/0.0.0/.codeql/libraries/codeql/typetracking/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-queries/1.0.4/.codeql/libraries/codeql/typetracking/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-queries/1.0.4/.codeql/libraries/codeql/typetracking/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-examples/0.0.0/.codeql/libraries/codeql/typetracking/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-queries/1.1.0/.codeql/libraries/codeql/typetracking/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-examples/0.0.0/.codeql/libraries/codeql/typetracking/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-queries/1.1.0/.codeql/libraries/codeql/typetracking/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-examples/0.0.0/.codeql/libraries/codeql/typetracking/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-examples/0.0.0/.codeql/libraries/codeql/typetracking/1.0.4 codeql/typos is found in 2 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/typos/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-queries/1.1.0/.codeql/libraries/codeql/typos/1.0.4 codeql/util is found in 16 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/util/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-queries/1.1.1/.codeql/libraries/codeql/util/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-examples/0.0.0/.codeql/libraries/codeql/util/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-queries/1.0.4/.codeql/libraries/codeql/util/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-examples/0.0.0/.codeql/libraries/codeql/util/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/swift-queries/1.0.4/.codeql/libraries/codeql/util/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-queries/1.0.4/.codeql/libraries/codeql/util/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-examples/0.0.0/.codeql/libraries/codeql/util/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-queries/1.0.4/.codeql/libraries/codeql/util/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/ruby-queries/1.0.4/.codeql/libraries/codeql/util/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/go-examples/0.0.0/.codeql/libraries/codeql/util/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-queries/1.1.0/.codeql/libraries/codeql/util/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-examples/0.0.0/.codeql/libraries/codeql/util/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-queries/1.1.0/.codeql/libraries/codeql/util/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-examples/0.0.0/.codeql/libraries/codeql/util/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-examples/0.0.0/.codeql/libraries/codeql/util/1.0.4 codeql/xml is found in 11 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-queries/1.1.1/.codeql/libraries/codeql/xml/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-examples/0.0.0/.codeql/libraries/codeql/xml/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-queries/1.0.4/.codeql/libraries/codeql/xml/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/xml/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-examples/0.0.0/.codeql/libraries/codeql/xml/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/csharp-queries/1.0.4/.codeql/libraries/codeql/xml/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-queries/1.1.0/.codeql/libraries/codeql/xml/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/java-examples/0.0.0/.codeql/libraries/codeql/xml/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-queries/1.1.0/.codeql/libraries/codeql/xml/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-examples/0.0.0/.codeql/libraries/codeql/xml/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/cpp-examples/0.0.0/.codeql/libraries/codeql/xml/1.0.4 codeql/yaml is found in 5 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-examples/0.0.0/.codeql/libraries/codeql/yaml/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-queries/1.0.4/.codeql/libraries/codeql/yaml/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/yaml/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/javascript-queries/1.1.0/.codeql/libraries/codeql/yaml/1.0.4 /home/USER/Projects/codeql-home/codeql/qlpacks/codeql/python-examples/0.0.0/.codeql/libraries/codeql/yaml/1.0.4 legacy-upgrades is found in 2 same-priority locations, so attempts to resolve it will fail: /home/USER/Projects/codeql-home/codeql/legacy-upgrades /home/USER/Projects/codeql-home/legacy-upgrades ```

Alternative: Extract into the codeql-home directly

So do the following after the above is tried out: 

mv codeql codeql-tmp 
mv codeql-tmp/* .
rm -r codeql-tmp 
./codeql resolve qlpacks

The output is the same as above, so no change (except in the paths)

The manifest file

The documentation states that codeql should automaticly detect the packs in the repo as long as it is in the same root dir: But when I look at '.codeqlmanifest.json` then that does not seem to be configured to do so: 

> cat .codeqlmanifest.json
{
    "provide": [
        "*/codeql-extractor.yml",
        "legacy-upgrades/qlpack.yml",
        "qlpacks/*/qlpack.yml",
        "qlpacks/*/*/*/qlpack.yml"
    ]
}

I did find a qlpack.yml file at ~/Projects/codeql-home/codeql-repo/java/ql/src and tried ot add that to the manifest file: 

> cat .codeqlmanifest.json
{
    "provide": [
        "*/codeql-extractor.yml",
        "legacy-upgrades/qlpack.yml",
        "qlpacks/*/qlpack.yml",
        "qlpacks/*/*/*/qlpack.yml",
        "codeql-repo/java/ql/src/qlpack.yml"
    ]
}

No changes when I run:

./codeql resolve qlpacks

Looking at the qlpack.yml files does not reveal any issue

Q1: What is the best Language in the World? ``` > cat qlpacks/codeql/java-queries/1.1.0/qlpack.yml --- library: false warnOnImplicitThis: true name: codeql/java-queries version: 1.1.0 buildMetadata: sha: dc20b0d19e9144624668a30d61855e709276bea2 cliVersion: 2.18.0 dependencies: codeql/java-all: 1.1.2 codeql/suite-helpers: 1.0.3 codeql/util: 1.0.3 suites: codeql-suites extractor: java groups: - java - queries defaultSuiteFile: codeql-suites/java-code-scanning.qls dataExtensions: - Telemetry/ExtractorInformation.yml ``` ``` > cat codeql-repo/java/ql/src/qlpack.yml name: codeql/java-queries version: 1.1.1-dev groups: - java - queries suites: codeql-suites extractor: java defaultSuiteFile: codeql-suites/java-code-scanning.qls dependencies: codeql/java-all: ${workspace} codeql/suite-helpers: ${workspace} codeql/util: ${workspace} dataExtensions: - Telemetry/ExtractorInformation.yml warnOnImplicitThis: true ``` They seems to be the same, and even have different versions, so it should be fine, right?

Try deleting the pack file in the shipped packs

cd qlpacks/codeql/java-queries/1.1.0
mv qlpack.yml qlpack_backup.yml
cd ~/Projects/codeql-home
./codeql resolve qlpacks

Now the shipped library is removed but nothing else is added. So it is not a matter of the shipped pack overwriting the one from the repo.

Conclusion

Can you help me setup the repo and codeql locally so I can make and test changes to the repo?

RobbingDaHood commented 3 months ago

I just tried another test:

Move the qlpacks: 

mv qlpacks/ qlpacks_backup/

Then run: 

./codeql resolve qlpacks

Observe that nothing gets found.

Change the manifest file: 

sed -i 's/qlpacks/qlpacks_backup/g' .codeqlmanifest.json

So that it now points at the new folder.

Rerun:

./codeql resolve qlpacks

Observe that it now finds all the packs again. Showing that it is in fact using folders in this root folder. Removing the risk that it did find these packs some other place in the system.

mbg commented 3 months ago

Hi again @RobbingDaHood 👋

I am on a linux machine, I did something like this:

mkdir ~/Projects/codeql-home 
cd ~/Projects/codeql-home
git clone https://github.com/github/codeql.git codeql-repo
wget https://github.com/github/codeql-action/releases/download/codeql-bundle-v2.18.1/codeql-bundle-linux64.tar.gz
tar -zxvf codeql-bundle-linux64.tar.gz
codeql/codeql resolve qlpacks

These steps look correct to me and, if I run them on my machine, the output includes results in the codeql-repo folder as expected. None of the other things you tried should be necessary.

Can you confirm that your codeql-repo checkout contains the files you'd expect? For example, is there a codeql-repo/shared/dataflow directory that contains a qlpack.yml?

RobbingDaHood commented 3 months ago

Interesting, it works now.

To be honest then I did not replay the steps I posted from scratch. I must have done some intermediate steps to screw it up.

@mbg you can close this stating that my initial script does in fact work as intended :)

Thx for the fast reply.

PS: just noticed that I am able to close the issue myself.